⚡️Open Questions in Agentic RL — Will Brown (Prime Intellect)

A quick lightning talk given at Intelligence Unbound. https://lu.ma/wxberrb8?tk=2pGwtb Introduction to Open Source Agent RL [00:00] The talk will focus on the frontier directions in open-source agent RL, aiming towards training models that resemble 03 in their capabilities. The goal is to develop general-purpose agents capable of performing long-horizon tasks, such as web Browse, code writing, and image analysis [00:20]. While models like R1 excel in single-turn tasks, the focus is now on creating more general-purpose agents [00:47]. The speaker believes that the challenges in achieving this are solvable, and the talk will outline the roadblocks and potential solutions [00:59]. The Need for Multi-Turn Tool Use [01:13] The ability for LMs to "do stuff" effectively relies on the use of tools [01:20]. Multi-turn tool use is crucial, especially in diverse settings beyond just code-related tasks [01:36]. The concept of "10-minute AGI" is introduced, suggesting that models should be able to handle tasks a human can do in 10 minutes [01:50]. Multi-turn RL is identified as a key area, training models to use tools iteratively to solve problems and receive rewards based on the final output [02:28]. Challenges in Scaling Agentic Models [03:04] Current models can perform many tool calls (e.g., 100 by DeepMind's research), necessitating intermediate verification and turn-level rewards [03:10]. Challenges include verifying the correctness of actions outside of math and code, and managing context length when processing large amounts of data like websites [03:41]. Efficient infrastructure is needed to manage the computational resources for these rollouts, especially for tasks like data science [04:01]. The importance of asynchronous and overlapped processes is emphasized to avoid inefficiencies in computation [04:19]. Recent Progress and Community Efforts [04:45] Recent work has shown progress in incorporating tool calls into multi-turn RL [04:50]. The open-source community is act