Google's NEW Dual-Agent AI

Key Takeaways

hello Community Google Deep Mind has a brand new architecture out and it is fascinating we are now talking here about a dual agent architecture a talker reasoning architecture and Google says hey we therefore divide one agent into at least two agent for the Fast and the intuitive talking we have talker agent and for the slower and deliberative reasoning we have a Reasoner agent and Google says hey they should both have access to a common memory where the Reasoner agent will upload new insight from the deep thinking for the talk agent to use and the conversation with the humans and Google says hey let's use this react this chain of sord prompting so we have both reasoning traces and task specific actions like tools that we can call with our llms plus we will add reflection to extend here the react framework with self-reflection to improve here the causal reasoning of our reasoning agent here's the perfect video for you if you want to learn more about self-reflective AI it's some beautiful 40 minutes for you to enjoy or you go here with the chain of sword with the graph of sword and you might ask hey why does googol not use the graph of Sword it is a much better algorithm well chain of Sword is simpler and cheaper and Google says here for the reason is an agent here we will have a continuously update that it updates its belief and you might ask why it's belief I will explain this in a second about the user's goal the user plans the barriers we have in the communication the knowledge barrier the motivations we have what the user might have in the form of a structured object or schema and yes you guess it we will go with Jason and Google says hey you know this talk of reasonal Architecture is a dual system framework inspired here by this idea by Daniel kiman about sinking fast and slow and if humans do this sinking fast and slow let's simulate this in our AI systems so it aims you to address dual challenging face by agents especially here system one syncing and more complex system two syncing and Google says you know what we will do here everything that is beautiful we will have here the environment and the interaction with the environment we will use here the partially observable mock of decision process that we know perfectly well how to do this in a mathematical framework plus we will use your in context learning between the learning of our two interactive agents and the whole system we will use here reinforcement learning so we have the perfect communication of our eii system with the human but also we have here get a perfect harmony between our two agents and let's start so if you're not familiar from MC green grasshoppers with the marov marov structure I have here this video where I show you a game theory explained for multi-agent and I use your mark of game this is a a little bit of an extension of the mark of decision process but here in this video you'll find everything to know about Mark of game structures if you want to learn about in context learning a little bit more and and Visa the pre-training AL methodology in integration of rack this is the video for you unsupervised in context learning Plus or if you want to go here with the reinforcement learning I have here a complete code example where I also explain here a new mechanism here DPO but this is already five six months ago with path Laur forbit hugging face sft and so on so but now let's dive into this new IDE yeah let's have a look at the agents what is each agent doing in particular so system Wonder talker you know the one talks with the human says hey how are you yeah great to see you it's a beautiful day so you have a task for me it is great so the talker the function is to handles fast intuitive and realtime conversation with the human with the query that comes in the responsibility of the toos are generate natural language response let's say in English inter interact continuously with the user you know it should be almost like a human conversation and to retrieve and utilize the latest belief States from the common memory with the system 2 to inform here the responses here on more details on more insight because we have a backup system that is running here all the deep thinking the system to and the implementation is easy we utilize here a powerful in context learned llm let let say since we have with Google at Gemini 1.5 flash conditioned with specific instruction to maintain the coherence and the empathy in the interaction with the humans this is our talker system one reason our system two is now this now has the function to manage a slow deliberative and logical causal reasoning task here we have the thinker of our system the responsible of the Reasoner is perform multi-step reasoning and planning so this is one of the crucial agent activities planning the strategy The Next Step that we're going to take here that the talker is going to communicate with the human this system too can now have external tools or connect API calls for an information retrieval through tools and databases plus it will update and maintain the structured belief State and I will talk about this in a minute about the user and the environment so this Reasoner tries to understand what is the environment we are talking what is the user doing right now where are we in the environment what are the intentions of the user what are the expectation of the human being how can I respond in a way that is in accordance with the expectation management plus it should be like a human human conversation and then we develop and adjust here all the plans that we have based now in real time on the user feedback so you see we continuously update dat our plans it is a selflearning system and if you seen my last video you see what a coincidence that now after we looked here at Microsoft and openi now Google comes out with this brand new insight well what a coincidence implementation employs a hierarchical reasoning model leveraging chain of sord prompting and Tool enhanced approaches here with the classical I think 2-year-old now the react framework with reflection as shown to you in my video now this interaction between agent is nice it's simple elegant beautiful so we have an asynchronous operation the toer operates continuously using the most recent belief State available in the common memory shared between both agents while the Reasoner Works in parallel to update all the beliefs and generate here the complex planning the complex strategy how to solve the task how to continue the conversation with the human and of course memory integration both agents interact through a shared memory system where the Reasoner updates the belief that the talker can then retrieve to inform its response to have a human human conversation let's have a very simple idea this is here from the original convers original publication here by Google and you see here if we have only a single based agent that talks and extract with multi-step reasoning you know you need a real powerful a real expensive llm so therefore here you see here we have the user we have the feedback we have the llm beautiful we have memory multip we have the interaction with the world we have the observation we have the feedback and you need a powerful LM you can't go with a little 8B or 13B mark no but wait a minute if I have here now two agents I have here my reasoning agent and I have here my talking agent and the interface is here a shared memory now the idea is I have here the human and here the talker talks here with the human conversation if I have the talker now as a cheaper LM let's say a Gemini flash model a 1.5 8B model this is enough to run here all the human conversation that are hey nice weather today hey you have a task for me and then only when it is necessary we connect here to the Reasoner agent and the Reasoner agent listens of course here word is now uploaded to the memory for specific task from the conversation with the human and then starts to syn start to Mi plan start to connect to other tools on the internet to other databases what whatever there is does now here the multistep reasoning does comes up with a plan how to solve this task and shares the older information and uploads it back to the shared memory and then the talker says hey I have something new in the memory and now communicates with the human so it's cheaper to run AI system and only on demand when there's some real deep thinking necessary or some connection here to the internet some computer simulation then we activate a much more expensive llm isn't this a beautiful example so Mar of decision yeah agent environment is formulated as a partially observable mock of decision process and in this setting the agent interacts with a world it includes you the user all the external knowledge bases and the sensory inputs and as you know classical the agent can formulate sorts around the action it can take and decide which tool to select API calls engine like search or specific functions to fetch here all the external knowledge necessary to accomplish the task by the users and by combining the series of sord and action along with the results yeah the agent can now create a plan for solving the problem a strategy our deep thinking agent comes back with a strategy to solve the user task now if you're not familiar here with this process we have States the States represent the true state of the world including the user content and the environment and we have then belief and a belief state is updated through iasion inference I will show this in a minute you know the actions external action like doing API calls and internal action here Google says these are our SS our belief updates because we do not have the absolute knowledge about the external world we just believe or the AI system just believes that there is a certain state of the world if I only have a partially observable Mark of decision process I have the observation I have the transition I have the reward function and the policy for oral classical term and the belief update is here I ask your jet GPD for and I said he update the beliefs that to aasan inference means adjusting the agent understanding or the partial belief about the state of the world based on new evidence and observations and the ban inference is a meod of statistical reasoning for hundreds of years we noce but combines here prior knowledge with new data to update the probability and you know by theorem here about probability distribution and this is simply what we apply now important that here this Mo framework here implicitly and explicitly here models here the UN certainty that we have because we only have a certain sensor array infrared lighter whatever ever but we don't know the exact state of the external world so an agent is in a partially observable environment it only receives partial information of the true state of the world and the agent maintains here A belief state which is a probability distribution more or less over all the possible State reflecting in the uncertainty about the real true State and this is here statistics this is mathematics and here we have all the mathematical formulas developed over hundred of years here we know how to handle this so Mark of process necess is necessary in the talk reasonal architecture because short question to my green grasshoppers because it handles here really nice the partial observability it simplifies your the decision- making process it supports Your Efficient learning and it formalizes your sequential interactions that both agents have to perform this is the beauty of having here a mock of process and you know what this action space we have an augmented action space and we have a looping there so in our marro framework the agent action action space is augmented to allow now for a richer set of behaviors including that the reasoning capabilities the acting capabilities all the tool use that are now available to us and the belief modeling as I just showed you and in the action space we have sorts those represents the agent's internal reasoning traces we have to Tool usage agent can interact with all the external apis databases and knowledge sources to retrieve information that is necessary we have to believe updates I just showed you the agent extracts and models beliefs about the user at the task at hand and these beliefs are structured well what a coincidence as Json or XML and stored in the shared memory between the two agents and the conversational response gen ated by the talker agent based on the current belief State and the interaction history of the human conversation so all the information is there in our action space beautiful and as I told you we have now here this beautiful idea here that the talker system that engages you with the user in real time generating quick conversational response based on the belief stored in the memory and this is now the responses that are driven by the user input and the agent's context window this means the shortterm memory and the beauty is we can have here a rather not so causal reasoning excellent model like a nice little Gemini flesh model the Reasoner the system tuner operates in the background running only when it is really necessary running but then multi-step reasoning task solving all the sub problems using here calling the tools updating the belief State here and once the reason is complete the belief is stored in the memory for the talker to access this knowledge for future interaction we have a new strategy developed by our system to and therefore we need a more expensive let's say the Gemini 2 pro model nice nice energy optimization problem memory integration yep uh belief state in the talk model is presented as structured objects that store the agent's understanding of the user goals the user preferences and the complete context of the conversation and how to do the progress toward the task completion and this beliefs data as I just showed you is continuously update through the interaction with the user and the tool calls by the Reasoner agent in the memory have we have here the long-term storage of the belief States and the interaction history and when the talker engages with the user the talker simply pulls here the relevant information from the memory to Prime here its conversational responses so you see see we have kind of a selflearning system now suddenly because the more intelligent question the human will ask the system therefore the more complex and maybe more intelligent answers will be stored in the memory and therefore here our Reasoner will have to access higher complexity solutions to Simply solve the task but a human reinforcement learning yeah the classical oral framework the agent employs here reinforcement based approach to learn the policies that balance now two system that talkers Fast Response with the Reasoner Sor reasoning optimizing it task completion and user satisfaction this is the classical that we know there's nothing special about it talker Reasoner agent both formulate here a reinforcement learning framework and the key is that the policy Pi for the agent is complemented or implemented via in context learning of course what else we have we have a shared M so what we use we use in context learning our old friend in context learning and this combines here the agent current belief State the user input in the agent's memory to select here and specific action for a green g h us this is policy explained these are the rewards explained in a second and then the policy function is updated based on the feedback using techniques such as proximal policy optimation remember openi years ago po DPO whatever we have now and the methods of train to maximize now the cumulative reward over time ensuring that agent improves its a conversational strategy with the human and be the problem solving capability to really solve the query the task by the human in context learning and prompt engineering let's have a look at this beautiful so the talk reason assisting relies heavily on in context learning we have a problem with all this additional information we have in context learning wherein the llms are provided a carefully constructed context window that includes the last user input the observation the current the complete current belief state from the memory maybe in Jason the interaction history a log of the previous action and responses and a set of instruction this instruction guide the agent's Behavior such as being empathic friendly nice providing accurate information or breaking down the time into actionable steps so you see in my last video where I put all of this here in the intelligence Shield if you have seen this video now Google uses here a little bit modification here not in the shield but it simply splits up the agent in two let's say horizontal valued agents but you see the idea is the same the future development is heading to a convergence to this idea yeah external knowledge integration you know all of this as I've showed you here this whole video is real similar to the last video I showed you here by Stanford and open ey they put here a lot of this functionality in this logic controller here to separate the task break down high complexity task in multiple simpler complexity task and and and but you see it is happening now for all the global EI corporations let's look at the future what is the outlook here what can we improve well system performance can be easily improved hereby just adding multiple reasoners why only one simple agent it might be specialized only here on logic and mathematical causality let's have another reason about a different domain knowledge specialized in different aspects goal setting planning knowledge retrieval we have so many Reasoner and Google I think will offer this for you to you as a paid service what is a prise and then automated Reasoner probing the talker should automatically probe the Reasoner when it detects the situation requires your system to reasoning so you see you can fasten this interaction cycle you can much better tune here the responses here you can add multiple Reasoner so their future is beautiful bright and the eii systems continuously evolve they become more and more complex more and more interwoven but at the same time the corporation try to use here the cheaper smaller models like a sweb or an 8 billion free trainable parameter model because it is faster it costs less to do the easier jobs so you see this task decomposition is really taking place taking care about the different complexity level that you encounter on in here the human query okay great I hope you enjoyed it I hope you have a new idea what is coming now from Google Google Deep Mind and it would be great to see you in my next video