The Eigenvector of Multi Agent Systems w/ RAG

Key Takeaways

Hello community so great that you are back today we talk about rag in multi- aent system and you might ask hey how do we build a team of y agent that are just perfectly coordinized now you know now we move from prompt engineering away now to a cognitive architecture of multi agents and they will have a social quotation mark dynamics the machine will have a social dynamics so we move now from an LLM whisperer to NEI society architect with multi- aent structures now easy what is our task today we have a brand new paper it's gorgeous the best day paper of today solution two we have here if you want here a benchmark and look those are the LLMs that you know Kimmy deep one QN gemini claw 2D5 Gro 4 and the performance the accuracy is the best one is Grog 4 30% this is How can we go to 50? This is what we're going to talk about today. So, let's start here. HLE, if you're not familiar, humanity's last exam, it's a little bit on the scientific side. Biology, chemistry problems. No. And you see our current LLM here, the performance degrades substantially. If you go a little bit more complicated and now we combine this that we say, hey, we need external data from a deep domain knowledge. Of course, this is our rack system and we will have to integrate rack into a complex multi-step reasoning done by multiple agent at different times and different complexity levels. So you might say, hey, finally AI is getting interesting. Absolutely. So two fundamental architectural limitations we're going to talk about and their solutions. So we do have a fragmentation of the logical flow of the reasoning traces of our GVD5 through explicit tool invocation whenever we have to call rag or no retrieval augment more or less just to get new external data the reasoning process is interrupted plus we see that there is an inefficiency if we have a democratic multi- aent collaboration where you have majority vote so how can we cope with those problems and I will show that we have two error types. We have an error type one simply that our LLM is overconfident. There's no rag at all. Just the LLM goes to its parametric knowledge and says, "Hey, yeah, I know for this particular formula, this is the mathematical expression. Turns out it was a pure hallucination." No, it is something else. But why do you need a ra system? No, the LLM knows it. Quotation mark. No, it hallucinates it. And the air t is much more interesting. It's a rack calling that disrupted you the reasoning traces of the LLM. So it's like a human you thinking about something you say hey I need more information. So you go have a deep dive, maybe you Google, you come back with the solution and you say, "Hey, where was I in my reasoning process? How do I integrate this new result now, this new data, this new knowledge into my old reasoning traces? And how do I modify now the reasoning traces?" And we can show LLMs have the same problem. Of course, they mimic human behavior. We have an error type two with red calling disrupting the reasoning. So let's solve this. Now if you take a step back and you look at the multi- aent collaboration you say you know what we have two separate problems. We do have again a structural complexity topic that we solve now in an architectural hyper plane and we have a problem here regarding here the communication complexity. So the amount of information and time and everything that we share between our multi- aents the information exchange itself the amount the details when at what time frame we have to do this what information to provide to which LLM and which agent is taking here decision do we have a majority voting system on this so my goodness it's again complexity and you guessed it what is the solution yes of course we integrate this with rag because otherwise hey this would be a boring video so let's have some fun. What is the solution? We have a new paper and it's called IGEN 1. Egon one is like IGN value or IGEN function. IEN 1 is a beautiful idea and it pushes here exactly what I showed you here our benchmark close to 50%. 48.3. Okay. So you see our GPD5 in itself has a uh performance accuracy of 22.8%. And even if you have now an openi deep research here done by the old 04 mini for example you also achieved here for this particular benchmark here humanity's last exam 22.8%. But we want to go to 50. So how we do this? We do this with ien. So it is a new framework and this is it. Listen it just has three components. It's simple as hell. It just has here the retrieval argument. It's not direct. It's a retrieval augmented process uh framework that is continuously monitoring the tokens. So you have a monitor-based rack system that goes here and I will show you this in an example and I will show you this in detail. No problem here. Operating continuously at the token level to detect knowledge gaps via semantic uncertainty. It will then generate immediate queries for the rack system and will inject the retrieved information seamlessly into the reasoning process. We will have another element. This is a hierarchical solution refinement something beautiful because now with this information the system now generates multiple solution. And now with this idea we take now let's say 10 solutions and we take one candidate solution as an anchor and then we have nine other solution that we cross check against and we do this here iteratively. So we squeeze out every piece of information that we have in those 10 solutions and we compare and we generate a hierarchical solution refinement process and finally we have an iterative reasoning process also with multiple agents. So it's full of agents here in our video. Let's say I have here a look. So we start the first is the monitor-based rag. So retrieval augmentive checks here continuously. It monitors here continuously the reasoning at the fixed interval of 512 characters within the read next token generation reasoning trace generation. 512 characters with an overlap of 128 characters. It doesn't miss anything here at the border. Great. So you have a problem then you go and you see in the reasoning trace of your LLM of a GP5 or no maybe an open source that you really see the reasoning trace and UI tells you hey I know a little about this pi parameter and this is detected immediately by the monitor-based rack system says okay so we need to provide additional information we don't wait another 500 characters in the reasoning trace no immediately we have a query that goes to a rack system to a database no SQL whatever you have you have additional information. You have an injector. You project it back. You remember here we have multimodality can possibility. You project it back here and suddenly you have additional context. So context optimation now. But look at it. You see what we are doing. Take a step back and look at this. This is here a structure that you are familiar with. Now why we do this? By ensuring that the queries here that we send off are as fine grained as possible as minimal as as simple as possible the queria our agent here avoids unnecessary expansion of the search space immediately we say hey I don't understand a single term so immediately go and I check what is this term give an explanation I don't want that my search space expands to higher dimensions I want to have it minimum and immediately responds Therefore maximizing here the relevance of the retrieved evidence. So a complete new rank system here that operates here within 512 characters of the reasoning trace. But you know what we do? This is also a complexity. you know 512 characters or a reasoning complexity and we reduce this complexity of an uncertainty that we are not familiar with in rag into its smallest elements as fine grained as possible and those smallest element we query. So the com the intelligence of the system is not that the system has a complex intelligence. No, it's the complete opposite. What we do is we go to a complexity, we chop it in pieces, hundreds of pieces, and then we just solve each single little piece of those 100 pieces. And then hopefully hopefully we can put it together in a coherent solution back. But therefore, we have here the second part. So just wait a second. So injector here sorry injector coming back with the rag results filters and compresses here the raw rag outputs into a concise utility focus snippet. Yes of course we have here another agent and you know exactly what's great. So if you want we have provided additional context now injector here has the original context and the rag results and you are familiar with this. Great. If you want to see this, remember here with the injector, you can have problems too. If you want to see an example, let's have a look. This is here the human question. This is here from our benchmark here one query. So, LLM begins reasoning set. Okay, I have here two inre whatever. And then you see the monitor comes and say detected an uncertainty in the reasoning trace and re combination induced change points. No. And the query says, oh, I immediately define a query. I generate here and extract keywords, key phrases. I formulate from those a query and I immediately query here. Hey, how many upload type recombination change points whatever I get a knowledge back from the retriever and then the injector brings it back here this additional information everything that we know at all. But now that we have this information, now starts the real work. Now, because now we have an augmented query if you want and now we have multiple agents operating here. We have a proposal that generates now with this additional context initial solution. Let's say 10 different solution. You know we are in probability system. So who knows what's the right solution. Then we have a corrector. Corrector checks here one by one. And then comes the beautiful thing here of a refinement. This refinement is here our second step. You remember our hierarchical solution refinement and this is special. So for each of those let's say 10 solutions we apply in our repair strategy for the logic completion for the numerical corrections for the methodology replacement or for an expression refinement for technical terms. So we do have selected one of the first of the 10 um elements here. We have one anchor and nine if you want solutions. And now we just cross check cross reference and here it depends of course of the LLM how intelligent are you. So this is a lot of fun if you want to see it here in text. What we have is a peer informed repair mechanism. This is here HSR challenges the assumption that all the solution let's say 10 here should contribute equally to the final output it is not that every solution has the same beauty or the same intelligence no and a vote of AI system is maybe not the best way to do this no so therefore structured relationship mirror expert collaboration patterns and peer informed repair structure and then we go on here to the quality aware their iterative reasoning process. And here we have another LLM based evaluator, another agent. And here we def the team here defined three quality dimensions, logic, answer, and explanation plus a suggestion for improvement. And we have a specific threshold here for multiple dimension. If you cross this threshold, you the answer crosses here the threshold. You have a go select the best solution. You have a ranker ranker. you that you know from the classical rank rack systems and you have a final answer. So you see simple familiar but the complexity of agent agent agent agent agent and agent interacting and having given here particular task on a very low complexity it tells you oh we have to reduce the complexity to the maximum otherwise it would not work. But given this new architecture that we design here of the multi-agent cooperation, we get a result that is just great. Let's have a look at this. This is the study one adaptive multi-agent refinement with monitor brace drag for scientific reasoning Yale University, Shanghai Changdong University, Fanang University, University of California, LA, Shanghai EI lab, University of Oxford and Aen EI and published September 25, 2025. So nice. If you want to see the complete example here, if you have done the step two with LLM resumes here, the reasoning goes here for the coding, goes down for the extraction here of the coding results and the final summary. Beautiful. I record this video just 9 hours after this was published. And as you can see on GitHub, they say more details will come soon. So whenever you will see this video, maybe tomorrow, I hope you already have here the code for the GitHub repo. So let's look at the result again. This is here numerical and you know the first benchmark I just showed you this at the very beginning of the video a GBD5 gets 22%. This is not famous and other agentic system that we have go up to 34%. So okay there's a significant jump if you have multi- aents but look at this if you go with Ian and you go with an open source a deepseek version 3.1 at a posit one you are close to the 50% that we want and if you go with a passet five we go over 60% performance this is really nice and you have here other benchmarks that you can compare and you see okay it jump from 22 to 50 or 60% accuracy for a multi agent system. It seems to be working. Now they give you here and this is now a screenshot from the annex here. They give you exactly how the rack database how it was constructed the 10,000 PDF papers in biology and chemistry extracted here the PDF plain text and everything and then the positive and negative keywords and everything plus they give you the complete prompt. So for the rack bullet point summarization real nice and they also give you here for the rack in chapter they although it's not in the GitHub at least in the annex of the publication you have here the complete prompts but you know what's really interesting we take a step back and we look here at the complete study the insights it provides here I one a micro architecture of an efficient and effective agent collaboration system now and it's to subsystem create here a formal protocol if you want how agents should interact and should rerank and should operate to refine solutions here scientific solution and converge faster on the truth and what I've not told you here they argue that they use less token than the other multi- aent system here that are competitive because since they are monitoring continuously they immediately identify any missing knowledge, you know, the knowledge gap or any misunderstanding of a technical term and they don't go somewhere else explode into or going into loops. Say they're much more efficient, which is great. But of course this is just the first part of the story because in my next video we're going to have a deep dive into the micro dynamics now that govern especially those interaction between our multi- aent system that we just looked at and this will be a video that complements this now from a macrodnamical perspective. I hope you enjoyed it. Subscribe. I see you in my next video.