Apply LLMSelector to your AI Agents (Tutorial & Code)

Key Takeaways

hello Community there no is beautiful out there if you want to go skiing do it now but before before we have to look here how to optimize youri system and we're going to talk about optimization for your llm in multi agent system and here we go and you no this no you have multiple llms gp4 CLA Gemini llama whatever who cares and you have different benchmarks like life code bench or common gen generative data set here the hard version and you have different performances from single llm so Vision language mod no you know this or here simple question and answer data set where you have 4,326 short facts here and you see my goodness the performance is almost zero now or you go here to other ones where there's a beautiful saturation but this is not important you know what is important this is important then we have other benchmarks like table arithmetic or table bias and here you see all the models are almost at Zero Performance except if we combine those llms that have zero performance in a particular way in both cases we achieve 100% so what is the miracle that is happening here and is this possible at all so let's open up this video so multii agent system so you know in our agent we have llm we are not rule based and I showed you here for example here nine deep seek agents here augment a knowledge graph or if we go here with Google Stanford the co scientist we have six agents plus a supervision agent and I talked to you about here memory agent for a crystal lean structure or we go here with a multi-agent graph rack system multi-agent is the Future No discussion about this but you know let's make it simple because it's such a beautiful day today what about the simplest configuration let's take the three just two l well a locate and a solve then we have a self- refine configuration where we have here we generate something we critique it and we refine it and then let's go for a very simple multie agent debate structure where we have here from the generative AI one two three ideas and then there's a short discussion going on and then we have a consensus couldn't be easy you know you know what's amazing look at this if we have now and let's go here and let's take this simply this case a so we just locate here the the query and then we solve it no couldn't be easier so we have here entropic and entropic so the first one of the task is easy this is the simple task so we do it and then we solve it and the Sol is wrong or we take Google and Google no you seeing yeah the Google will work now okay it's Gemini 1.5 Pro the old one but never mind now look here it fails in the first run in the first element but it can solve it but it did not understand the question but you know what's crazy so both answers are wrong if we comine this that we choose now particular llm for the first part and a particular llm for the second part although we have two wrongs this now delivers you the correct answer and this is done by llm selector this is a new method and this is the topic of my video today and I thought this is simple crazy what nobody thought about this before and he would say hey wait a minute if you want to combine two llms that already did provide wrong answers suddenly a multi agent system cannot deduct the correct answer yes absolutely this is it and you might say who had this idea well it's easy Microsoft Stanford University Princeton University and UC Berkeley great but you say hey wait we we don't need this multii agent system come on it's too expensive I don't know what it is I have not watched your video here what is a reward system I know nothing but but we have modern VMS which language mod like Gro 3 no this is it this is all we need I know that you think gr 3 here is the smartest I know but you know what this is just the marketing so let's test Gro 3 on this simple arithmetic expression we just saw in this simplest test in the simp plus configuration and this doesn't even require any form of advanced reasoning or logic this is just CR rep so we have exactly the same question hey what is a particular number plus and then we have an arithmetic expression and we compare simply two numbers and you have here this is it on yeah calculation and the answer is 48 and you look at this and you see that this is the wrong answer it gots everything right except it doesn't know which of those two numbers is greater than the other one so the arithmetic operation is wrong so Gro 3 given the simplest kind of complexity fails already here and now you understand why we cannot use gr 3 and I don't use gr 3 anymore for any physics or chemistry or medical or financial operation we simply cannot solve this task it solves it in the wrong way so we have to choose another llm no problem you have 100,000 I don't know how many llms you have let's go here for 03 mini High you know see absolutely the same yep 49 this is the correct answer great so we take this LM that's all is this is the whole I don't know whatever complexity we just choose the right llm for the right task so we subdivide the complexity into simple task and then we choose for the right model that can solve the task and and I don't care if it's open or I or grog or what the hell I don't care I just want performance and of course price is another element but more about this in 5 minutes time so here we are here we have our spark of Genius so what do we do as a first step yours here Howard and everybody told us here systematically study model selection in a static compound AI system those with the number of the modules the sequencing of mod calls and the mapping between the mods and the models are fixed and in this context they found here as a first idea that allocating different llms to different models really surprisingly lead to substantial higher performance then allocating here the same llm to all the models this is amazing I never thought of this I thought if I go here with the biggest the most expensive llm I put this llm in all my age and in all my models know because it's s hey they have the best performance and you know what it turned out to be wrong I was wrong about this I just have to find the right llm for the right subtask and you know what this right llm might even be cheaper and I can run it locally I don't have to pay here a cloud provider want to give you an example next example here the next most complex here self- refin we have a critique and a refiner step so I thought I have here the same llm let's take gr 3 for example and you do the test and you find it fails so so consider again to self- refine here multi-agent system consisting of three models a generator yeah a critique yep and a refiner great so llm a that we have here maybe better at providing feedback but much worse at generating and refining answers than llmb so this case allocating llm a for the critic and llm B for the other two is better than allocating here one for all the modules so what we just have to do we have to find out which model is best for our task and if you think about a multii agent configuration this could not be so complex now it just goes with factoral okay third example multi-agent debate you notice the agents here debate among them themselves who has the best stud who has I don't know the greatest solution this is called the model selection problem careful this is MSP not mdp so not the mark of decision process but MSP so we decide now I mean the a know from Princeton and everybody we design now an llm selector this is a simple framework that optimizes now the model selection problem for any static compound AI system giv here a particular training budget time money whatever you have and the llm selector now iteratively nominates one module and allocates it to the llm with the best model-wise performance as estimated by another llm and this other llm we call an llm diagnoser beautiful name okay let's go with this so we have two llms and this is their job couldn't be easier let me give you an example because it's always so nice to see an example now we have here a question never mind and then we have here a debate of our AI agents no so we generate here three answers and let's have the case that we generate three answers with gbd4 only the answer is eight and the answer is eight and the answer is eight beautiful temperature zero and then we debate and we have three GPD for om debating here the same answer you what they come up with the wrong answer so it is not a good idea to use the same llm here for all the different agent and all the different mods you know what's much better look gbd4 Omni okay but the next one Gemini 1.5 Pro or Gemini 2.0 pro whatever you like well llama 45p doesn't matter just give it diversity and then for the debate we still have like here the gbd4 Omni so it's the same still stupid gbd4 Omni that has here the deade between itself but now different models provide different outputs and you know what it is able to find the correct answer check isn't this amazing I find it absolutely fascinating yeah for the definitions always what is an llm module a module utilizes an llm to process the inputs typically concatenates all the inputs to a text snippit obtains an llm response to the snippit and sensor respones an output we can now in the mathematical description see this as a graph no why because we have a compound AI system of our multi agents and of course you can see this with vertices here and with edges so we could theoretically have here a graph theoretical a mathematical approach and there's all the notation and the definition but you what I want to show you this simple way I just want to give you the feeling for this I want to explain it to you maybe without mat so this llm diagnoser this secondary llm that judges here the modules let's have here a direct flow through the problem so we have now simplest case no locate solve this is it evaluating the Sol module with CLA 3.5 Sonet I know you love a coding CLA 3.5 Sonet with get an input we have the same fascinating question you know the answer should be 49 and then we have to locate module output and this is simply here this is what is 48 plus and this expression so we correctly extracted let's say from my human prompt here we located here exactly the problem we formulated him and great and now the llm just has to solve it now we come now to the solve module the output of the Sol module if we take CLA 3.5 son the latest one correctly outputs 48 now as the numerical mathematical solution it's course it simply makes a mistake so we have a final output of this system now combined 48 and which is incorrect I know you're not going to believe this so I just went there I said okay CLA 3.5 s in the latest version what is exactly this expression and it's 48 so this is incorrect now what happens the LM diagnosa analyzing this would see that the locate module did the job correctly it took my prompt and Ood exactly here that it has to solve this particular task now to solve module outputs 48 this leads here to an incorrect mathematical verifiable final answer because it should be 49 so now if we have here if you want the diagnoser and let's have a diagnoser that is a little bit more intelligent and Sonet and can solve this it understands that Sonet is the wrong model for the solve module here in this particular case for this particular task so now the llm diagnoser would reasonably charge the following the soul module with claw 3.5 son 20 24 1022 seems to be contributing to an error in this case and now understand that this is a simply ification in a certain sense because yeah this is wrong this is absolutely correct the Judgment but we are just making now a simplification that we say now with our diagnoser we just go for a binary decision correct or incorrect we don't go hey this is 70% correct or something so we just have binary yes no enter don't enter phito great I want to show you the because you might say how can I code this now in a diagnose of promp this is the code here this is the prompt you are you talk here in a system from the error diagnos expert for a compound AI system and yes yes yes you got an idea no problem so let's come to the core of this publication this llm selector is now specifically designed to avoid trying out all the possible configuration in a multi-agent system or any compound system and it uses here this particular if you want this particular solver this particular diagnoser for this task so this is if you want to trick to this model because we have here a system architecture that could be like this for our multi AI agents and now we have to find the right model for the right subtask that is defined in the Consortium in the interplay the interconnects here of the complete AI system and this would be get a perfect optimized model allocation to our specific subtasks now if you want to know a little bit more let's just stick with me for 30 second if not skip ahead for 30 seconds how this is done in detail let's have a closer look so we have our llm selector and what it does is a model wise iterative optimization so instead now of considering all possible combinations simultaneous our selector breaks down the problem into a series of sequential module by module optimization steps so this depends on the quality of the llm you use here for the selector so I would recommend going here for let's say mathematics or your domain knowledge for the best model in your domain so the llm selector focusing on optimizing one module at a time keeping the model allocation for other models temporarily fixed so simple if you want estimation everything is fixed and we have just one degree of freedom for this particular mode module then we have a guided search in the search space then we have within the selector framework the llm diagnoser and it does not empirically test every possible configurational whatever we're running here full end to end system evulation because it would be too costly if you have multiple systems so instead and if you want this is to trick got a beautiful thing here it uses the llm diagnoser to estimate the model wise performance and have shown you this on the example already in mathematics is easy because it can calculate if this is right or wrong in code you can have it here simply do you run the code and is it working or not if you have a little bit more causal reasoning it might not be that simple so interesting but this llm diagnoser estimates it a model wise performance of different mes and this estimation is much cheaper than he the full system evaluation if you want you can do the full system evaluation but all the aors here tell us hey it is maybe not really necessary to spend this much money because we have here this directed search and if you remember my other one about the reward system here to optimize here in the TTS and the test time scaling here we had the reward model and now if you want the diagnosa is kind of our guiding star here in the search space so the llm selector uses now the diagnoser estimation to guide its search in this open search space selects the model yes directed search strategy beautiful intelligence yes and then of course you have a limited number of evaluations depending how much money you want to spend at the time you want to wait let's say you have K models and yeah but remember it does not try all theoretical possible combination of all model for all modules so there is a tiny but exist a certain inaccuracy because it's not really testing out all the knowledge but here we depend if you want on the intelligence here of the diagnoser of course we can calculate this for known models and here you see it for our three cases here in this two this is the simplest case for locate solve here we have our fine with our three step and here in this two we have here the debate structure so you notice and here we have all the models whenever you see this please fill in your models that you have I give you the code so you can have this immediately for all the different Benchmark what is interesting and I just coded here DSP you know if you want this uh next step after code uh engineering code optimization prompt optimization and then text grad you know it is coming it is performing better than DSP it would be interesting to see Tex gr here but this llm selector has really an outstanding performance if you compare it here with all the different benchmarks so interesting I definitely will use this and I'll give you the code in a minute so there we are we are now here with the facts and you have here this beautiful publication Microsoft Stanford princeon and Berkeley February 2025 and they have here this idea here that they say you know what we should have here optimization to select the best llm for the best subtask great idea absolutely amazing so if we apply this now for our own what we do first you define your set ofici language mod you want to integrate you want to compare for your multi a agent configuration give your specific task so maybe you define your local LMS that you have on your machine or whatever you are in your uh particular Cloud as your preferred set of AI agents or maybe you sort your preferred llm according to the cost to the pricing here for cloud interference in your region so however you define your set here of llms or VMS this is up to you and then you simply run this code that is here given here to us by the Consortium and beautiful so just days ago it was optimized this is great we have an Apache 2 license so we can use this and they really give you beautiful examples look quick start no API key needed you have for example the life codebench data set you can evaluate here the first refine system using here the fixed mods the expected outcome then for those three is here you have now a numerical score given and if you run now the llm selector to optimize this system this is just the five line six lines of code and then you see yes we 95% is much better than 86 89 or 86% so this shows you here a performance gain you can have with I don't know 5 minute of coding of course you can customize this and I was asked to talk a little bit more about code I thought if I give you the GitHub it it's enough okay no here we show you customize system and task API key needed so if you have either don't know anthropic or to together EI or wherever you are or you pay for gini or you pay for open wherever you have your keys great you just have it look this is the way to go they have the pen files for you they have the Jupiter not books everything is there and they even so kind imagine they say hey can I request the feat and they ask you yes we are happy to hear from you please feel free to open an issue for any feature request we will also be happy to coordinate the ongoing effort please send an email it is now open available 21st of February 2025 the code best you have more examples here in the GitHub you have Demos in the GitHub up have a look yourself so I think some great work beautiful coming to an end yes try it out I definitely will and I already started I think it's so amazing now if you think hey yeah we go here for hour and it is only about is this Gro 3 or is this I don't know uh open ey 05 or whatever is coming up I don't know it's not about the single model it is about an intelligent combination in a multi my agent system configuration and you see with a little bit of a mathematical optimization like llm selector you can reach 100% I think this is simply amazing so we do not depend here on single LMS or single real lamps I hope you enjoyed it and maybe you decide to subscribe then I see you in my next video