Multi Agent Framework: A Comprehensive Crash Course | Community Webinar

Key Takeaways

for the introduction. Hi everyone, I welcome you all to this webinar and without further ado, let's get started and jump straight to the agenda for today's topic. So we'll be discussing multi- aent frameworks a comprehensive crash course. So let's get straight to the agenda. We have we'll be discussing what is an agentic behavior, why do we need multiple agents and use cases of multiple agent workflows and then we're going to discuss how to build multi- aent workflows. And at the end of this theory session, we're going to do the hands-on exercise where we'll create a supervisor of multiple agent and then it's going to route it to the relevant agent and then get the response from it. So I remember from the last year on the same month I was teaching how to build uh chat bots like chat GPD that can generate responses based on the text or based on the question you are giving. But now as the uh as this LLM ecosystem has been uh developed and it has uh widened its use cases now we we are at this platform where we have to create an agent behavior. So a a food for thought for you is consider you planning your travel or you're planning u a travel to another country. So consider an agent that can actually uh book your hotels uh book your calendar meetings send out the invites and then also set up your uh restaurants uh meetup as well. So, an agent that can have multiple scenarios catered out for you just like you have your personal assistant or you have uh a legal assistant or you have an advisor that can actually uh work your way through legal issues. So, that's how an agentic behavior looks like in an in concept of LLM based applications. it actually uh has this idea of acting autonomously and making decisions on their own and performing task with minimal human intervention. So as I mentioned before uh you can think of a hundreds of such tasks that you can automate with an agentic behavior and that's the beauty of AI that you can actually avail all of these with just one click and one fire of the supervisor and it's going to uh do the reasoning for you and it's going it's going to do the decision making for you all at once. So in this theoretical part we'll be discussing what are some of the use cases of multi- aent workflows how we can have a router agent how we can have a supervisor agent that can have multiple hands of different agents and then iterate it as the need occurs. So let's move forward over why do we need multiple agents. So consider you are leading a team of 12 different people. One is from the marketing team, one is from the sales team, one one person is from the uh HR or legal team. So these uh different domains can only be can only be worked efficiently when you have different owners of this domain. So in order to understand a multi- aent scenario, you must think of it like you you are working with an organization that have all of these different domains and then you will be picking one point of contact from all of this domain and that point of contact will do all of the chores related to that domain. So if I draw something really quick on this slide deck. So it would look like something this like you have an organization a CEO over here and it has all of this different domains. You have a sales team, you have an HR team, you have a marketing team and then you have a legal team. So a multi- aent workflow can actually help the CEO as a personal assistant whether he wants to ask what was the sales revenue. So if the question was what's what was the sales revenue this this personal assistant agent would actually route this query towards the sales agent and if if this CEO ask uh if they are going through if they are going through a set of uh HR policies or if they are if the CEO wants to ask about some HR policies he he can just this this personal assistant can just route the query towards the HR agent. So consider all of these different agents as separate logical units and then this personal assistant would actually route your specific query towards the domain you have you are being asking you are asking the question from and that's how just one supervisor can actually skyrocket your efficiency and productivity and each agents get to focus on their area their expertise which actually helps in minimizing the bottlenecks and maximizing the output. So this is the new era of AI and LLMs that people are moving forward towards and this is how uh this superficial intelligence and the AI would actually rule the world with multi- aent scenarios and then they will have multiple skill sets that they can actually acquire as they are learning their way forward towards that. So here is an example of what I have previously discussed. Let's take an example. If you just have a single agent uh system where it is just used for writing the content for you. So now most of the people critic about how do we know that the responses AI is generating is good enough or maybe they are not hallucinating. So then we can also have an agent that actually judges the response as output. So I have been in a conversation with a number of people uh as I meet them, they always ask how do I rely on the responses from AI? How can I be assured that this response is 100% true? So in order to answer to those questions, let's say we have a critic agent as well. We can have an LLM. There's a concept of having LLM as a judge. So whatever the response we will get from this agent. Let's say I have this agent one. So I I ask him to write the content for me. And on the other hand, I can als on once I get the response from this agent, I can still pass this response to a critic agent or the judge or the panelist that's that is going to actually rate my response whether it is four out of five, whether it is three out of five. And based on that rating, I'm going to pass this to an another agent to minim to incorporate the feedback. So I'm going to collect the feedback from this judge or critic agent. So I'm going to so it it will look like a closed loop where you are iterating over the feedback and once you once you are once you have the accuracy of 95% or 98% then you would call it that this response has been iterated over and over again we have incorporated the feedback from the judge or the critic agent and that's how you actually progressed in this era where you actually try to reduce the hallucination of these agents. So that's why we really need a mult multi- aent scenarios where we set up those agents so that it can put our evaluation it can put our guard rails to a certain extent where we don't have to be on the back foot that whether this agent would be a good fit for the response or would be a good fit in healthcare or not. So there are a multiple scenarios that we can think of while we are ensuring the compounding effect in multi- aent scenarios. So as we move forward there are multiple use cases that we can uh utilize while we are working with multi- aent scenarios. As I mentioned earlier, let's take uh this as an example of CEO of the company and this router is the personal assistant to that CEO. And then we can check and then we can uh check this agent as I'm going to call it as legal agent. I'm going to call it as uh marketing agent. I'm going to call it as sales agent. And then this is the final response that I'm going to get. So if the CEO asks a question that is what's the revenue of from the last year or what's the revenue from last month last month revenue? So to my intuition I would know that this uh question should be asked from a personal uh from the sales team. So a personal assistant of this CEO would actually route this query towards the sales agent. And if if I ask what was the what was this what was our spending limit of on Facebook ad? Uh how how much amount of uh dollars we have invested? Uh what was our investing investment? Uh what is our investment on Facebook ads? Investment on Facebook ad last months. So this query would be routed towards the marketing agent and these agents would have a set of their own prompts would have a set of their own knowledge base and then they are connected to their own ecosystem. So consider them as a different logical unit that have their domain expertise. They they have their they are trained on the data that we have uh given to the saleserson or given to the marketing person or given to the legal person. And then we can ask a similar question that this personal assistant or you can call it as a router that is going to route the towards the agent given the set of ask given the set of questions that have been asked by the CEO. So this was one set of example why do we really need multi- aent scenarios and how we can actually incorporate it in our both in our daily lives and how we can actually productionize this for the use of our customers. So as we move forward we have we also have an example of a consolidator workflow. Consider consolidator as somewhat uh let's take another example. Let's say you have three different agents and let's take the similar example. You have an agent from from your HR domain. You have an agent from your sales domain. You have an agent from legal domain. And what if the CEO asked what was the commulative uh what was the commulative uh investment we did in our sales team with our legal team with our HR team. So these this consolidator would actually get the response from agent one, get the response from agent two, get the response from agent three and then it is going to consolidate all three of these responses, all three of these responses and then consolidate all of them and then you get the final answer. So this is how a consolidator workflow also works in in a setting of production environment and how you're moving forward with that. Now take another example of a sequential workflow. So in this sequential workflow, you actually want your agent to perform certain task after they have completed the first task. Let's say I'm going to ask it uh let's say my question is this agent is for web search. It is going to do the web search for you and it is going to uh write a code for you. write code and also execute it. And this is for writing a detailed report and then all three of them would be consolidated. After that you will get the final response. All right. So for me if I if I read a couple of questions. So let's think about a scenario uh how this consolidator workflow would work in sequence. So let's say if I am working in a medical institution. Let's say I'm a radiologist. So for me I want to search something over the internet. What was the what are the basic principles of of of writing an MRI report or what are the basic principles of uh going through a city scan? Uh you can ask such similar questions that you can get the information from doing a web search. So considering that I am from a medical uh domain, I'm going to ask this tool to actually search the best practices the best practices the best practices for for writing for writing a detailed MRI report. So this would be the task one of this agent. So this would be executed by this agent one. Then I'm going to ask this agent to to calculate or generate some bar charts or plots for this given this given the data I have searched over the internet. So I'm going to ask it uh calculate The calculate the the increase in salary. The increase in salary over the past few years for radiologist for a radiologist. Radiologist over the past five years. Five years. and draw a bar chart for it. And this is what we are going to do in the hands-on exercise as well. And then this would be the task two of this agent. And then the task three would be considering the plot and the graphs and all of these examples. Generate a detailed report. Generate a detailed report for my assignment. And this would be done in a form of chain. You can also call it a chain of thought. So these agents are called react agents which actually works their way in such a way that it first reason it act according to it. So they are called react agents. they reason it g given this set of tasks and then they act accordingly. So this was one of an example of how uh these uh consolidator workflows works. Now you must you must been you must have been arguing it what if I want to skip some of these what if I don't really utilize all of them. So then you can also have some conditional edges where you don't want to actually incorporate all of them in your working environment. Let's say you want to skip the legal agent for quite some time. Let's let's say you want to skip the sales agent. So this uh this back and forth of agent conversation is what we call multi- aent workflows. And please feel free to drop uh the questions in the chat. I'm reading it one by one and at the end of this session I'm going to respond to all of your questions. Uh just have some patience. All right. So let's move forward towards how we are going to build an agent workflow. How we are going to build a multi- aent scenario and what are some components of building such agents. So in order to build in a multi- aent workflow we need to have we just need to remember three things. First is the nodes. Nodes are basically your legal agent, your sales agent. So nodes are just the Python logic that actually drives your agent. So this is just the logic unit that actually drives your way towards it. Then you have edges. Edges are there just to connect the nodes together. Let's say you talk uh you are let's say you have these edges where you actually connect all of these nodes together and then you can sometimes you can also have conditional edges which are your decision m makers that you have to actually uh enable the decision making within the graph and then these three principles are more than enough for actually building an hands on uh building a multi agent workflow. So in today's session, we are going to build a similar supervisor agent and I'm going to share the notebook with you guys after this uh after this entire webinar as well. So you guys can go ahead and try it and then tweak it as per your u requirement and then you can move forward with how you actually want it to be. So this would be a user you can call it as uh any customer if you are trying to fulfill any request you can uh call it. So there was one one scenario we have been working on for quite some time. Let's say uh this is this supervisor agent is is build up for an healthcare industry. So think of some uh agents that a person who has been diagnosed with some issues. Uh let's say a person is a uh is a cancer patient. So think of some agents that this cancer patient actually need. uh think of some doctors, think of some consulting appointments that this cancer patient really want so that he can actually uh move forward in his life with certainity. So the agent one would be a doctor that knows all of the all of his history, a doctor that has all of his medical records. And then the second doctor would be uh his nutritionist or anyone who who has all the knowledge of his past history of his past medical records. Uh so this could be his personal nutritionist and the third agent would be uh anyone who would be there for his paliative care. So now as a cancer patient if I ask about some questions related to uh uh related to what was what was the summary of my last city scan. So this supervisor uh last city scan summary. So this supervisor going to route this question towards this agent one. What if I ask uh what vitamins should I consider while I'm taking this tablet? What vitamins uh should I take when I'm on these tablets? when I am taking XYZ tablets. So this question would be routed towards the second agent towards the nutritionist who have the complete understanding of what are the uh tablets uh this person is taking and how he can perfectly align it towards the nutritionist. And then the third question would be if if your uh cancer patient is uh is having some issues with his health this if if he's having any health issues it is going to directed towards the paliative care uh person. So consider all of these agents. They have their own personalities. They have their own skills. They have their own personas. They have been built meticulously based on the uh agentic information we have built on it. And having said that, then you build a supervisor or a router agent that is actually going to route these queries towards a certain ideas. Or think of an idea that it needs a consolidator. Sometimes and the consolidator would look like something that this agent would be directed towards uh from this after getting this medical after getting this medical records it is going to send all of this information to the nutritionist send information and based on this information I'm going to get the final answer. So this is the context I'm sending it towards the nutritionist and having said that this would be a this would be a consolidator workflow. It would look something similar to what we have learned here that this is your nutritionist agent or or you have your medical records over here and then you pass those medical records to the third agent to actually give you some ideas about what medical uh what is your medical history and then once you have your medical records and then you can pass it towards is the third agent which is going to which will be your nutritionist and it is going to advise you based on that. So it is going to give you advice uh on what vitamins you should take. So that's the overall uh theoretical part of it. Why do we really need workflows? What are router workflows? got. And now we're going to move towards the hands-on exercise. And as I've said before, I'm going to give you guys the access to the slide decks and give you guys the access to the notebooks so that you guys can try it on your own. So without further ado, I'm going to take you guys towards the multi- aent supervisor I have created. And in this agent scenario, uh first and foremost, we have we'll be going through it line by line. But you don't have to get overwhelmed with a number of with a number of technical jargon I'm going to use. You don't have to get overwhelmed. You will have the notebook. You can just install the dependencies and then try it on your own. So let's get started with our hands-on exercise. So in the first uh cell of uh cell in this code cell we are installing all the required packages. So it requires langraph, lang chain and all and lang and pandas libraries so that they can smoothly run towards it. So in this working session we'll be creating a supervisor agent which will have access to two different agents and based on our portion it is going to route uh to a specific agent. All right so as you mentioned we have uh installed uh our library. So the warnings you can neglect them you don't have to worry about it just the version of pip that actually gets uh updated again and again. So you don't have to worry about it. Uh and then you set uh the environment variables. So we'll be using teley search. Uh we'll be using teley search uh for our web search tool and we'll be using open AI API key for uh getting the responses and using GPD40. So moving forward we'll see uh how we going to create tools for this agent, how we are going to set up the nodes, how we are going to set up the state for this agent and then we're going to move forward uh to this. All right. So before I go into this into the intricate details of this tool, let me actually have a slide deck uh a plain slide deck so that I can explain each block cell again and again. Let me copy and paste all of them here and then we can have a discussion on it. Okay, just give me a minute. I'm going to create a new slide. uh take the screenshot of this because this is the way of I like to teach instead of running over the code. I would love to explain it using the screenshots so that it is easier for you guys to actually absorb all of this. Okay. All right. So, I'm going to go back to sharing the screen. All right. So as I mentioned earlier that this supervisor you can call this them as agent or you can call them as tools. So let me actually draw this draw the graph again. So we will have a supervisor agent and this supervisor agent will have access to tools. This would be a web search tool. And this would be a uh this would be a uh tool called Pythonripple which is actually which helps you in writing the code and then executing it and executing it and based on the and based on the question that has been given to this tool. tool it is actually going to route to a specific agent. So if I have to draw it again so based on the question based on the question I'm going to give it this to this supervisor would actually detect the intent of the question it detects the intent of this question and then route it either to the web search tool or the python grapple tool. So first and foremost is we are creating the we are creating the tools the tely search tool and the python ripple tool. It just we imported from the library uh these functions and then we just initiate these functions so that they can actually work work uh their way towards it. So this is one set of thing that we have initiated the tools. So this is the code cell that actually helps you in initiating the tool and as we move forward now you see we are creating a supervisor and we are passing a prompt to this. This is the prompt for this supervisor that you are a super supervisor that actually manage the conversation between uh the following workers and it actually respond and the respond and take the task based with their results and status and then it actually perform those things in a sequential order. So these are the prompt that we have already written to our supervisor and it is going to adhere to these things and then they they're going to move forward from there. So this is just the gist of it and we're we're going to see this uh we're going to run this cell of block and see how we are moving forward with it. All right. So up till now we have discussed how do we create tools. So I'm going to run this and then we have helper utilities. It is just uh the message uh block that we actually seted uh for our agent node. That is one set of thing for us. And then we have agent supervisor. And this agent supervisor has this system prompt. And based on that it also comes with a number of c it also comes with a prompt template where you actually give this prompt to to your specific tool that given the conversation about who should act next and then you give the options for this tool whether you want to have the option for a researcher whether you want the option for the coder. So these are the names we have named it. We have named it as researcher and this is as coder. So let's move forward towards the next block of cell. And please feel free to drop questions if you are facing any difficulty and if you if you didn't understand any of this line of code uh we would be happy to help. uh just uh drop the questions and then we're going to respond to it in next 5 to 10 minutes. So once we run this cell uh we have initialized the super uh supervisor agent. Now we'll create a graph of this agent. As I mentioned earlier we have these uh specific uh elements uh separately constitute. We have the tools, we have the supervisor. Now we'll create a graph of it. And as I mentioned earlier to create a graph you have to mention you have to uh bring the you have to mention the nodes and the conditional edges as we mentioned earlier. So this would be our node one. This would be our node one. This would be another node. And then we will we will have the conditional edges and and so consider you have these logical units created. Now you are spacing them or mapping that them out uh in a form of graph representation. So this is what we are going to do in this block uh block of code. We are initiating we are uh importing all of the required libraries. We are initiating a class and based on this class we are going to set up the nodes of this uh workflow. So we have a researcher node, we have a coder node, we have a supervisor node and based on this node, we are setting it up our we are mapping out our graph on this and once we have all of those nodes ready uh if I go back so this would be another node of supervisor. So now as I mentioned earlier we uh in order to make a graph what do we want? We want a node conditional edges or just you can call them just edges. All right. So if I go back to the code, I would see that after we have uh set up the nodes, we will create edges in the graph and these edges would be the decision makers whether should I go towards uh the coder, whether should I go towards the researcher etc etc. So now we will initiate and compile this graph and based on that uh all right so it says that the supervisor already exists so you don't have to actually rerun it again and feel free to drop the questions in the chat um uh I'm reading out the questions um from um we have a question from Mike we have a question from Robert so uh this the notebook link would be provided to you George uh don't worry about it we'll share it in at the end of the session and we can take the questions as of now once we once we have constructed this graph. So if I go back to the slide, you can see that once we have these nodes ready, all we need to do is set up the conditional edges. And these edges would be what what you actually want uh from this tool to actually work. All right. So we have a question from George and I really want to answer it as a uh right away. How deep should the context and information that is given to a specialized agent so that he can reliably carry out his task in that domain? Uh George the context to this agent u you should actually pass it 100 lines of prompt or 200 lines of prompt but the the the gist of it or the recipe of work of a good working agent is the LLM you are using behind it. So if you are using a GPT uh 3.5 versus a GPD4 it's going to uh it's going to make a huge difference behind it. So yes I agree with you the domain knowledge is uh really important but it also important what what is the LLM you are using behind this behind this agent. I hope that answers your question. Um all right so Mela has asked is there an easy way to generate a uh a UML diagram from the langraph code yes uh there are uh there are a number of startups that have that have just solely that are solely focusing on creating multi-agent scenarios. Uh there are some uh applications that don't really rely coding. So there are some applications that are no code multiple agents. So if you go to the internet and ask how you create an agents without code you will find all of the names to that and the question from George again that whether it's not the case to have a uh to have a better understanding to have a rag agent. Yes, absolutely. On in the third case, you can have a rag agent. Rag which is also called retrieval augmented generation uh is also something you can have uh in as a third agent which actually gets the data from your knowledge base and based on that knowledge base you actually generate the responses. So it is something that we also are working on. We we have a multiple agent scenario uh where we actually set up the web search agent the uh the python ripple agent the rag agent both all of them all of them together. So now as we move forward towards uh this in in uh how we invoke this. So see this question I have asked from this agent. Write the write the detailed complete code for HTML, CSS, JavaScript for a simple port uh port portfolio for James. The website should have a sections for about me project skills etc etc. So right now I'm asking this question for this agent to generate a response to generate uh a detailed complete code for HTML CSS. And once we run this cell uh this code cell it is going to give us the response that uh the supervisor has started uh the coder which is exactly correct and then it has also gave us the response uh of this that the code to this portfolio website is here and you can just copy and paste this portfolio website code in index.html and you will have your portfolio ID. So there is an a coder agent which actually writes the code for you. And then you you also have an researcher agent. So I have asked this researcher agent to write a detailed research report on radiology. And it has a recursion limit of 100. So it's going to iterate it over and over again. And you can see that this supervisor has routed this query towards the researcher agent. and you get the report in a structured format. This is the introduction to radiology. This is the history of radiology. Uh types of imaging techniques, you have x-rays, you have ultrasounds, etc., etc. So there is a wide v variety of uh use cases that you can utilize if you are working in a healthcare domain, if you're working in a sales domain, you name it and you can build a multi- aent scenario for that. So that's the entire uh walk through of the theory and practical part of it. But we still have some time left and we'll be answering all of your questions directly. So please feel free to u ask questions if you have any. All right. So, I'm going to scroll up the chat and going to take the questions from the start. All right. All right. So, here is this question uh of how would you how would this multi- aent works in an healthcare environment. So, I'm going to address that. I I have already addressed this before that consider you you have an you have a patient that that has been diagnosed with uh with cancer. So having said that uh this cancer patient would require uh consultation from the doctor that already knows all of his medical records the nutritionist and the palative care unit. So having said that any question that occurs uh on the fly towards the supervisor it's going to route it towards uh the specific agent whether it's about his medical records whether it's about the nutritionist whether it's about the paliative care all right so are there any other questions so far George is saying you can also have an oncologist. Uh yes. And you can also have another agent that would be the legal advisor for your cancer patient so that they can get uh the insurance benefit for them. This would be a legal advisor. All right. Can I use OS LLMs like llama outside of lang chain to build this type of agentic framework? Uh sampri absolutely you can use LLMs uh that are deployed on your own plat that are deployed deployed on your own cloud platforms or on premise you can absolutely use all of them. It's just the point of calling an LLM and then just passing it the right instructions. It is going to work as uh per your requirement. All right. So there is uh this question for from David Taran uh and then there is this question from uh George again. Can you please talk about optimizing cost and token consumption when it when it comes to multi-agent system? So I'm going to address both of them together. Let me actually have uh another another white screen. So let's take this question first from David. Uh so uh David is asking whether the chaining creates separate round trips to an LLM. So David is asking how the chaining works and how does it work in an in an LLM setting if we have a agentric behavior. So David, if I remove all of this again and I come to this example again, let me remove it. Have my razor. Okay. All right. So chaining with multiple agents somewhat look like this. It's a sequential chain that have uh that have all of these agent set up. So for chaining we really need to have a basic principle of uh the chain of thoughts. So chain of thought is something uh we have to actually understand before we move forward from here. So consider chaining in multiple round trips of of coming to the same agent which means uh you can have a web search tool over here that is going to crawl the internet and going to get the responses from Google or Bing or Doug go or table search and having said that this web search agent uh is one of a kind of chain that is going that you are going to to utilize and then this web search uh let's say if I want to ask uh about uh the stock market that about the x uh yz stocks uh rate from the past rates for from the past five years and getting this response from this I'm going to ask this agent to generate a generate plots for me. So it is going to write the code and generate the plots for me. And once you have this chain executed, now you will ask it to write a detailed report based on the plots plus uh the plots plus the data we got from the internet. And once you have the detailed report, you're going to ask the consolidator to generate a final response for it. So that's how chaining works in and they are also called react agents which means they uh they reason on a certain idea and then act according to it. All right. So we have two more questions. Can these agents use different model for their processing? Absolutely. So if I if I really believe uh GPT40 or any other model is good in certain ideas, I'm going to pick that up and I'm going to uh let me actually draw it again. Uh so I can call it as this this this agent can use llama 2 or llama 3. this uh plots for plots. I always admire agent from Claude. You can use any agent uh from u for generating plots and I'm going to say that the detailed uh report writer would be my GPD40 and this could be any agent from claudic. So yes uh all of these agents can use uh different models for processing. Uh are we going to have a similar session on multimodel as as well? Uh so for multimodel scenarios you you will have multiple models that that can generate images that can generate videos that can generate text and then setting up all of them together. we have a certain thing in in one of our project we we have been building that so it actually looks like something similar if I have the pen ready so a multimodel scenario is when you have access to both configurations whether it is generating a voice uh whether it is generating voice or generating images or generating videos. So these are some capabilities that you can uh avail using multimodel uh workflows. So there is this question from John. Do you recommend that that we create a system where an agent can actually create other agents as needed or should the agent should should be established and consistent? That's a really interesting question. So John, you are saying that we just create a supervisor and it is going to work its way on its own whether it it it wants to use a certain agent or not. So instead of that think of developing 100 skills for your agent and then on the fly that supervisor would actually detect the intent of the query and out of from that 100 skills they picked one skill and then do the iteration on it and then do the uh the end to end work on it. So I would rather recommend having 100 skills and then letting the supervisor pick one from it or just building in super or just building a supervisor that can create agents of its own. It would be somewhat dark because what if it creates an agent that is going to hack few website over the internet because that would be something out of control. So that is what u a big organizations and companies are targeting over uh that super intelligence in AI that the agent would actually create uh minor agents on their own and then they are going to do their task and once the task is accomplished they're going to destroy those agents but what if they never destroyed them so that is a food for thought for you Uh so Sean is saying why not have the 100 plus agent debate and decide who is the best to take the task or even collaborate between each other for different functional roles. So so it looks like we are going towards more or like bureaucracy where we are thinking about what to do and what not to do. But I would rather recommend you guys to actually start with the basic principles. Start with creating two to five or 10 agents then discuss how how do you really want to move forward with it? Okay. So there is another question uh do you have examples of real companies that have productionized a multi- aent platform? what are some of the state-of-the-art software out there in the world. All right. So, in uh so there are a number of companies who are you must have heard about Creo AI. They have their autogen uh which actually do all of the work for you and then there are a multiple such uh companies that are building on top of it. So you can go over the internet and search uh about them. We are also working on multiple agent scenarios for our for some of our clients. And then uh in our boot camp we do uh teach a lot about how to create such workflows and how to create such uh softwares from the scratch to build multi- aent workflows using open-source uh libraries and uh repositories. All right. So, any other questions so far? Anyone? I think Okay, you have one more question. Okay, let's take the last question. How are you evaluating the quality and correctness of individual components? Router correctness, agent task, final response accuracy, human feedback. That's a really good question and uh it is from Rich Young. So Rich uh there are a number of companies that are build on just evaluating these LLMs and multi- aent scenarios. So what we can do so far is on every LLM call on every wherever you wherever you're using an LLM call you can just directly when once you get the response you can just directly pass it to your evaluator and based on the evaluation you get it whether it is 90 or 95% accurate based on that value you are going to give the feedback to agent and pass it towards the pipeline. If the accuracy was 70% or 60% it is going to withhold the task there and not allow you to move forward from there. So there are some other examples you can take of how whether wherever you just call an open open AI LLM or just uh call it uh uh to actually execute a task you can always have an evaluator note there. All right guys, uh it looks like uh we are uh we have our time and then I see there are no other questions left. I would.