Understanding Multi-Agent Systems in Generative AI using AutoGen

Key Takeaways

Uh welcome everyone. Uh it's great to see a good number here and uh thanks for taking out time uh to discuss uh the multi- aents uh together. Uh so uh maybe what I can start with I can start with a quick uh uh introduction about agents and uh then we will follow uh uh uh we'll follow a list of uh topics that we'll be talking about. So we'll be talking about what are agents, what are multi- aents and how they are useful and why they are needed kind of a uh discussions that we will do and there's a lot of confusion around agents versus assistants versus co-pilots and there's so many terms. So we'll try to understand what is the difference between these different terms and uh then we will also quickly discuss about different frameworks available. So again there are so many frameworks available in the market. we will try to uh just understand uh some of them on a higher level and then uh focus on autogen for this particular session. Uh then I will talk about autogen uh what is the uh autogen library where is and what is the good thing about autogen and I have major experiences using autogen and uh langraph as of now. So that is what I can talk about and uh then we talk about what are different agents present in autogen and then some of the agentic patterns because uh agents can interact with each other in a way different way. So we can talk about some of the patterns which are standard but this can again be uh done in a very uh uh different form based on the requirement based on the scenario based on the use case that we have. And in the end we we will just talk about uh a case where we use multi- aent setup to write a blog about a particular topic. So that is something that we will look into in the end. So I think uh this has been already mentioned. So if you have any questions feel free to put it in the Q&A. I'll be monitoring that uh very frequently and also chat but Q&A would be more preferred because that is the window that I'll be looking at for sure. All right. And uh okay so let's start with this now. Right now what we are seeing um I think uh I'm just trying to go a little bit from the history uh so in the past few years we have seen uh using mult machine learning a lot uh in a different ways we are doing a forecasting predictions uh classifications and all those kind of uh algorithms and uh NLP uh LLMs or even u uh what we say the native AI as a as a concept is not a very new concept is something that we have matured with the transformer architecture. So overall so the there's a lot of uh usage that has come up since LLMs have come up in a way that they are more consumable in the format. So I'm not going into that but that is what I'm assuming that we all have some understanding about. So large language models is majorly the the kind of model which actually is in the back uh bone of chat GBD cloud and perplexity kind of a tools where you can ask your questions prompts pass your instructions and accordingly get the response out of that right so so the agents becomes uh little bit more extension to what we have as um chat GPD uh and different LLMs and the tools that we have right so using these kind of LLMs and NLP and these kind of technologies we were able to uh uh create some of the models even create some of the chatbots that we have uh what we are trying to do with agents uh is to uh make it more auto autonomous. So uh the agents are actually the kind of a uh you can say entity which can take some decisions based on uh some of the understanding that is coming from LLMs and another way is to uh use human in the loop so that human also is being uh taken as a as a one of the u participant to that agent and then agents can work around it automat automatically right so let let's take one example and then talk about agents and multi- aents here so uh let's Okay. Uh there's a there's a manager who wants to do a uh quick understanding or analysis on let's say um the impact of uh uh let's say uh weather on the traffic. All right. So and uh so for that there must be some smaller tasks needs to be done by his team or her team. Right? So uh for this analysis we need uh data extraction to be done. So that becomes one of our uh work module. Another could be to uh take that data and do the analysis. All right. Third could be uh look at some other factors that impact weather and there could be uh a weather API that we might be hitting to get the weather data. So these are the smaller tasks that we have. Uh so with this uh this each individual task can be done by a particular agent which specialize into a particular task. It can be a LLM based. It can be a uh uh what we say uh any technology based. It could be a function. It could be a tool. That is what uh agents give us a flexibility on. But this agent itself is a module you can say is more specialized on a particular task. So especially for let's say just get the weather data based on the requirement. the that is how these individual agents are designed and then uh combine the interaction of these agents uh we can create the final analysis. So uh you can just imagine one manager as the the orchestrator kind of a agent which is kind of talking to each and every agent getting all the information and then finally collating it to a final information which can be reused or used by the by the end uh user. So that is how uh that is how we are trying to uh use multi- aents. Now these agents have a lot of capability uh not just using the LLMs. So they can be uh uh a regular automation bots also. So uh that is the reason they are being adopted a lot in the industry because they are not just uh uh u the automation they are not just the LLM cause they're a combination of both right so uh we can make this orchestrator more smarter uh with the with the use of LLM so that it can take uh some of the decision based on the prompt and the the workflow that we design or provide it to the orchestrator and then automatically uh use or use the agents uh based on the requirement. All right. So here we are just talked about an example where um we have talking about just analyzing the impact of weather on the on a traffic. It's a very specific problem that we talked about but let's take a very generic information. So there's a CEO of a company and they have different departments. They have HR department, they have finance department, they have uh technology department all these department sales and all these different we have. So just assume on a very high level. So all the department have uh one agent and uh the CEO becomes the master agent and they are now C is asking about so now if the CEO has to look at how my sales are going on in this particular geography for these products right so it has to only talk to the agents which are into the product and the sales maybe right so I'm just giving a very vague example here but this is how we can actually understand that CEO agent is making a call. Okay, I need to talk to these agents to solve my query that I have got. Now another query could be around u the uh let's say attrition rates and impacts of attrition rate on my deliverables in the technology domain. So now they have to again talk to the technology team, the HR team and the the maybe the revenue team as well. So all these three agents can represent their team and they're specializing on a special data and internally it might not always be the LM. So that is the the myth that I wanted to break here. So agents um are getting smarter and useful when we are using LLMs in terms of taking the decisions calling and everything. But uh it it is not mandatory that agents that we are talking about here is always be a LLM based agent. It could be a uh a Python function calling, it could be another function calling, it could be a API calling and there are so many tools already available. So that is what we call as tool calling. So that is where we we have a lot of usage of agent. So uh multi-agent setup I I generally define personally as the uh the autonomous uh uh modules which can work on their own and uh decide the interaction on its own instead of hard coding the interaction or defining the workflow. uh it can do it on your own uh on their own based on uh the the the role that we provide to each and every agent and the overall uh final goal that we define to the agents accordingly they can do all these different task and give you the final output right so that is how we define uh I personally define multi- aents in my mind right okay so now uh I would like to move ahead uh so that is where we need to talk about uh some of the terms So we talked about just the agents right? So we have agents, assistants and we are hearing copilots and there are so many other terms as well. So uh when we say agents again I just quickly uh define it in a more structured format. So I would call them as autonomous systems designed to perform task independently based on instructions or environmental cues or it could be by some of the events that we define. Right? So this is how we define a agents which are more autonomous in the terms of the action the they are more independent uh on the type of action they wanted to take the different kind of decision they wanted to take assistants are majorly the the they are more for what we say reactive or conversational kind of a support so let's say I have uh created a chatbot which can talk to my uh different documents internally and give me the final answers right so that become more like assistant. This is just kind of supporting you in providing the information, answer to your questions based on your uh what we say prompts or instructions that you are giving. Right? So they are majorly reactive. They just give you uh react on the uh questions that you are asking or the prompts. Right? Now co-pilots uh I believe it's it's level up on the assistant where we are asking the questions as well as asking uh to do some of the task uh for for us or it could be specialized for a particular task helping you on your day-to-day task and giving you assistant on that with a with some actionability. Uh the good example here would be the GitHub copilot. So what it does is whenever you are writing a copilot GitHub copilot stands right uh to your uh IDE when you're writing a code it give you a suggestions if you write some instruction it writes a code on your ID directly so that becomes more like a assistance with some actionability and uh when this actionability becomes more uh multimodule based more autonomous and uh can take also decides a lot of decisions automatically then we become suda agent so that is how u I personally defined the difference between all these three and how we level through right so again so we have to take the uh right queue on uh should we should we stop at assistance when we are whenever we are building the solutions do we need to go to the copilot state or do we really need a lot of automation so let's go to the agents in that case so I have a question here so what is the difference between multi- aent and multimodal agent so I think uh the the multimodal and multi-agents could be uh so whenever we are creating the agents which are more only textual giving the textual or one type of answer or output or input then we call them as multi- agent when we have multiple agents but they are only let's say using the textual kind of a input output kind of a setup but if we have uh different modes of input and output let's say we are giving the uh text and getting the uh image in in the in the return right uh like Microsoft designer is one of the co-pilot where ask some uh write some prompt it gives you back the kind of a image and we create a agents of that that is what we call as multimod. So that's how we differentiate these two. Uh next is copilots are not a autonomous. So they are autonomous on your reactivity. What we say is uh they are more autonomous than assistants but less autonomous than agents. That is what we can say. And copilots are majorly uh uh to let's say uh specialize over a particular task. Uh I'm again taking example of GitHub copilot. So in the GitHub copilot again you can ask some generic questions but it is majorly designed for ask uh helping you out on the coding part. It is integrated with your uh IDE. So uh when you are writing a particular code, it can understand the particular code and give you the next uh code itself. You can just put a tab and you get the final answer. So it is little bit more autonomous uh understanding of proactive than assistants but it is not completely autonomous like like to create the whole code auto automatically and run it and everything. So if we uh do that part like generating a code from a agent and then we take that code run it out if the code has some issues we go back to that agent update it. So that is a kind of a autonomous uh setup we are talking about when we say about the agent. Okay, I have a question on Q&A as well. Uh, this is a question on the business side. I heard that AI generated blogs get downloaded by SEO, downgraded by SEO, leverage generative blogs to a correct on your site. So again I'm not an expert on this but I think uh so it depends on how uh the particular portal or your blog setup is uh kind of designed on that if they are a if they're able to detect because I have not seen a very uh efficient way to detect a particular text generated from AI or written by human right so u so I I don't believe this can be easily downgraded by SEO uh because there's no direct way that I can see it's uh AI generated uh text or not. There are few tools I have seen but I don't believe in that because I have put my own text which I have written it also saying 60% AI generated which is not at all the case right. So what I would suggest as of now just uh use the different copilots, different chat bots that we have assistants to uh uh to write your uh blogs and accordingly uh uh do a SEO optimization also. Again I'm not expert on that but maybe you can use even GPD's tools uh to do the SEO as well. Okay. What are the most valuable skills in a IML domain? Training the model, finetuning, creating agents. So again uh as of now based on what I've seen in the market uh first of all prompt engineing is very very important. Uh training the model is happening very rare because it needs a lot of data as well as the infrastructure. So that might be required very less in terms of uh if we talk about let's say we have 100 jobs maybe uh two or three of the jobs might need training of the model but fine-tuning and creating a agent is very important and it's going a lot. So maybe that is what you can focus on as of now. training the model maybe you can uh do it later but understanding how LLM works how everything is working how transformer works uh that is very very important so please please do that and then move forward on that okay so uh let's move forward uh so uh we have we have a lot of frameworks available uh to be honest and uh you can also create agent without any framework that is for sure but uh these are the some of the very famous uh frameworks that are available in the market. So I start from lang chain. So langchain is pretty comprehensive. It has a lot of components and we have a extension of this as langraph which is more structured more designed in a better way to use agents uh in a multi- aent setup. But overall lang chain and lang graph combined has a lot of agent tools. By tools I mean meaning is you can call a different tools like you can call a search API, you can call database API. It has a lot of integrations with different databases. So those kind of uh accesses are already integrated inside it. So it be becomes easier uh within the library itself if you have that interaction available. So those tools are available a lot of tools are available already. Uh you have memory management. So by memory management that different agents are talking to each other each other. So you can manage the memory uh around it and it provides a very structured way of u managing the memory uh to the different agents which is kind of I feel common to all the libraries uh memory and the tools as of now but some of the libraries are good at something and some of the good at something. So uh whenever you are trying to explore maybe you can do some kind of a benchmarking on your own and understand which kind of library is working out better for you. But in general I have more experience is autogen uh which I feel is pretty great. Uh and that is what we'll be using today. Right? So lang is a is a comprehensive framework majorly for building applications. Uh uh so it has a lot of integration with the large language models and so many different large language models. So you don't need to create a separate u uh line of uh setup for uh getting the language model. If you have the deployment let's say in your Azure, in your Mistral or any other hugging face model, you can directly uh use some of the settings here and do that right in Autogen. Autogen I believe uh excels in a multi- aent scenario. I think this has a very good setup if you are using a multi- aent. Uh it is also uh giving a very sophisticated way of uh doing the conversation between the agents and it also has the option to put a human in the loop. So in case some agents are talking and if you put a put a condition on if this happens talk to the human and then take a next decision. So that human in the loop option is very very uh impressive here in autogen right and uh now Qi I think Qi becomes also very similar to autogen I would say but uh it has a lot of role based agents. uh autogen also has a lower base agent but I think crew AI has a lot of uh different roles that are already there. It has option to uh do the process management task planning. So again it it becomes more like a how you define your agents in a better way and this also becomes very uh interesting when we are talking about multi- aent setup like talking uh agents are talking to each other right and agent GPD is kind of uh kind of uh very recent it has a it has a very uh userfriendly way of uh uh autonomous agents it has a way you can also see like a particular agent and how it has decomposed the different task which is also there in lang chain which I have seen but I think agent GPD has more details around it but yeah again so there are so many features to each and every framework but I'm just exposing you with few of them um but you can explore uh more as well I personally have not explored all of them to be honest because you can't do that because it's lot of it right so uh I'm also trying to explore if you don't use any of the framework what happens right we can can we create it from the scratch so that is also a possible uh possibility that you can do uh But again you have to uh go to u uh some of the frameworks and compare and then decide on that right. Uh one more thing that autogen has it has a studio available where you can actually do a lot of UI based agent creation. Uh you don't need to actually uh write a lot of code in case you wanted to do that. So that is also one of the possibility with autogen and I think with QI also is there right and and and going forward I think every month a lot of changes are coming so all these libraries are trying to catch up and provide more and more details on u uh the features in terms of that right so uh langraph also has a lot of uh what we say u you can see how your agents are workflow how they're talking to each other but um the task they composition may be not there. So all these different things have different pros and cons. Okay. So let's talk about some of the questions here. What is the difference between autogen and AI agent service? Um I didn't get your question correctly but uh autogen is a library is a package which provides you a lot of different agents capability like you have different packages let's say uh skarn and machine learning you have different models available already into it and you can create different models around it. Similarly, you have let's say autogen which has all lot of agents available into it. You can create your own agents and then make a setup to talk to each other, right? And AI agent service could be more like when you deploy these models uh or agents separately or as a separate microser and then you can call these agents uh with a API call and that is what I'm understanding could be the difference between the terms that you have uh talked about. Can different frameworks communicate with each other? Um I have not seen that happening. Um to be honest I think this is something that we can explore but I I have uh uh what I have seen uh is if you're using one of the frameworks and try to even use any of the other uh setup because these uh libraries are designed for a lot of uh uh what do we say specific setup like autogen is mostly designed by Microsoft so they are focusing on Azure kind of inter interactions but if you use any hugging face model it might break out So these kind of uh issues are there. So uh I should I I think it should uh be the possible in future that uh different frameworks can talk to each other or maybe we can create some kind of a common layer between the different frameworks but as of now I have not seen this happening. So which of these is the most commonly used ones which which more feers? So again it depends on the use case. I use autogen a lot but uh I have seen a lot about ui being uh picked up uh by uh by a lot of uh people around me and lang chain even lang graph I have seen so autogen I've seen a few people but personally I have used autogen mostly so I can say about that so if you are picking up based on the popularity maybe you can start with langra but what I've seen is if you are not very uh what do we say software engineering kind of a skills it might be a little complex to do a langraph kind of a uh agent. So autogen becomes more uh user friendly easy to create for especially for the Python coders. So maybe you can start with autogen and lang and then explore the further ones. Okay. So maybe I'll take few more questions. So so there are not a uh no code tools available for creating agents. Do you think knowing these framework is much advantage? Yeah, I personally believe that because uh obviously if you have the underground working of how agents work, how we can design the the patterns, we'll talk about the patterns as well. Uh then designing it on the UI would also be helpful, right? But in the in the case of agents, if you uh even uh don't do that, it should be fine. But I believe if you do that if you understand uh at least with one particular setup let's say you have taken autogen and created different agents and see how they are working and how we are passing the information to each and every agent within itself. Uh then if you go to a no code tool uh then it becomes faster and easier for you to consume and understand and even debug what's happening with there. Right? uh Microsoft AI agent service used for individual agent creation but if you want to use these agents to collaborate then you use autogen uh yeah so I have not seen Microsoft AI agent service as of now so uh what I am assuming is so it's it's a separate service where you can create uh copilot based uh AI agent so this is what I'm understanding so I'm not aware about this particular service so I cannot comment more on it but if that is available as a API call Right? Then you can definitely use autogen and u make a API call using a tool calling functionality. Right? So I I just quickly explained the tool calling functionality and the function calling functionality for the agents in general is so uh we have these agents they are uh using LLM to uh be more uh intelligent in in some sense right now if we wanted to let's say make a uh extract the data from a database right so what we will have to do we have to make a let's say uh make a connection to the database and pass our query and get the response right so what you can do is you can create a python function and register that function with your agents. So whenever your agents needs a data from a database, it'll use that function automatically. So that kind of understanding and automatic autonomous uh behavior can uh agent be done. Right? This is like a function calling. And if your uh framework or your setup already provides you the tools like API calling let's take you wanted to do a web search for a particular question and then accordingly uh move ahead with the next steps of the agent. So then you have to make a uh API call to your service and your framework or your setup can create this as a tool and then make a API call whenever agent wants. So all these things can be done uh with the setup of tools and the function calling. So that's the reason we were talking about these AI agents are not just LLM based. So it can also be more autonomous by using just the uh basic autonomous setup uh just calling different APIs get the data and once we get the data analyzing the data and making that data uh consumable uh for the end user is something can be done by the LM. So that is how we can do it. I'll go to the Q&A. Uh I think we have already answered these. Now can you please give a real world example where each of those plays a distinct role? Uh uh each of these plays a distinct role. You talking about the frameworks or the agents? Okay. So I I I'll go ahead and maybe I'll also talk about uh your point uh in between right. So uh I think autogen I think that is something that we are uh taking up today. Autogen has a lot of uh flexibility and like I said it becomes very easy to create u the uh the kind of uh eg choosing it you don't need a lot of u expertise in terms of uh coding and everything if you have a decent Python understanding and llm understanding and how to make a calls and everything uh autogen should be good to go right so it it provides you a lot of uh uh multi- aent setup so you can create different kinds of agents and then they can all talk to each other and accordingly give up uh come up with a final output based on the interaction with with they have and even they can do a multi rounds of interaction. So let's say one agent has done something and second is trying to review that and second agent says okay this is not looking good can you make these ABC changes it'll make those changes and do it again so that is how it can uh work out it also has uh uh like I said it has uh yeah so it it actually has a lot of u uh interactions with the with the different kind of uh domains and LLMs it is majorly designed by autogen uh Microsoft so they are uh more compatible able with Azure but it can also work with the different LLMs that you can uh see and during the interaction it actually uh make sure the way it is using the memory they way it is using the historical uh chat with the different agents uh they are trying to optimize it so that you have a lesser cost you have a better inference output uh so it also provides a lot of uh working systems um by the with the wide range of applications from various domains for example I was talking about the tools like search APIs uh the database integration so all these different tools are available with that what happens is uh whenever you are thinking about the problem in a round you don't need to go to a lot of setup and you can even create your own custom agents uh in autogen so uh let's quickly talk about the different type of agents where we have autogen are lot of different type of agents uh we are taking up these three which are ma more most common and majorly used but you create use other uh agents also which are in the experimental state as of now. But also if you want let's say these kind of agents are not working out for you, you want to create your own agent, you can use the same shell within the same autogen as a framework and create your own agents as well uh with your own logic the way it has to work and all those kind of things right. So uh so these are the three type of major agent. So one is assistant agent. Uh the assistant agent majorly functions as like your chat GPD. It actually um uh makes a LLM call based on your input and the uh agents task description or system prompt and perform the particular task and these tasks are majorly based on the LMB. So that is the reason we are calling them as assistant agent. So let's say if I give a topic uh can you create a plan for me to write a blog around it right so that is what is kind of a thing that it can do uh it can make uh even coding suggestions but cannot uh execute the code and those kind of things it can do so it's majorly like how a chat GPD works or a llm uh works it is how it is working in a assistant agent right now next is the conversational agent so conversational conver conversable agent is uh is for uh majorly uh is a is a agent type where uh where you need a lot of uh conversation required with the other agent. So within this conversable agent you can say okay whenever you are trying to do this particular task talk to the assistant agent for this particular task. So accordingly it can do a lot of uh uh talking to in in each other. So we'll see some example when we do the demo how conversible agent can be used right. And the third type of agent uh is the user proxy agent. So this is majorly serves as a orchestrator or a facilator or a kind of a admin. And this can also handle the any of the human input that you wanted to do. Uh for example uh let's say the whole assistant uh agent setup that we have created uh for writing a blog, right? Uh so first layer first agent could that that our system can hit is a user proxy agent which actually understand what is the problem and uh uh then gives a lease to the other agents and also take any further input needed from the human to execute the task. So let's say uh if I uh we have the uh agent setup already done and we just say write a blog right but we are not given any topic. So maybe it can ask you back can you provide us a topic to do that right? So that is how uh user proxy agent can can work out. So it can actually do the human interaction. It can actually uh do uh talk to the multiple agents uh and also uh more like a uh create a u admin kind of a setup more like a facilitator more like a orchestrator can act on on on your task and accordingly get you the final output. Okay. So, uh before we go there, I'll take some questions. Can you also please discuss one business use case that you have worked on? How much was the cost of the API? How did course go? Yeah. So, yeah, I think uh maybe we can talk about the same use case that we are we taking as a demo. So, but uh in one of the so obviously it is making a lot of LM calls and uh it can also do iteratively. So, the cost generally goes higher uh than the regular assistant. But uh the output quality the kind of autonomous uh activity that you get uh can give you a lot of uh uh uh lift in your uh task right maybe saving hours dollar values or whatever right. So again you have to make a call on like to do you want to really spend a cost on that. So that's the reason we talked about the difference between assistants co-pilots and agents. So you go by the level because according to the level you have more cost more complexity increased. So accordingly you have to uh go around it right. Can you um explain more about user plexi agent by giving some examples. So I think I can talk about the same example. So uh uh let's say I wanted to so let's say we have created a agent setup and that is the same demo that we have. So maybe I can explain more in detail there. But just to repeat on the point so it becomes more like a watcher stateup right. So when we are talking about the manager or the CEO example, the CEO would be your uh uh user proxy agent where it'll be taking up the orchestration role talking to the different uh agents and uh finally taking the hit and also providing more inputs to the agent. So let's say uh I wanted to analyze the sales uh for a particular customer, right? But what customer name has to provide by the humans which may be missed in the initial prompt right so user proxy agent will go back to the human and ask can you provide me the customer that you want to analyze right so it becomes more like a uh first hit point more like a statator admin to uh perform the particular task that we have right okay so I'll take some questions from Q&A I think we have already answered these questions so I am dismissing these would that be okay you're talking about the frameworks right so frame frameworks are more like a library. It's a different way of doing the same thing. So, uh real world example there's no direct way to say okay this uh particular framework will always be good in this particular scenario that you have to uh understand and see what is fitting best for your particular role. But uh what I can say is so these are different options that we have there are so many other options and if you feel these options are getting more complex you can actually understand agents as a concept and implement them on its own right so using the scratch uh basic Python and setup of LLM you can do that as well so when we are talking about multi- aents we are talking that okay these are talking to each other so obviously there could be different patterns available so some of the patterns we have mentioned here but just wanted to make Sure and put this point as well that these are not only the way that you have to interact. You can interact in any way you want but these are some of the ways that we have seen um in in very commonly used as a pattern right. So one is a chain of responsibility as a pattern where uh we have agent one talking to uh I hope you can see my cursor. Yeah, can you see it now? Okay. So uh one is a chain of responsibility. Uh so when we have like a chain of events and we just take one a agent perform a task then second agent pick it up then third agent pick it up and then finally we get the output uh there uh and then similarly we have a single agent pattern which is just a basic thing we are uh having a agent perform a task and get you the result right so this could be more like a uh granular uh module when we talking about the CEO and the different departments right so this could be a single agent pattern could be a granular module and the chain of responsibility can be within that particular department and then we can go uh and see the other patterns as well. Right? Uh other could be the observer pattern where we are having a main agent and talking to the different agents and then finally uh uh getting the final output. So it's more like taking a bigger task dividing into smaller task and providing it to all the different u uh agents. But in this case we are not performing any task in general. We are just observing and getting the final output. So that is how we are just saying the different uh names. But in in in uh in in the in principle what we are saying is there would be a orchestrator and there would be a smaller task agent which will interact with the orchestrator and give the final output to them. Right? Another very interesting uh pattern that I uh feel is very very useful is a peer-to-peer pattern uh where each agent is talking to others and taking taking the reviews, comments or uh whatever is could be the wrong with the output that they are generating and then finally generating the newer output. So they're reiterating. So taking example of let's say a coder agent which is kind of coding in Python let's say and there would be another agent which is more like a reviewer or technical reviewer and uh so once the coder agent writes a code reviewer executes that code and see if any errors are happening or perform a particular testing kind of a task and if you find any failures in the testing or a code execution it reverts back to the coding agent and say okay this is not working this is the error. Can you fix it? Now the the coder agent fix it up. Send it back to the reviewer. Now reviewer does the same task. If it executes perfectly, has the task or the test done, then it goes to the next agent and say this is working now you can do the further task. So this is how uh the peer-to-peer interaction the multi- aent setup talking to each other works out uh great here. Yeah. So I think observer and role the hierarchal one is very similar. So the major thing is we are talking about uh they are worker agent when we are talking about they are uh more actional they can also have uh uh what we say they are very similar in in in principle but I have just kept it in in a way that observer are just generating the output based on what they have as a knowledge. uh controller agent and the worker agent in the setup can have more actionability when we are talking about calling the the APIs databases search or other kind of a setup right so that is what just wanted to show that difference but in principle they both are same we have one agent which is kind of a orchestrator or user proxy kind of agent and then we have worker agent or observator agent 1 2 3 uh they can be assistant agent or conversable agent based on the requirement yes observer itself is agent so all these all these boxes that we are seeing they are all agents uh but they are just trying to uh talk to the a uh each other and accordingly do the so all these are agents so uh now now we will just uh do a quick uh coding walk through and the and the demo what we have so in this uh what I'm doing is I'm creating uh uh the flow that you're seeing on the on the right of the slide we have a particular admin who is kind of uh given a task to write a blog let's say So that admin actually assigns a task to planner and planner actually creates a plan of what all different task it has to do right and then we have engineer uh that engineer will actually be writing a code if required uh if not required it will not do that and uh once that code is created executor agent will actually executes the code and checks if this is working out fine or not if not it goes back to the engineer the same loop that we just talked about. And then once everything is done, we just say okay uh the code is executed. Now writer can take all these different inputs and finally write the output right and then we have group chat and the manager kind of a uh agent which are more like uh um making the group chat possible. So this is group chat and manager is more specific to autogen but in general we are talking about we are just collecting a a group of different agents or we call as a mixture of experts to create the the whole flow so that they can talk to each other whenever required. Right? So we are not defining a particular flow in there like like we have done in the in the diagram here. We are doing this just to understand how it should work ideally. But it is on the group chat and the manager or the planner or admin to decide um how they will interact, how they will reinteract or uh talk to each other in in that way. Right? So let me quickly show the the the notebook and do that. Let me quickly put that up here. So uh I just go by the block. I'll not explain each and every line because of the time but I just explain the block and just explain the agent part more right. So we are in general uh importing a different libraries. So uh there are different uh autogen library. So just focus on autogen autogen core is also something being done parallelly but I think autogen uh is something that you can work out with. Then we are using Azure OpenAI and OpenAI to get our LLMs and uh pandas data reader is something that we need to more structured data read. Uh this is not needed. I just included for some of my own task right. So overall we are just importing all these libraries and uh from autogen we are actually getting uh the Azure openi completion chat client because that is what we are using to create a connection uh from our agent to any of our LLM uh hosted. So what I've done is in in my Azure setup I have created a deployment for uh chat GPD 432k model and that is what I'll be using to to run this. Okay. So now uh so these are the agents that we have. We have a a kind of a planner agent which is a admin. Uh so ideally it should be user proxy but I've created a uh conversible agent as of now because that was giving me some error. So I I took some time to fix that but right now we can use also conversible agent because we have defined the task in this sense. So what we are doing is we are naming a agent. So this is how it will interact or understand within the other agent. So let's say other agent wants to interact with this agent. This is the name is something that it has to interact with. So name has to be uh put properly uh by the role or whatever you do that. And then this is a system message. So whenever uh you hit this particular agent, this is the system message will be used to uh uh perform the task. And there's one more option of description where you define a particular agent what this particular agent is. But system message gives you more like actionity. Whenever you are generating the output, this is your task. This is your system prompt that you have to follow. Right? And I'm keeping the code execution false because it does not need to execute any code. It does not need any it needs a lm configuration. So that is what I'm passing. And right now I'm putting human mode uh input as never because I am not needing any human mode uh human interaction in this. But if you wanted any human to provide any inputs here, you just put it to uh always or some of the options we have here. Right. Next agent we are creating as planner. So in this planner we are telling the planner that given a task please define the information that you need to complete the task. Note the information will be retrieved using Python code and uh we are providing all the different steps that a planner has to do. So eventually for writing a blog it has to ask the uh engineer to write the code get the code execute what the output is and based on that it has to uh create the blog right so eventually what kind of blog we are trying to create here is I'll give you the final output u we are trying to create a write uh write a blog u about the fundamental analysis of a stock perform uh price performance and we are providing a date and it'll use a yahoo finance to get the stock data. So let's say if I put a a stock um u apple or something. So it'll take uh the data. It will first of all create a script to get the data from Yahoo Finance for that particular date on that particular stock. Once we get the response, it will then create the whole analysis uh u blog around it. Right? So this is how it has to work. So that is the reason we have given that okay you have to uh have a code uh coding agent which is more like a engineer. Uh okay so going going back to the planner. So planner is actually uh giving the plan. Okay. So these are the different steps that you have to do. And now those steps being picked up by the engineers and the executors right and the writers going forward right. So engineer uh itself is a is a assistant agent because we just need to write a basic code uh based on the task that has been given. So the name is giving as a engineer and passing the ln configuration and here we are giving the description ID it has to be a system message. So where what we are saying is an engineer that writes a code based on the plan provided by the planner. So eventually planner will provide what exactly it has to write as a code then engineer will write the code there. So the next one is uh the executor. So what happens is once the engineer writes a code uh that will actually execute uh the code here and based on the code execution it will just uh mention that there's any issue or not if there's any issue it'll go back to the engineer it will keep on doing that. So planner and the uh this uh admin will handle that coordination and get you get the final working code done. Once everything is done uh the based on the plan being created by the planner writer will take all these inputs and write the output uh write the blog. So for that the system messages please write blogs in the markdown format with the relevant titles put content in a pseudo code. So whatever code sample code to access the Yahoo finance and everything will also be part of your blog. You take the feedback from the admin and refine your blog. So it'll also take the feedback from admin or the planner and accordingly uh change the blog or if required right and then we are defining the group chat which is uh a function of autogen which is kind of creating a list of all the agents so that they can interact within each other. So here we are giving the list of agents that we have created and here we are giving the max as 10. It means that okay overall um the conversation between all the agents should not exceed by 10 because sometime it goes into the the uh infinite loop keep on tagging says thank you well done thank you well done kind of a thing. So we have used 10 here but again you may have to uh find a uh a different way and there are few termination condition also that you can provide uh in the message itself that uh that will also terminate the the the conversation between the agents but for for safety we have given as a 10 here and then uh this chat manager just uses this group chat with the configuration of LLMs to just manage uh which agent to call when all those kind of thing right so overall this is our agentic setup that we have created. So if you see we are not creating a lot of complex code here. We are just using these functions providing the information uh in a very structured way and then we just use group chat and there are different ways we have nested chat also and you can explore those as well. Uh but but based on this you can automatically uh set up the interaction between uh the different agents right. Okay. So now so the question I think about the notebook I think it will be shared uh going forward and the recording will also be shared right. So uh the question is where are you controlling the flow? Yeah. So that is what I think we are not controlling the flow here. We are just giving the bunch of agents and if you wanted to really say uh that okay this particular agent let's say planner should always talk to engineer maybe we can put it as a part of the system uh message like I'm doing it here. Engineer that writes a code based on the plan provided by the planner. Right? So this is where actually I'm trying to uh say that okay you have to understand or interact with planner for sure. So based on that this you can do if you really wanted to control like okay I have to uh call this agent then call this agent and there are other methods also available. So in that case maybe you can use net chat or even just create uh agent wise calling and then it can do that right. So that is what you can do. uh Azour side setup I think I don't uh I have to do a lot of login and everything and we have a very less time here but eventually what I can say is it's a very standard way of creating a a LLM deployment in in Azure right so that is what I'm doing I'm not doing anything different there the only thing is if I use a different version of chat GBT it was giving me some error to me so I'm using 32k GBT4 for specific so I I don't understand the reason around it but that is what I've seen uh recently happening Okay. So now the the execution part. So I think we are just giving the today's data. I given a past date so that we don't have any data issues. And then we have writing the okay write a blog post about the fundamental analysis of a stock price performance. And this is the date and uh use the Yahoo Finance to do that. Now we just uh initiate the chat with our first agent uh and then it should automatically take up uh all the different uh uh different kind of uh uh interactions happening and this is how it is happening. So maybe uh I just save some time we have only 2 3 minutes to just show the output here. Uh so what we have got is the first thing is first of all admin which is our first agent has taken up this task. This is the task that it has received. Now it has decided okay the next speaker should be planner right now planner takes this particular task and writes the plan right to write a blog post fundamental analysis uh we need the following information first of all we need a historical price we need the financial statement of the company and we need all these different information once we have collected all the information we can do the analysis and everything so it has created a plan now writer is talking to each other and creating a very basic blog here right now writer is again uh doing uh another iteration on this. So it is kind of doing a multiple iteration on this and finally creating the output right. Uh now agent is also reflecting on whatever is being written by the writer and planner has decided. Now it is saying some of the pointers that how it can refine the blog written by the uh by the uh and based on that planner is again uh taking a charge here trying to incorporate the changes and again writer is writing the the uh the blog here right so if you see in the previous blog it has written there was no code involved now it is also writing the code uh in in the written by there right so that is what is happening eventually right and then we have the final blog being written. Right? So I just copy pasted the final output and put it as a markdown format here so that we can read the final written output block by the writer here. Right? And we can also use the group chat dot cost and there are few function as well where you can see exactly how much cost it has been done, what is the uh cost it has been taken, how many prompts been taken, cost completion and everything. So all these things you can also have a look at there. Right. Okay. So I think we have just few minutes. So I just talk about few of the challenges generally I face when we are doing this. Um the major challenges like we just talking about is uh the workflow control. U many a times we don't see uh workflow control. It may go here and there. It might not even uh uh talk about u u to some of the agent which might need. So that is where your prompt engineing plays a very good role. uh you have to be very very consistent with the prompt the role that you are defining and you have to uh choose the right structure that right pattern that you wanted to follow. Uh based on that uh uh based on that you can actually uh make your uh agent setup uh more concrete. It is because LLM already has hallucinations and agents also have uh some of the randomness there. So that controlling uh controlling part becomes more uh what we say uh crucial here and also evaluating the agents it's another task right because uh that also becomes very very important um for us to understand how we create the agents and even evaluate their output. So that is where uh we have to be very very cautious that I I think this is this is something at least you should start with. So uh I hope with this session at least you got some idea about the agents. Again this uh session was not designed to give you a very deep understanding of it but at least to make you started with by the agent right so uh I hope this really helps. I can take one more question we have from Dant. Are there any best practice on this? If this long test is provided all the time is it not correct we may have to repeat the process leading to high cost. Right. Yeah I agree. I think cost obviously makes a lot of uh uh uh problem and one of the factors to optimize but I think when we are choosing agents we are obviously going towards the higher cost that is for sure uh but when you're talking about internal messaging I think that is where you may create your own uh creativity to optimize the interaction so instead of passing the whole uh interaction maybe you can do some kind of a summarization and those options are available in some of the frameworks so that is something that you can explore and See right you can use prompt chaining kind of a setup to have uh lesser information being passed out. You can even use uh common memory kind of a setup to have a common me common memory whenever it requires it goes to that instead of passing all the information all the time. Right. So that is what we can do. Okay. Uh I think we are at time. I've taken a little bit more. Thanks for being uh patient with me. Thanks for uh attending the session. I hope this was really helpful. And uh I'm available to connect uh whenever you want uh to over the LinkedIn uh or anywhere possible. Uh there's a poll being posted. Feel free to put your comments and uh feedback there uh so that we can improve these sessions going forward. All right. Uh uh the recording and other things will be shared I am sure uh by the by the program team right uh thank you so much. >> Thanks a lot Nathan. Uh on behalf of thank you for your time and for delivering such a wonderful session. Our audience for sure found it very insightful. We can see that in the chat and the feedback poll and we can conduct more such sessions with you in the future. >> Thank you so much. Take care. Have a nice day guys. Bye-bye. >> Uh thank you Nathan. Bye-bye. So I hope everyone has filled out the feedback form. Uh if not, please do so as your feedback is very available and help us conduct more such sessions. And if you wish to conduct a webinar or are facing any difficulty registering, contact us at dataritantiswith.com. And the recording of this session will be available within 2 days on our community and events platform and the link is in the chat section. And we'll be back with another session data and you can find the link to upcoming sessions in the chat section as well. Till then bye-bye and keep learning. Thank you. Bye-bye. Take care.