Autonomous Multi Agent AI Systems

Key Takeaways

Nathan you're up next and Sam are you gonna be able to stick around and chat with us after this yeah yeah I should should be able to stay around cool so we can open it up and have more uh conversations Nathan did you have something you want to share your screen I'll share my screen it's nice to meet you guys today uh my name is naton I actually like the found this AI company called anote um I won't really talk about this today and attempt to just kind of share what I'm like currently working on which I'm trying to open source for like everyone to use um and I've recently gotten really interested in like agents so um I'll kind of talk about like some things that I'm interested in and what we're trying to build this winter as like an open source uh fun AI research project and um and also like stick around for questions at the end um but the guess the general idea is there's this kind of rise in like AI agents and multi-agent AI systems I know like um everyone uses the words agents it will kind of Define what I mean by agent um but essentially I wanted to kind of share a few things today we'll talk about like the architecture of how you can like build these multi-agent systems we'll talk about like how you can kind of coordinate these teams of AI agents that will work together we'll talk about how you can kind of monitor optimize these agents to be intelligent we'll talk about a few use cases of like where agents just can be like applied and super helpful and then we'll talk about at the end like this kind of idea of like a generalist agent framework and why it's like a really exciting concept um so I think today um just to start we should kind of get on the same page of like terminology so just kind of taken from some sites like crew Ai and fi data we'll call an agent like an autonomous unit um that can basically perform tasks make decisions similar like how a person is there there's like an agent and these agents can kind of do tasks so you can kind of give these agents tools and a description of what they should do in a system prompt then it can kind of do these tasks and what's cool about agents is rather than just having like a single agent you can have these teams of Agents or Crews so similar to like how in the real world you'll have like product managers and software engineers and writers and researchers and salespeople and they all kind of combin together a crew will basically be these kind of teams of agents that do these tasks and the tasks that you can kind of give them could either be things that are done in parallel or in series where you have one thing going to the next thing to the next thing or it could be hierarchical where you can have if then then do this right um and this kind of entire endtoend process of the task is sometimes called like a workflow and the building blocks of these kind of Agents um is essentially like an llm like GPT or Claude or Llama Or mistl and tools which are things like searching the web or being able to use a computer or you know actually being able to implement code so you can kind of have these teams of agents that use LMS and tools to execute really complex tasks that individual agents can't um I kind of first got into the space when I was building my company anod AI uh we built uh a chat bot that look something similar to this where you'd kind of upload these files you'd ask questions and you'd get answers and we kind of realized that you know if you uploaded like tons of files and you asked questions that were super complex or domain specific it was really difficult to kind of get high quality anwers to questions so we were looking into like a lot of different ideas like fine tuning or enhanced rag or labeling or different evals and then we kind of stumbled along the idea of agentic rag and that kind of went to like a whole flow of Agents um I'd say um what I'm trying to build as like an open source product this winter just for anyone to kind of use and try this way to like build autonomous agents um have these teams of Agents like work together and like have the ability to kind of Monitor and optimize these agents you can kind of have them perform really really well I think it's worth kind of talking about like how this could work so like one way is maybe you'd have a specific agent that'll be go to these specific use cases like email Outreach or financial analysis or an automatic coding agent then another frame of thought is like you can have agents that would be like this generalist framework so my opinion personally is that the latter approach the generalist framework will kind of be similar to like how um you have generalist model like like GPT and CLA and llama um and then there'll be like domain specific agents similar to like how you have fine tune models for tasks um but I'll kind of first talk about a few of these like use cases and then I'll talk about like why I'm really interested in the generalist framework and how I think it could work and and I'll share a link to like what we're building and where it's going to be and why I think it's cool um so I say like one use case is like an obvious use case like email Outreach so a while ago I was actually building this um random product um called upreach and essentially what this was is just like a hack together product where it was like you'd kind of want to search for people then email them like a automatic spammer right um so you'd basically find people you'd generate emails and you'd send it and you'd ideally do it at scale and the idea was that this could help uh save time to out reach out to people if you wanted to like host events or host uh Drive awareness or kind of causes right so in the ideal World rather than kind of have this product where you'd actually have to go on a UI and search and find add people list and generate emails it' be cool if you could just basically tell like a chat bot like chat gbt hey let's reach out to like a list of 10,000 people that are heads of AI in New York that work at these midsize finance companies and invite them to like our AI event right and you'd have this kind of agents that would basically be like almost like a series of apis that might find the right people look and enrich the data from the web generate emails and send them out right and maybe youd kind of repeat doing that or maybe there's kind of like a use case where it's like hey you want to apply to all these grants to get funding but um let's say you wanted to apply to these rfps but you there's maybe like 100 rfps and it just takes a ton of time to find them to write these applications and submit them but if you could actually use AI to do that that would be amazing that way you can get a lot of really awesome opportunities um so the idea is like when you would apply to these grants maybe you'd want to do research you might want to write these grants you might want to ensure that it fits the format you might want to revise the Grant and then you might want to upload it to some portal and submit it and for each Grant you want it to be like super tailored and unique so you can imagine in in general software engineering framework you might have a series of apis for each of these things but with agents ideally there could be like a way that you can kind of use tools like like writing researching using the web to kind of automate this process and we built something related to this too um obviously there's kind of use cases like Finance right where this is an example using the FI data framework where you might want to kind of have a web agent that searches the web uh a finance agent that can kind of grab information about stocks then maybe you'd want to kind of call these in the team to summarize information on a specific stock you're interested in and then there's obviously like the event planning where it's like hey you might have these different agents that might one might help you find a venue one might help you do Logistics one might help you market and communicate event and you can kind of chain these together and give each agent these different tasks to basically potentially in the future not today but maybe in the future host these awesome AI events i' say these are kind of specific use cases that you might think of in a framework and each of these kind of use cases could use like these open- Source building blocks like dpy or things like pantic or Lang chain or L Index right where you can kind of use code and have each agent be some sort of module and you can kind of give each agent tools and techniques and try to optimize them and obviously each framework has its pros and cons um and I think it's what's cool about them is you can have these kind of compound systems of module with inputs and outputs and each module can have these tools and tasks um there's been a shift recently um within the research Community to this general idea of generalist agents so what this basically means um in theory is that rather than basically being like hey I'm just going to do task X and Y and Z for outreach what I'm going to do instead is I'm going to have these kind of templates of Agents um that have ACC to tools and then when I'm going to basically put in a query like hey I want to reach out to like all the people in Ai and send them an email about this cool event I'm hosting uh you'll have this thing that's called this orchestrator and this orchestrator essentially is ideally going to be really smart and it's going to be able to spawn and create these different agents in the order that they should be in either in series or parallel um and whatever kind of and give them the tools they need and then this orchestrator will assign these agents the order give them the tools and tasks they need these agents are going to go out and actually try to do these tasks and then ideally if they can all return the right values um you'll be able to just have this framework that will be like super generalizable in theory it's great you know um if it all worked you just kind of ask a chat bot to do something similar to like chat gbt and it just basically be able to use all the tools and templates and figure out how to do everything um in practice it's going to be like really hard I'd say there's like a lot of these open-ended questions of like how do you optimize orchestrator and make it really smart how do you actually evaluate these agentic systems to ensure reliability can you fine-tune like each module and what data do you need what if you want to run these agents privately how do you monitor these agents via logs so we're actually basically trying to do this like project over the winner where we're going to basically open source our code um and just let anyone who want wants to like learn about agents and how they work like work together on this like centralized place um we've already have you know like just like a typical like Standalone like thing right where you can have this chat bot you can ask questions get answers and we're like providing that to just everyone for free no strings attached so you can just kind of Clone the repo try it we have like instructions to actually like set up the code if you want I included also like um this deck which kind of just has the information I talked about today um in case you're interested and I'd say like some of the kind of themes right is you'll have this um general purpose agent framework that we want to build this like registry we're going to have these domain specific agents the ability to kind of do smart orchestration and use these tools some way where you can like log and monitor and understand how each of these agents like work um and then ideally as a developer you might want to interact with it via code so um something that would be able this um it'll be like a really interesting project and hopefully we'll learn a lot and uh ideally in like middle of February we'll have some cool presentations on things um so yeah you know there's a lot of use cases where this could be applied I kind of share this all here um from videos to like resources on like things we've built at my company anote um that are just we're just kind of going to provide to the community as well for like free and I think our goal you know is to try to like learn as much as we can about how these agents work how you can build these teams um learning about all these awesome Frameworks that are out there and we're excited for like the opportunity to hopefully like learn together and U happy to answer any questions too awesome dude the last slide so while if anybody has any questions throw them in the chat the uh last slide or second to last slide that you had in there about um what is going on in research I'm not sure I fully understood that with like the it's a it's the common design pattern what was it yeah so essentially what's going on is rather than having to build an agent that will be like your Logistics manager or your Venue coordinator or your like marketing and Outreach person you're going to have these things that are templates and when you will kind of input a query to a chat boot similar to chat gbt or the chat bot I showed this orchestrator is going to basically be able to take in this query figure out which agents or templates it should kind of call what order they should actually like do the operations and then from the order you're going to have and the agent you're going to call in this order each agent will kind of do the task you'll ideally be able to kind of Spawn via yaml or markdown or be able to edit um you know these kind of agents to share it works and monitor and log them and ideally optimize the system but it's going to basically be able to figure out which tools to use and actually like do the thing like autonomously and uh that's I think like the cool idea that um I think it's very open unded on how to do it but like there's a lot of things like evaluation and fine-tuning and what Frameworks to use and we're trying to like learn about it yeah awesome