Learn How to Build an Agentic System

Key Takeaways

Hello, we're live. Welcome back to our live streams beyond the algorithm with NVIDIA. This week we're going to be talking about our NVIDIA agent toolkit hackathon and we have a couple of guests here joining us. Uh probably some new faces to most of you. Um give them a quick run through and then you know they can talk about what it is that they're going to help you guys with today. So, first I'd like to introduce Duv. He's an agentic AI and cyber solutions architect um here at NVIDIA who specializes in enterprise scale agentic AI deployment and development. And so, you know, I think Drew, this is your first time here. Yes, it is good to be here. Thanks, Zach. And next, we have JB who's a solutions architect manager. Uh JB leads a team of solution architects who focuses on accelerated data processing, cyber security, and agentic AI. And finally, we have Matt who's a senior solutions architect uh on the NVIDIA AI enterprise. Uh he's technical lead driving innovation and performance in enterprise agentic AI and cyber security systems. So guys, welcome. Thank you. And thank you for those of you joining today from the community. You know, always appreciate joining on our live streams. You know, these streams are really for you, the community to ask questions. So, please, this is an open forum to you. Ask questions and we'll kind of pause in the middle of uh presentations and some of the code to take your questions. So, without further ado, Droo, I'll hand it over to you. All right. Thank you, Zach. Uh, and again, it's good to be here. Thanks everybody for joining. Uh today we're on the live stream we're going to be talking about our agent toolkit hackathon and hopefully give you all a lot of great resources and be able to answer your questions about building with the NVIDIA agents intelligence toolkit and some of the outcomes that you know you can hope to see through the hackathon and at the end of it. And so by the way of introductions the NVIDIA agent intelligence toolkit is an open source library for connecting evaluating profiling and deploying enterprisegraded agents. Um we all know deploying agents is hard. uh there are a lot of moving parts and a lot of complex pieces that are involved with deploying agents and so we're hoping that the NVIDIA agent a agent toolkit will be able to simplify some of that reduce your time to go to production and give you kind of full stack observability and profiler and evaluation etc along with it and so in order of an agenda today we're we have a pretty packed agenda for you we're going to start off with Matt who will be giving us a live code walkthrough so that everybody can understand how the agent toolkit is structured how to build with it maybe give you an introductory example after which we'll open up for QA. We'll go around to our panelists here including JB and Matt and understand you know uh what some of the design philosophy is behind the toolkit um and what maybe some advice we have for um those of you who are building with the toolkit over the next coming days. So really excited to get started and please feel free to post your questions in the chat and you know we'll we'll get to them um through the session as well as you know at points where we h where we can stop to take questions dedicated to answering the questions you have and so uh without further ado I'm going to hand it over to Matt. that um live coding example all you all righty let me get my screen up here okay getting a little bit of a this is live and in person so it's not pre-recorded getting a little bit of a delay here on my side with Um, give me just a moment here. Uh, Drew, why don't you, uh, describe some of the design philosophy in parallel here. Sure. Of course. So, when we built the agent intelligence toolkit, um, there were some core pillars that we wanted to use to be able to build the library. The first of course is framework agnostic which means that whether you're a developer who likes to build a lang chain or lang graph or kur some of the other incredible frameworks for agents that exist today. We wanted you to be able to bring that to the agent intelligence toolkit and build with it without too much of a without too much of a difficulty in migrating that over. Second we wanted to be able to give you full stack observability evaluation and profiling which means you developers want to be able to understand how their agents are doing in production. are they accurate? Are they relevant? You know, are there performance bottlenecks, etc. And so the agent intelligence toolkit really helps developers understand that and see it end to end before moving to production. And finally, um it is completely optin and extensible, which means that if there is because it's an open source FDK, if there are libraries or other tools that are not supported today and you want to extend it, it's very simple with a plug-in architecture. You use what you need and only what you need. And if you need to extend it, we make that really simple, too. So those are some of the core design philosophies here. Um I see a question from Moscone is will there be open source frameworks from NVIDIA for building agentic systems with LLM? Um JB did I'll let you take this one if you have any thoughts about that question. You're you're learning about it today. Um so I think I think one thing to to make clear is right like the um the agent intelligence toolkit isn't trying to replace things like line chain or llama index. Um, it's a it's a framework to build around those things. It is completely open source. Um, and like Drew mentioned, it's there's, you know, composability, reusability, evaluation, profiling. It's more than just running agents. It's building, running, testing, evaluating. So, you can think of it as a toolkit to help with the entire software development life cycle of agentic AI and and workflows. Um and you know some of the we'll also give you a lot of great examples and documentation. The a the IQ toolkit um GitHub repository has a lot of great code examples for you to get started. If you go to the hackathon website, there are a lot of great resources there as well as the documentation which has you know fantastic guides on how you evaluate an agent, how do you profile an agent, um how do you extend it and there are some great blogs that came up as well. So definitely recommend checking those out as you go through. Um, just as we wait for Matt to come back, uh, we can take a look to see if there are any other questions from the community. Um, and or in the meantime, JB, maybe I can ask you a couple questions about the IQ, the agent intelligence toolkit, and we could talk about um, why it was designed the way it was. So, do you have any thoughts or ideas you want to share about um what the core benefits of building with the agent intelligence toolkit are and any details you can share with everybody about building with that? Yeah, for sure. I think you know one thing that's interesting it's like right tool for the right job. Um, and a lot of these frameworks that you're coming, you know, you're seeing every other day there's something new, right? There's MCP, there's A2A, there's line chain, crew AI, there's a ton of these frameworks, a ton of protocols. Um, and you know, they're all do something differently and they all do something, you know, perfectly, right? And so with the ability, if you're building an agentic system with the IQ toolkit, you're not just building an agent, but you're building all of the pieces of the agents. Um, and it was designed as sort of like software engineering first, right? Like you take your first 101 coding class and you're learning about functions, right? Like the this small unit of logic when you're writing your software application. And what the team has done is they've taken that that core fundamental building block and used that as the building blocks of agents with the toolkit. And so everything within the IQ toolkit is developed as a function. So a tool call is a function. An LLM call can be a function. An agent itself could be a function, a series of agents can be a function. And so we really built it down to those those key building blocks. And because there are those individual functions, we have the ability to use the best pieces of all these frameworks. So if you want to build a rag system with llama index, you can build a retriever within the IQ toolkit with llama index. With the composability and reusability of those functions, we are able to send that or uh show that rag function to a a crew of agents maybe in crew AI. um if there's a specific tool call that you want to build in langraph lang chain you can serve that up as a function and so it's it's more than just agents in these frameworks it's these fundamental building blocks can be changed and and built in these various frameworks as well yeah thank you JB that was fantastic I actually saw a couple questions come up as you were speaking so if if you wouldn't mind I might ask you a couple of those as as we wait for Matt um the first is what about MCP do we support that with the Asian intelligence toolkit uh And what is the extent of that support? Yeah, so um it's a function. So an MCP tool, MCP server can be a function. Um so within the configuration for an agent, you list out various functions, the types of functions that they may be. And one of those configurations options is an MCP um client. And so you can configure these multiple tools. If you want to build your own tool uh in one framework, you can, you know, locally or if you have a tool that's remote as an MCP server, you can configure that MCP server as a tool as well. That's one side of the picture, right? The client side, but we also can serve up an a workflow within the IQ toolkit as an MCP server. And so if you have other agents or other tools that you want to connect to your workflow, um you can also serve up your workflow as an MCC MCP server as well. That's awesome. And so I I heard you mention you know lang chain lang graph or aai. What about the LMS? Do we does the agent toolkit support openai LMS versus NIMS which are Nvidia inference microservices and or is there only support for one of the other? Yep it's both. Um so um we have the ability to hook into you know OpenAI compliant APIs. Um and that's either you know running locally with the NVIDIA inference microservices. So there's the ability to download a model that's optimized for your hardware, whether it's, you know, running in the cloud, running on prem um and then configure your workflow to connect to those nims. Um and it's also the ability to reach out to remote endpoints. Um so build.envidia.com Nvidia.com serves up um some of these Nemo inference microservices uh you know for for de development purposes but then you can also reach out to things like an open AI server um you know if you have an API key to connect to that that's awesome thank you for sharing I saw another question about you know can we build workflows with Lang graph just like we would you know currently and I think that's a great segue to to Matt who's back Matt um is your is everything okay with this the tech right now and are you to show your screen. Nothing like a live demo. I think everything is good with the tech. I turned off my blur in case that was competing for for resources. Uh so let's try this one more time. Okay, we've already made it further than the last time. This is a good sign. It's great. Okay. Uh if if someone could just let me know that they're seeing my screen, uh that'll be good. And then I'll get going. Just got to share it. I think you you're up. Yeah, you're all set. All right, perfect. Um, so what we're going to do here is we're just going to take a look at how to use the toolkit. Um, and I'm going to come at this from the perspective of uh, you know, a lang chain developer. You know, many folks might have some code that they already uh have implemented or maybe are familiar with a with a LLM framework already. This is a common entry point. You probably have some some scripts like this sitting around. This script in particular was derived from the lang chain documentation kind of the onboarding to agents. Um so what we're going to do is we're going to walk through this here really quickly and then see what the process looks like in order to bring this into um into the AIQ uh programming model. So um we've got here is um okay so we've got here is you know some standard imports um and we have a uh internet search right this tabi internet search this is a common tool that folks might use uh in order to provide their agent with information that would only be accessible by quering the internet right and uh you know since we're talking about agents agents typically have a set of tools And those tools are are used in order to uh basically solve problems as the agent reads through it. Right? So we're going to create a list of these. Right? Now our our list of tools is of size one. The next piece that we're going to do is create a LLM client. Right? So this LLM client is going to be kind of the main reasoning uh brain behind our problem here. And we're going to use here llama instruct model in order to do that. Then as we scroll down, uh, we're just going to use an open- source prompt, uh, instantiate a React agent. You see, we're going to pass in the tools and our LLM and this prompt. Then we're going to instantiate an agent executor. This agent executor is basically going to create a loop for which we kind of a reasoning loop. And then towards the bottom, we're going to invoke it with a simple prompt here and then uh take a look at the output. Um, so we should be able to run this as is, right? All right. And then we're going to see what it looks like to uh bring this into the into the toolkit. Okay. So, here heads into the loop. It's going to go ahead and call this tool that we want and it's going to give us the answer. This is great. Um, so what we're going to do at this point is I'm going to use some of our uh command line uh utilities in order to build some scaffolding. So, I'm what I'm going to do here is I'm going to create just a a sample uh library configuration set or some scaffolding. Um, and then we'll talk about why I'm doing that once I have this thing built. Okay, this looks good to me. So what this is going to do is it's going to in my temporary folder it's going to create uh basically some uh some scaffolding since every uh toolkit um plugin is effectively a library. What this does is this creates a simple Python library for us with some setup uh pieces here like this pi project tol file. And what we'll notice is this looks you know pretty standard. Uh we're bringing in some toolkit dependencies here. Um, but I'll have folks pay special attention to this section here. Um, what this is basically doing is it's saying, okay, well, we're going to be implementing some things within this, you know, Python package, and in that Python package is going to include some uh toolkit plugins, and the plugins are going to be found in this register. py file, right? So, this is going to be contained in our source directory. This is very key. Um, so everything that we're going to be do, everything that we're going to do here is going to be registered as a plugin. Right? So, we're going to take a quick look at that and that's going to be this here. Right? So, we have this AIQ or this uh this toolkit uh registered function that's implemented and imported here. Um because it's imported here, it then gets picked up by our plug-in hooks here. So, that's important. Um this is all created for us. Um so, I'm going to go ahead and close out of this. And the next thing I'm going to do is take a look at this actual registered function. All right. All right. So, this registered function, if I can get this thing over here on the side, is going to be the common uh kind of developer interface that you're going to be um working within. Right? So, um this top section here is a uh basically a named pyantic model. Pidantic model uh pidantic models are really good at enforcing types. And uh we're going to use this in order to define some runtime parameters for this registered function and give it a name so that we can look it up from the toolkits kind of local registry, right? And we're able to this is important because every function uh or every you know agentic workflow is built up from a YAML configuration file where we're looking up things by names. And we're going to take a look at a configuration file uh here really shortly. Actually, matter of fact, let's take a look at one right now. All right. Right. So this is a very small and simple one. Um, right. So this workflow here is going to actually map to this registered function that we have implemented here. And that's all built kind of default by the by the scaffolding. And you'll see that this name maps to this name here. Right? This is looking this thing up because this plugin is registered locally and it has exactly one parameter which is uh largely meaningless right now. and it's paired up with this registered function where we pass in this configuration object. Right? This is how this thing gets registered. Um we're actually going to extend this here really quickly and we're going to include I'm going to make this a little bit wider here so we can see it. Um this this section called framework rappers. We heard Drew talk about profiling. Um this is how we're going to enable profiling, right? We're going to be implementing this in lane chain. So we're going to add these framework wrappers uh that focus on our u that focus on lang chain so that we can profile this thing. So I have to do one more import to get this right. So I have that. The next section here is kind of the main um uh the function implementation. Typically you'll implement a bunch of setup code here and then in here you'll put you know whenever a message shows up what you want to actually be done and then at the end uh we you know uh yield this intermediate uh function info object. This lets us construct these functions and wrap them at runtime and let's say a llama index or a lang chain or a semantic kernel or pick your favorite framework. This is great because I can build this plugin and I can release it to the uh developer ecosystem which includes more than just lang chain developers, right? People who are more than lang chain developers can take advantages of this in the end and then some cleanup right at the end. Um so that's the main setup. So here what's going to happen by default is just this hello world type of output. Um, and we're going to now bring in our lang chain example into this. And we're going to start doing it in kind of a naive way. Uh, the toolkit is very optin. And I'm actually going to make this just a little bit smaller for just a moment. And just bear with me while I do that. Um, right. So, we're going to we're going to start by opting into very little. Right. So, I'm going to take my setup code, which is going to come all the way through here. Right. And I'm just going to take this wholesale and I'm just gonna paste it right here. Then I'm going to Oops. Do this. All right. I'm going to tab this over. So Matt, you're just taking you're just taking your lang chain code straight line chain and then copying it right into the IQ toolkit framework, right? Bingo. That's exactly what I'm doing. Right. So, this is typically the first thing folks will do uh when onboarding is they'll take their their code, they'll they'll move it over. They may not, you know, opt into everything, right? And they'll just get it going to start. And this is typically what I recommend for folks, right? Let's just bring in your code here. Let's get it working and then let's think about ways for which we can start to optimize this. Um so one thing I'll just pause on here and address is um the toolkit is async first and the reason for this is uh you know we spend a lot of time waiting for responses to come back from a large language model right so if I can instead of doing things synchronous uh I can do things async I can have a lot more messages in flight and I can have a higher uh throughput put for this micros service that I bring up. It's all based on this uh toolkit implementation here. So you'll see you'll see things like await and you'll see things like async all over the place and the reason for that is because we're async first and we do that for per um for performance reasons. Okay, change this to a return. Matt, would you mind if I ask a question please? I know you talked about async and performance. So I was wondering you said the agent toolkit is able to track, you know, my link chain because I decorated with a framework wrapper. Does that make my agents slower or is that async as well? Everything's async. Um, so there's very little overhead here. Uh, when running this, the the overhead that that you'll notice is, you know, there's a little bit overhead when we bring up the micros service, but that's true for just about every micro service. Um, everything is async and non-blocking. Um, so the overhead that you'll see is negligible. Um, but you know, the overhead that you you will notice is that we did have to take our code and kind of put it into the programming model. But as we can see here that that lift is is actually quite light. Um, so we should at this point have our code completely ported over to the toolkit and I can now Oh, go ahead Drew. Were you gonna say something? Just a really quick followup as well. Um, we see a question on the screen about Lang graph. So if I had a Lang graph agent, would I still just be able to copy paste and do exactly what you're doing right now or that look different for me? Yeah, the the process would be identical. the the only difference is you'd be defining your graph instead of using the kind of barebones link chain interface. Awesome. Thank you. You bet. Okay. Um let's see. We're going to get this served up here. Okay. So, what I'm going to do now is I'm going to uh bring up a micros service. um for uh this new toolkit application and it's going to give us the same functional functionality that we had on the left here but instead what it's going to do is give us microser using all of our um toolkit goodness. So when this comes up probably now make this a little bit bigger for folks. So we now have a microser running on port 8000 and um what I can do is I can go ahead and test it, right? So that's exactly what we're going to do here. And I'll ask it the same for fairness, I'll ask it the same exact question we asked it in uh in our standalone um uh example. Uh let's try this out. what is the current weather in Santa Clara? And go ahead and run this, right? And we can see that this is being invoked. Um, but the key difference here is um we're using the uh toolkits fast API front-end plugin in order to bring up a micros service and we're handling everything async on the way in. So we can actually you can imagine this being the entry point to your uh uh container maybe running in some kind of Kubernetes cluster right um you can see here that it made an internet search and it gave us uh a response and that's you know indicative of it using the tool that was kind of called as it's and we and we saw that output as it was scrolling through but we can do a little bit better. Um so let's get rid of this for a second and let's talk about opting in. Right. So we have we have a number of of things that we might want to do. Um one of those things that we might want to do is um one of the things that we might want to do is add some parameters here. Right? This is this is a very standalone function that I can't use with anything else. Right? So let's actually make these parameters meaningful. So the first thing I'm going to do is add uh tool names. Right? Oh, that's really nice. Kind of gave me everything. uh tool names is important uh because if I wanted to use this agent with some other set of tools that are more than internet search I should be able to do that right I can and I I would want to be able to configure that from my uh YAML file right so that's kind of what we did here llm name is going to give us uh let's say we didn't want to use that llama model let's say we wanted to use a 3.1 model or you know a smaller model or you know an open AI model or something else right we want to make that configurable Um let's say we wanted to pass in uh history you know historical information like chat history or we wanted to change the iterations of uh you know our our agent executor so on and so forth right change the description of what this thing is capable of doing in case another agent wants to call this as a tool right so what we're doing here is we're adding in this configurability I see a few squiggies though um and we're going to go ahead and oh go ahead Matt really quick a question just before we move on uh there was a question from a user about a combination of different techniques for example other machine learning nims other kinds of LLM maybe SQL as part of an agent. So if for example a developer were to build the code for support that to support that would those also be configurable using the config object and would the IQ toolkit support that? Yes. Um so this is an extremely flexible data structure. Uh so if you can effectively define it in a pyantic model, you can use it to configure. And because each of these functions is a black box, you can put whatever you want in here, right? I could put a SQL agent in here. I can name a standalone SQL agent here and have this whole thing be very SQL oriented. and then I can just call it as long as it's been configured and I can use that in any kind of uh kind of tool calling or just directly invoking scenario. So I think the answer is yes, Drew, that type of thing would be supported and it wouldn't be too hard. Thank you. You bet. Okay, so now that we've updated our configuration object, let's let's maybe uh see if we can update our configuration file. So if I'm going to pass in a tool name, I need to have that tool defined. So conveniently, there actually is a Tavly tool built into the to the library. So let's go ahead and use one of our other command line utilities, right? So what we're using here is, you know, after after a while, you're going to have a lot of registered components um or plugins and um it can be really helpful to see what you have and how to configure those. So here I did just a quick search to see if we had a another tool that we could use and as it turns out, we do. Right? All right. So, what I'm going to do is I'm going to app amend our uh configuration file with a new tool that has a name. And Matt, Matt, as you fill this out, um why did we use Tavi here? And if I wanted to switch to something like Duck Go, uh would the lift to include a tool like that be significant? I'm sorry, what was the tool that you'd be considering? For example, if we wanted to use duck.go search instead of taby, um would I be able to write a function to do that and import it just like you are right now? The process would be identical. Um the only the only difference here is if uh that tool didn't exist, you'd have to create something that looks a lot like what we have on the right so that I can refer to it by name. The name would be here and then the process would be the same. Um so you very much could do something like that. In fact, we've designed the toolkit to support that use case and many others like it. Hopefully that answers the question. That was great. Thank you. Do that. Okay. So, we're going to pass in one tool name, right? We have one function defined and we're going to use um let's just do this. My internet. Okay. The reason I'm naming this different is just to show that I could have multiple instantiations of the same underlying tool type and you want to refer to it by this name. So that's what I'm doing here. Um we don't have we don't yet have an LLM defined. So what I'm going to do is do that right now in this LLM section. So, you must have like a huge GPU running this LLM on your machine or how are you connecting to these? I'm connecting to these through the uh build.envidia.com. Uh, and I So, this actually the toolkit does not require a GPU at all. Um, so the system I'm running this on uh doesn't have a GPU. So, that's really great. Um all the GPU computations are happening remotely and it's through build and Matt in the demo that where you ran the agent just now before looking at the config was the search also tavly was the same tool you were talking about identical um yep uh we found that that tool was so useful that we kind of created a pre-wrapped plugin so that others could take advantage of it uh in fact we looked at the the lang chain docs when doing that um we used the the lang chain tavly search uh inside of this blackbox wrapped tool. The key difference is it's very easy to use this uh with any other framework as well. Um so hopefully that answers that question. Um so what I've done um here now is I've I've added a LLM that can be used right configurable. I could have many different LLMs in this section. I've defined a function, right, that can be called as a tool and I'm going to pass that function in or that that register function here as a tool to our React agent to try to make this thing a little bit more flexible. I'm now going to use uh the name the runtime name of this LLM here. And I'm going to change this up just a little bit because the prediction was not quite right. And I'm going to validate this against my configuration object. So I have tool names, LLM name, max history, and they're okay. All the rest of these have default values. I don't have a description. Um, not that anything else is using this, so it's going to use this default. That's probably fine. And I think that will do it. Uh, as far as configuration is set. So now let's change our implementation. Take advantage of some of these. Um before but before I go too far, I'm using a couple data types that you may not be familiar with. So this function ref and LLM ref are designed so that you can have tools or registered functions call other registered functions. And when doing this, you know, what's kind of implied there is we've got, you know, this nested structure. And what these basically do is ensure that all things get brought up in the right dependency order so that you can have these really fancy hierarchical agents or tool calling scenarios. So if you're doing something like that, it's great to use this function ref or LLM ref for any of our plug-in types. Um, and that's what we're doing here. Um, they're effectively treated like a string uh with some additional enhancements in order to support that runtime dependency graph building. Okay, so now uh it's worth talking about this config and this builder, right? We didn't talk about those before because we weren't using them. Um but now what we can do is we can reference anything uh that's defined in our configuration object through this variable and we can use you know a lot of this information to instantiate LLM clients and tools and registered functions with their runtime settings. And that's exactly what we're going to do in order to make this more flexible. So, we're going to go ahead and start with with this section, right? So, we're going to do this tools equals builder dot get tools. Okay. So what this is going to do is it's going to look at our configuration object that is populated with the settings here and give me an instance of all the tools that are defined and then wrap them in lane chain which basically gives me this line here in a single shot in a much more flexible way. So that's what we've done. So now, sorry Matt. Um, if if my developer if another developer had written their tool in Llama index, could I still import it as a lang chain tool because I'm working with lang chain? Yes. So if if if they had implemented something in llama index, um, I would instead do this. Uh, well, so you would have no idea what's in that black box, number one. And if you were working in lang chain, it would look just like this. Um, so you would have that would be completely transparent. Um if you instead uh wanted to work in llama index, it would look like this, right? So you'd be able to mix and match uh very easily in a scenario like that. Drew, did that answer the question? Yes. Thank you. You bet. So the next thing we're going to do is we're going to get rid of this and we're instead going to construct a LLM at runtime. Okay, we're going to pull in our LLM name and we're going to create a link chain wrapped LLM client, which is really great. We haven't changed anything else except made that a little bit more flexible. So now I could I could add a whole bunch of other different types of LMS, give them a different runtime instance name and pass that in here, right? And my my now my registered function would be, you know, flexible uh in order to support those different scenarios. All right, so the next thing we're going to do is we're going to bring in some of our uh configuration settings here. So I'm going to do this now. Right. So now you can change the settings of your agent executor uh from EML at this point. Right? We pulled those things in. And then the rest of this should be the same. All right. So let's go ahead and run this again. If I haven't completely broken this, it should come up. Okay, so now let's bring this up again, right? Just a really convenient way to test your API. Okay, we see our new tool being invoked by the name that we've defined in our YL and we go ahead and and get an answer. Right? So that's that's great. It's the same functionality but now the difference is it's something I can share with my team. Right? So any if if I find that this agent, this general agent would be applicable to other developers I'm working with, I can now give them this Python package and if they have the toolkit installed, all they have to do is pip install it and they can see this tool by name, right? And or they can they can reference this tool by name in their YAML configuration. They can search for it very similar how we search for the Tavly tool and they can start to configure it and use it in their application. Matt, if I had tools with similar names, um, how can I differentiate these different tools or how can the agent differentiate between between different tools? Is there a place I can provide a description or something similar? Ah, yes. So, I I might take this question in a few different directions because it's a good one. Um, so these types are namespaced, right? So if you have collisions in this name and across different packages, you would just add the package name in front and that would handle the collisions. So that's one way you could do it. Um let's see what's what's another way to so if I wanted to um use this agent in let's see uh in several scenarios. Let me make this a little wider. Right. I could I could actually what I could do is I could say, "Okay, this is getting overly uh uh liberal with its uh corrections. What I could do is say my agent one and I'm using this same underlying type and I could say my agent two right so this is the reason for the flexibility um and I could say let's say another tool right so this other tool would have to be defined above in this function section. Um but you can use the same underlying type many times give it different runtime settings and you could actually call these from other functions or from this top level tool right so like another tool um rather uh my agent too. So you're showing basically right now like a multi- aent system. You're having an agent that has access to multiple agents and then can choose based on the description. Precisely. Precisely. Awesome. Right. So I'm going to I'm going to go ahead and delete Oh, go ahead JB, please. I was just going to say and that's I mean that was just you already have one agent configured and you're just adding you you already have the one agent coded up and then you're just adding it into the configuration file and now you're getting this multi- aent system. There's no extra coding needed on this example. That's right. And I think that's, you know, a lot of the beauty of the way we've designed this toolkit is to minimize the amount of redundant code that you would have to write in order to uh basically design, you know, up to a multi- aent system, right? We used the same underlying agent implementation, but because we made it more flexible, right? We could pass in a list of tools to it, give it a different description, and now it's a completely different tool, right? And that's super powerful. Um so that was a really astute observation JB. Um so another thing I wanted to show here really quickly is I can use this um command line utility. What did we call this? My react agent. All right. So if I shared this with my colleague and they pip installed it, right, they now have access to this on their system. In the same way we figured out how to configure our Cavali internet search, they can figure out how to configure their React agent, right? Oh, tool names, LLM name, all so on and so forth, right? As well as the default values. Um, so that's really the beauty of the plug-in system there. Um, so we're going to do one last enhancement to this and I'm going to show you how to use this with our open source UI. And um, I do have a cheat sheet here for for time and I'm going to pull this one in because it's a little bit different. Um, Oops. Okay. So, I'm going to comment this out and I'm going to put this in, but I need to do a few additional imports here to support this. What I'm going to do is I'm going to use one of the built-in um some of the built-in uh objects from the toolkit, in particular, these OpenAI spec uh data models, right? so that I can expose an OpenAI compliant micros service uh from the toolkit and that's what we're going to do here. So, we're going to use this different type and this different type also lets us pass in uh message history, right? So, I can support things like multi-turn and our UI can provide that information. So, I'm choosing to use this now just to support that for a broader use case. Then I just wanted to show you using that with the UI. Um, so we've added a few different imports, right? So, those two data models and then this link chain trim messages so that I can extract the message history. So, I've imported those And um within here, I'm going to go ahead and do a few things slightly different. Oops. See, just bear with me on the auto completions. Matt, as you type this, I might just take a question that we have from the audience, and I can answer this as you type so I don't distract you, but the question is, how can I integrate an existing agent framework that I'm using without lang chain? Do I have to redo everything with link chain? And the answer is no, you don't have to redo everything in lchain. Um, the IQ toolkit is framework agnostic, which means you can build to link chain. You can build a llama index, curi, etc. Those are things we support out of the box, including semantic kernel. But if there is another agent framework that you want to bring to a to the IT to the agent intelligence toolkit, what you can do is either just bring it in wholesale as a copy paste just like Matt did in the earlier demo or if you want to actually extend the toolkit to support your to another framework that you may be using. We actually have a great blog about that as well recently where we extended it with the AGM framework and we can share that with you in the chat as well as on the discord so you can understand how to extend the toolkit further. Go ahead Matt. back to you. Thank you. Oops. Um, so what I'm showing here is, you know, we now have a different data structure. And if you've spent much time uh looking at the OpenAI uh spec, you'll notice this looks pretty familiar. Um, so I'm actually going to what I'm going to do now is I'm going to look at port 3000, which I have the UI uh on, and I'm going to ask it the same question. This current Right. So this will do the same thing but using our UI now. Right. So we have a UI that's up that's communicating with our micros service using you know that's open spec compliant and it gives us the same kind of answer. So this is actually pretty cool. Right. So now we brought in the UI. You don't have to use this UI but it's um included in the um external section of the library. So you're welcome to use that or extend it. um or bring your own UI. Um so what I'm going to do now is I'm actually going to say, okay, uh there's a much fancier use case that we can do, right? There's a much fancier one that we can do. So I'm going to get rid of all of this and I'm going to now open up this pre-built demo that we have. This pre-built demo is a much more complicated DML file. Here we have many different LLMs. We have two, one that's using a NIM and one that's using OpenAI. We have an embedding model that's a NIM and we have a number of different functions here. So here I have some in-memory uh retriever setup that are hosting uh vector databases for a fictitious company called uh GPU experts incorporated with various you know HR policies dress code payroll PTO. Um I've given it a bunch of uh simple math tools. I've used the built-in React agent to the toolkit. Right? So one that we didn't implement and we've given it access to the map tools and the multiple rags. These rags are actually implemented in llama index, right? And then so now we have two agents as as functions or tools. And then we're using going to use our kind of userdefined agent and we're going to pass it two different agents and then kind of a notional uh datetime uh tool that we have defined above. And I can now bring this up in the very same way. Right? This plugin lets me use others work. And uh this is we're getting pretty close to the last section of the live coding here. Um I can now bring this up in the same way. And so Matt mentioned the pre-built agent. There's a handful of those that we have. So there's the React agent, there's a straight tool calling agent, there's a reasoning agent built into the toolkit that uses a reasoning model to create a plan. So you can have complex agentic systems like Matt's showing and wrap them around with, you know, a reasoning model as well. Um, and those pre-built agents, right, they're they're implemented within the toolkit in a specific framework and then you can bring your other functions and tools in in other frameworks as well and they'll work just flawlessly because of how we have that interoperability between frameworks. Thanks JD. Um, so what I did folks, I don't know if you saw it was pretty fast. Um, and it's kind of hard to see here in this kind of limited UI, but to keep everything together, I'm using it. Uh, is this chat history. So, I'm turning this on so I can take advantage of that new data structure that we're using. Um, I now have spun up that uh additional more advanced multi- aent that's going to be performing uh rag and a number of other things. And I'm going to ask it a question. Um I or I'm going to tell it I'm a a uh new employee at GPU Experts incororated. Right. So it's just going to notionally respond to us. The nims might be a little bit overloaded on the build right now. So, we'll just bear with this for just a moment. And while that runs, I'm going to uh explain a few things, right? Um, so we'll notice that we're using this description field in order to give this agent uh, you know, give give a tool calling agent information what this about what this agent is capable of. So we've used this generic uh, React agent. We've given it a series of tools. So this right this description lets us know what this tool is capable of and we follow that pattern kind of throughout each of these, right? So we're now using these in uh, in those scenarios. Um, so it's given us a little bit of information uh drilling in on some of those uh uh rag systems. Let me just ask it a specific question about uh uh dress code. Please tell me about the dress code. Right. So what this is going to do is it's it's going to invoke our multi-rag agent. It's using that description in order to identify that it needs to use that. It's going to give us the response at some point here. Matt, how does it handle UI interactions? Um, is there a connection between the open source UI and the toolkit library? So, the UI is a separate subm module, right? All right. So, it's a subm module or it's a subm module within the toolkit library. Uh, so we go to GitHub uh and you take a look at the uh library setup, you'll see this external folder, right? So, I've cloned the GitHub repo repo and you'll see that this uh AIQ toolkit UI um will be in there and that's its own GitHub project and you can navigate to that. Um, and that's actually what I'm using here uh to bring this up. Um, so it's a separate project, but we include it because it's, you know, really nice to be able to hook up a UI to this and give folks a starting point. Um, so that's what we did here. And then the last piece I'm going to show before we stop here is how I can spin this up as an MCP server. So I think I heard a question about MCP earlier. So if I wanted to do this instead uh as MCP and Matt, really quick, I might answer a question while you type this out. There was a question about if we're using short-term memory to combine previous interactions. And correct me if I'm wrong, but I believe how this has been implemented is the chat history is stored as a list inside your function, right? And so every time you have a new chat come in, it's appended to your list of interactions and then sent into your agent. However, the agent intelligence toolkit does support long and short-term memory as a module, including support for hosted um SAS type memory providers such as Zep or Medzero. Um, does that sound about right to you, Matt? That's well said. Um, yes, there's, you know, great degree of flexibility. Um, the way that the UI kind of supports, uh, sending messages to the running workflow is through that open AI uh, schema and the list there, which is what we were taking advantage of if we recall the data structure in the workflow. Um but we yes just as you said D we do support those uh memory structures. Um okay so it was really easy to bring up a micros service uh fast API microser to be specific what we did before. Um so we're going to do now is we're just going to create an MCP server. That's what we did on the left. So I used the AIQ MCP command instead of AIQ serve and that brought up our entire workflow as an MCP server. Right? So if I can I can run this and I can take a look at all of the tools all of the MCP uh servers that are or tools that are available via that server that we just brought up. And we can see that everything in our configuration files now in MCP uh can be accessed as an MCP server. Right? Right. So I just want to show the flexibility with just a different launch configuration uh or launch settings that we now can support building MCP servers. We can also connect to uh MCP tools uh from you know MCP servers, right? So we can basically have uh functions here serving as MCP clients and fold those in just as trivially. Um, so I think that concludes this the section of the live uh coding example. Hopefully this was a pretty quick onboarding, right? So just to summarize, we uh showed you how to bring in a uh simple lang chain react agent wholesale without opting into the rest of the toolkit. We then showed how to take that and then opt in to make that more flexible. Um and then uh we showed how to bring that into a separate uh separately defined um workflow and then use that in a pretty uh a much fancier uh multi- aent setup and took advantage of this plug-in system to do so. Um so anyway, that's what I wanted to share and I might hand it back to you, Truth. This was awesome, man. Thank you. Um it's the best onboarding I've ever had. But so really quick, I'm going to come back to to you all. um in a couple minutes to to ask you about maybe what you'd like to see out of the hackathon and if you have any um sage words of advice for for the people who are participating but in the meantime if I I could share my screen really quick I just wanted to give folks a really quick introduction into some of the resources they have to run compute um like Matt said and I'll wait until you're you stop sharing that but like Matt said the agent intelligence toolkit is is an incredibly light library um which means that you don't need a GPU to actually run the library. You can run it. I run it all the time on my MacBook without a GPU on it. And so you don't really need it to to run um the agent intelligence toolkit. But the question then becomes where do I put or host my LLM? Is there a place I can run um my large language models to run my agents? And one of the resources that we provide is the uh it's called build.envidia.com. If you go to build.envidia.com nvidia.com and you sign in and you create an account, you're able to look at any of these nims or NVIDIA inference microservices to be able to to run large language models. So you can click on any one for example our new newly released llama 3.3 Neatron. And if I wanted to use it, all I need to do is create an API key. Um by default there there are no limited number of credits that you get, but you are rate limited to about I believe 40 requests a minute with this. And what you can do is create an API key, use it just like Matt did where you export it in your environment variable with a config and that way you can use an LLM without having to host it on your GPU. So that is a resource that you have as well as of cou