Build Agents from Scratch (Building Advanced RAG, Part 3)

Key Takeaways

hey everyone Jerry here from llama index and in the third part of this tutorial series on building Advanced frag orchestration I'll teach you how to build an agentic pipeline from scratch and so this is an important step towards teaching you how to compose your own agentic orchestration to really Leverage The Power of llm reasoning on top of any existing tools that you use whether that is an existing rag pipeline uh or a SQL database tool as we'll show in the example or really any other function this is pretty key to making sure that you don't depend on just out of the box uh offthe shelf modules that libraries like us provide like an out-of-the-box react agent and that you really understand how to really compose your own orchestration um so that you can customize it really adapt it to your own use case so let's get started so uh we all are pretty familiar with the existing stack of a rag pipeline now you know you take an aqu query feed it through your rag pipeline which which does retrieval synthesis uh reranking all these fun things and then you get back a response and so as we go from rag to agents it helps to think about as a progression of what exactly does that mean right and one aspect of this is that you can add agentic reasoning at the beginning of a rag pipeline um to basically you know uh have a rag pipeline be a tool within an agent so that the agent can act as a reasoning layer on top of this tool um to give you additional query understanding for instance given a multi-part or complex question uh an agentic layer like a react agent can do stuff like Chain of Thought um step-by-step reasoning query planning to break down a complex question into smaller ones that you can then execute against a rag pipeline you can also add agentic reasoning pretty much at any step in this process you can add agents in the middle for instance you could have agentic like reranking retrieval you can have agents towards the end after you get back the result you can reason about about it to see if you want to redo or retry things before you get back response the definition of an agent um hasn't been Consolidated yet but here we're really just saying that an agent is just using an llm for automated reasoning and Tool selection and we'll go over four different levels of agents in the next slide um to really show you a progression from very simple agents to more advanced ones also an agent um is uh that that uh can can have more tools than for instance justess a ragp P line right um Adent can be composed on top of any tools and drag uh for instance an existing QA pipeline is just one tool that agent can leverage um agents ideally should be able to actually uh select what tools they want to use throughout the reasoning process and then dynamically you know reselect and select until they get back um until they solve the task at hand here's a a diagram showing you a progression of simple to Advanced agents on the left side of the spectrum um you have a an agent that is simple lower cost and lower latency and on the right side of the spectrum you have agents that are more advanced uh cost more tokens and of course are going to be slower so the most simple format agent really is just something like routing right a routing basically using an llm given a query to decide what Downstream module to pick so it's really just like a multiple choice selection problem that still is a layer agentic reasoning because it level it adds a level of um uses the llm to actually dynamically reason about what choice to pick given the input question some steps beyond that include you know um being able to decompose a question into multiple smaller ones over a subset of tools includes tool use so actually being able to infer Arguments for tools and being able to use those uh and it also includes more involved agents uh where we actually get into Loops like react is a classic architecture uh which reasons about uh which can reason about a question and do tool use in a step-by-step Manner and maintain conversation history throughout along the way there's also more agent papers coming out these days a lot of latest research focuses on Dynamic long-term planning and execution and how do you solve very complex tasks um in um well right and and kind of you're able to plan ahead instead of just reasoning step by step so in this slide we show you four levels of agents uh the first level is tool use which is more on the left side of that spectrum of something that's very simple not super complicated um but it's just a matter of given a query the agent can decide what tool to pick and the arguments to pick it with um and then that tool is selected it's run and the response is given back so that's a very simple example of an agent and actually routing Falls within this category because routing really is just tool selection across a set of choices and you pick those choices and you return the results uh Second Step is you know independent of tool use you can also Define an agent as having some sort of reasoning loop with uh conversation memory so if you have like a conversational um uh chatbot where you not only uh input like a oneoff query but you actually have a conversation over time you notice that it actually maintains the state of the earlier conversation and so instead of just having a dag where given an input you get back an output having it persist State over time so it actually Loops you know um uh so that there's a a loop from from the output back to the memory module uh for the next conversation um this is an example of something that is also agentic three is of course you combine the two so you combine tool use with some sort of reasoning Loop and memory um and so given a given a user input um you might want to uh one select the set of tools to use and then actually you kind of want to go back and see if the task has been solved right and if if it hasn't been solved you want to continue doing some light reasoning uh tool use um until you actually you know uh store enough stuff in the memory in the conversation history to feel confident that you have the right response and so this is something that basically this is basically what a react agent is uh right it's it combines you know the Chain of Thought stepbystep in with tool use uh to try to solve a task at hand and then number four is what I very informally defined as a fancy reasoning Loop uh plus memory Plus tool use it's just a step up from three and really the main difference is how fancy that reasoning layer is if three we count as step-by-step reasoning so for instance react agent is just you know um given the current conversation history just infer what the next question is um the this The Next Step Up is is just uh instead of just planning the next step can we actually plan out an entire query plan right like an entire dag of of actions to execute in order to achieve the task at hand um and a lot of recent agent papers are actually focused on this this includes papers like uh plan and solve or plan and execute um and this also uh includes papers like the recent LM compiler paper whose author is we hosted on our webinar so in the last two videos um we showed you how to build query pipelines and so far we've mostly used query pipelines to build dags uh this means that these dags do not have loops in them um the input uh just flows or flows from one end of the dag to the end from the root node to the leaf nodes and you get back an output and of course like the Intermediate steps the intermediate modules uh process the inputs and give back outputs this captures workflows like prompt training as well as the rag pipeline which we showed in the previous videos um as well as you know the simple Advanced Texas SQL demonstration this also includes simple agentic reasoning capabilities such as routing as well as one-step query decomposition what this doesn't capture is because there's no Loops in the dag uh by definition um we aren't able to capture some more sophisticated agentic reasoning so for instance if you wanted the output to actually flow back to the input um and actually run a while loop until the task is complete you're not able to do that yet uh the next thing that you're not able to do is uh that we haven't demonstrated is actually being able to maintain State because if you have a while loop flowing back through the input uh you'll you'll want to have some sort of internal State that's modified over time and so far you can think about a query pipeline as stateless so both of those things will change with um this video so in this video we'll show you how to compose an agenta query pipeline where you can execute a query Pipeline and a loop um this query pipeline um will can will be plugged into to what we call an agent worker and it's a special type of query pipeline that actually carries State and isn't stateless and by plugging it into an agent worker you get a lot of the benefits of the Llama index agent abstractions which let you execute a task End by end at end to end or step by step um and of course this special type of agentic pipeline also maintains mutable States throughout the query pipeline so that once you you know Loop through it multiple times the state can actually change and mutable State includes stuff like conversation history memory and other variables that you want to toggle so without further Ado let's go through a notebook guide um note will be using the very recently released llama index v.10 which if I go to this blog post uh we released this on uh Monday February 12th um and so what we did was we really um packaged uh just the core modules and llama index core and split off every other integration into separate packages um and so we did this work to try to make every single integration and um within the repo uh wellmaintained versioned uh and independently updated so that um this makes it that makes the dev ux a lot easier and should make the package less prone to breakages uh this did lead to some changes in syntax and I'll show you that updated syntax in the notebook walk through so now let's walk through a notebook example of how to build an agent using our query pipeline syntax specifically how to build a react agent if the first two videos showed you how to build a dag here we'll build you know a dag but actually plug it into an Adent worker that will Loop through this tag and actually carry states through throughout so what we're going to do here is we're going to build an react agent from scratch so not using any out the box modules and using a SQL database tool so this agent will learn how to will given a task uh query a SQL database to give back the right result so we're just going to run through some imports um and we'll use the chanuk database um this is uh a very popular sample database for testing you know SQL queries and we'll import uh the query pipeline uh we'll be using the open a llms so we want to pip install llama index lm's open AI this is a V10 feature where you know the openai integration um has been split off and so you want to pep install it um the Llama index package includes it by default but this is good practice anyways and this just download stream here we'll just set up a global callback manager um this will help in case you want to plug this into any Downstream observability module now in this case let's set up a text to SQL uh query engine and Tool this is the core tool that the agent will use and here we're not going to build this from scratch we're just going to use the pre-existing query engine but this uh just takes in a natural language query um and under the hood translate it and and translates it into SQL um executes against a SQL database and gives you back the result uh if you want to learn how to build this stuff from scratch and also make it more advanced check out our second video uh in in our video series notice that we also wrap this SQL query engine in a query engine tool and we give it a name called SQL tool and then the description is useful for translating a natural language query into a SQL query uh this is just basically some sort of prompting to help the agent use this tool and and know when to use it great so let's get to the fun part of setting up a react agent pipeline so we'll set up a react pipeline um to using our query pipeline syntax and we'll both execute a step by step and also execute it end to end we're going to use a few fun components uh like agent input component as well as agent function component um and some other components aren't used in this notebook include like a custom agent component um these are agent specific components that can actually carry state um and this is something that is important in building an agent because you want the state to propagate throughout the execution of an agent here we'll just import the query pipeline um notice that we Define a blank query pipeline for now and we'll add modules to it sequentially and the first step is to define the agent input component um and so the agent input component is something that's called at the beginning of every agent step um and besides passing along the input we'll do some sort of basic initialization in state modification so scrolling down here just moving past all the Imports here is our agent input component um this is the class and it takes in a function agent input function if you go to the agent input function this is a user defined function and it takes in two variables a task and a state the task is just an object that contains the overall task information um so uh basically the way we think about it is when a user wants to execute a task against an agent uh the agent will create a task uh containing a lot of the metadata it can execute step by step to achieve that task and might take multiple steps to achieve the task um so so this contains both the initial user input uh memory M as well as any kind of global State over time this is State that's passed into the asent input function at the very beginning of execution the state will be empty um throughout the course of execution on subsequent steps the state might not be empty and you'll see that in the logic here in the execution we'll say if current reasoning is not in state then we'll initialize that in state so on the first run on the first step this is true and so we'll want to initialize it on subsequent runs current reasoning will repopulated and actually we initialize current reasoning with what an observation reasoning step this is just a very light wrapper um that's uh a light internal abstraction uh to just you know contain the initial task input so State current reasoning. append reasoning step the output of this is just um a dictionary of input keys to inputs you're really free to um return whatever you want in this in that sense it's actually very similar to our regular input component in a query pipeline um and the main thing is you just got to make sure that whatever input you return you can wire these inputs um to the proper Downstream modules throughout the rest of the query pipeline so we did some basic initialization which just amounted to putting the task uh input in this query Uh current reasoning list and then the next step is to define the agent prompt um after you takeen the input uh what we want to do is we want to call the llm with the react prompt um and so this module does that we Define what we call an agent function component um an agent function component is similar to our default function component in our query pipelines allowing you to pass in any arbitrary function right and the main difference here is that um uh the the function component takes in a task a state and basically after that can take in any set of arguments that you want so it's very similar to a function component where you can define an arbitary function um the main difference with the function component is that these two variables must be defined uh in the first two slots this is what allows you to propagate States um uh throughout the agent because it passes in both the task and State variable so you can again modify the state throughout the course of execution if you want So within the react prompt function we call this uh internal module called react chat formatter um all it does is it has a little prompt template inside and given the right set of inputs it'll return the fully formatted react prompt so um the input to this prompt function is uh some input string as well as a set of tools uh we'll learn you know how to wire in these inputs from the uh agent input function in in the next section but given these inputs um it'll fill this in with tools chat history current reasoning and give you back a format of reactr and so we'll execute this um this module but actually just to show you what the react formatter looks like with an empty chat history and current reasoning uh it looks something like this here is just the overall system prompt um you're designed to help with a variety of tasks uh you have access to a wide variety of tools really because we pass in one tool there's one full pass here and we see here is the name that we pass to this tool here's the description to this tool and then here is the arguments needed for this tool this is represented in the pantic schema and we see it just takes in an input with type string and here's a classic uh react output format you know either return thought action action input uh or return thought answer if you're actually done so we're just telling the this this react pump is just telling the LM you know either call a tool or say that you're done and return the answer because we left the conversation uh history as blank you don't see the stuff below but um if the conversation history is not an empty then we actually inter leave the user and assistant messages underneath here um and so the inut to the LM is actually a list of trap messages with both the system prompt but also intered user uh user assistant messages from previous combos after calling the react prompt uh you're left with a choice because again the react prompt or the llm can either do thought action action input if it wants to call a tool or say thought answer if it's done um and here we'll show how to parse that output and basically kind of do an IFL statement to send you know the input down a separate path of this graph depending on what uh the output of the react prompt is that is in what we call this react uh parse react output function um where we Define another agent function component to call uh this again this module called react output parser all this does is parse that prompt here and parse it into a reasoning step um this reasoning step is just the object that contains uh is done property um as well as um if you know this is the final answer it'll be a response reasoning step and if this is an intermediate answer it'll be an action reasoning step um so it returns this output in a dictionary so whether or not it is done whether or not the agent is done and um what the reasoning step is we'll see below that we actually given the output of this uh of this function do what we call a conditional path so a conditional link to two different nodes and so instead of sending this input to both nodes at once we'll only do one or the other depending on the condition and what this condition is if this re reasoning step is not done it means that it wants to call as tool and so we'll actually Define a run tool function um and this tool run tool function will take in this reasoning step which is an action reasoning step and and this action reasoning step contains an action action input and we'll pass that to this tool Runner component to just you know call the tool with the action input um get back the output and append that to the current reasoning so this is the process of you know processing the action inputs calling the tool um and updating the conversation history on the other hand if the agent is actually done we'll actually have a separate component where the response step is assumed to be a response reasoning step and we'll append it to the conversation history um but actually just append everything to the memory um so there's like a special variable called memory uh where um this is just something that's maintained throughout the course of a task um and actually of an entire agent and we append the entire conversation into the agent memory um for use with the next uh execution um so memory persists across different tasks uh the conversation history is only for a given task and so now that we're done we just append everything to the memory and give back the final response we have one last component which you know depending on whether or not you call a tool um or you're done um and you want to process the response we just have a light converter that converts the response dictionary into a proper agent chat response and so um the final output of uh a query pipeline within an agent is an agent chat response um as well as is done flag so we just described a bunch of these different modules and it's it makes sense if you know at this point you're wondering how does this actually all fit together and so this is where we Define the links between these different modules to Define this agent query pipeline so we stitch together the top level agent pipeline um with links um so first We'll add all the modules to the query pipeline the next step is to add the links um so first we'll link the agent input to the react prompt to an llm to the react output parser this should be pretty straightforward but basically you know we take in everything um or or we we uh start off with the input component we take the outputs of that agent input component feed it to the react prompt um and and then we take the output you know the format of react prump send it to the lb get back to the output and then parse that output again into either a response reasoning step or an action reasoning step now we do something interesting which is the conditional link part we add a conditional link from react output to the tool call so here we see the react output parser to the Run tool component and the condition function really is if that output of the react output parser is done variable is true uh or or is is not true if it's not true then run tool this link actually runs uh and run tool runs with with the reasoning step as input there's another conditional link from the react output to the final response processing and this is toggled if you know the output of the output parser is done so the opposite of this link um if that is true then we also pass in the input X reasoning step so we see here it's really one or the other right either run tool runs if it's uh not done or process response runs if it is done finally uh whether or not run tool runs or process response runs um we're going to add the link for both to process agent response when you execute this tag uh or you know only one of these links will run at a time but basically both of these links will link to the same input key in process agent response um and whichever one runs will feed it then to the process agent response to give you back again the final answer which is a agent chat response and it is done so if we visualize this query pipeline it looks something like this right um you have an agent input feed it to the react prompt LM here's the output parser which decides whether or not to run the tool or process the final response and give you back the answer this is just uh one step of execution right so on on its own it does not Loop um and and so you know if we just execute this in one step uh and and uh it it'll give back an answer um but you know is done might be false so this is just a special type of agent Pipeline and to really you know take advantage of step-by-step reasoning and Loops what we want to do is set up an agent worker around this uh tax SQL agent query pipeline so now we want to set up an Adent worker around this Texas SQL query pipeline um and what we do is we import these two modules a query pipeline agent worker as well as an agent Runner we wrap the query Pipeline with the query pipeline agent worker and then put the agent worker within the agent Runner this overall agent Runner allows us to run this query pipeline either step by step or you know in a full loop And to clarify running one step of this query pipeline means running uh the entire dag once right and so running multiple steps of this query pipeline means running the entire dag multiple times every time we run the dag the state persists throughout the course of the run so the state continuously gets modified until uh is done is equal to True at the end so let's run this and then let's try running the agent on some sample queries so we'll first create a task called what are some tracks from the artist ACDC limit it to three right we create this task and then call run step we see that it's actually by running the step it's executing the entire dag uh on on you know in the first step and it runs through the modules agent input then the react prompt right so it passes the input to react prompt calls the llm goes through the output parser um you see that the input to the output parser actually is you know a basically infers um call the SQL tool with the relevant input select track name from tracks or artist name is there the output parcor does its job so you see the conditional link is actually working it calls run tool next and then process agent response so the current output from this uh agent is actually just the intermediate response uh where you know it's the observation step we can show that here um the step output has you know an observation um the output is just the tool output the top three tracks by ACDC or this but of course the agent isn't actually done with this task yet um because you know is done should be equal to to false let's just double check this really quick right it's equal to false so we rerun the step um this current observation from the tools been persisted conversation memory but once we rerun the step um we see it goes through the agent input calls the react prompt again of course with the tool output in the conversation history now when we go to the react output parser we see that here the agent essentially has inferred that the user has already provided this information in the previous observation so therefore we can give back an answer so the conditional link now goes to process response instead of run tool because it has all the information and now process agent response um has a response string and is done is equal to true so now when we go to is last we see that the output is true and we just need to call agent. finalized response and then we get back the answer so that's showing you the low-level agent API actually where you can execute things step by step until you're able to um uh until is done is true this has certain benefits of one debugging interpretability and also uh in future videos we'll show you how to you know actually inject user input in the middle of the state so that um you can actually guide the agent as it's executing uh the the task you can um also just run everything end to end so uh by just calling agent. chat you can just ask you know what are some tracks from the artist ACDC we'll reset this so it re-executes the loop um and you'll see it it runs two steps at once step one again runs the Tool uh with the inferred action goes back to the beginning of the dag runs the entire dag again and the Second Step goes to process response and the response is the exact same so I hope you enjoyed this video hopefully this gives you a flavor of how to compose your own Adent pipelines with our query pipeline syntax and how to compose you know agentic reasoning on top of any tools that you want um we'll try to dig a little bit deeper into what a jtech rag means in future videos so this includes you know routing uh query decomposition and of course back to this like react asent loop as well um but hopefully this is a good start so thanks and we'll see you next time