LangGraph Agents with Structured Output

Key Takeaways

hi everyone it's Isaac with Lang chain and today we're going to go over how we can add structured output to our react agents so to start off with we are going to do a quick refresher on what a react agent is and then we're going to explore how we can add structured output to it so on the left hand side of this diagram you'll see um what the architecture for a basic react agent is and this is a very simple architecture we're going to have one llm in the middle that agent node and all that llm is going to do is it's going to determine whether to call some tools or to respond to the user um and after it calls uh a tool uh those tool responses will always be forwarded straight back to the llm who will then be called again to determine whether it needs to call more tools or whether it can just respond if we want to structure our output what we can do is we can add a respond node right before the llm responds to the user and what this respond node is going to do is it's going to take the output of that llm and it's going to format it into the structured form that we wish to respond to our user with it can be really helpful to respond in structured formats because this increases the reliability of our agent because we are having the same exact expected output every time we run it and therefore we can use this agent uh in larger software systems since it's more deterministic and reliable um which is not always the case with llms so now that we've got a quick Refresh on what a basic react agent is and how we're going to add structure output let's go look at the two different ways that we can configure this respond node the first way we can configure uh this respond node is actually um by adding an extra tool to our agent llm and so you can see this blue box here the response format tool this is going to be a pantic model that we passed to our llm and it's going to be able to call that as a tool which is a very nice feature of Lang chain once it's decided to call this tool we're going to um follow that blue arrow we're going to format the response of that tool call and then we're going to respond to the user so the pros to this approach are that we only have to use one llm as you can see in the diagram which is going to reduce latency and costs however when we're only using one llm it can be a little finicky so the llm could fail to call any of the tools which is a problem because in this case the llm doesn't directly interact with the user it's only interacting with the tools so if it fails to call any tools uh We've run into an issue similarly the llm could call the response tool along with other tools and that could be a problem so these greenbox tools are intended to answer the user's question while the response tool is intended to respond to the user but what happens if the llm chooses both a greenbox tool and the response tool well we have to deal with that in our code somehow because most likely the LM really wants to find out more information and is not looking to respond to the user yet but there might be unique cases where it is trying to respond to the user or it isn't trying to use a tool at all and is just messing up so we need to be sure that we're careful with all those cases and these are some of the risks that can arise when using this option the second option uses two llms so as you can see our original uh llm that uses tools is going to only have those green box tools to it so only those tools that provide it information and once it's determined that it's done calling tools that it doesn't need any more tool calls it's going to route to the response node and that response node instead of just formatting the tool call uh like we saw in option one is actually going to call a second llm and this second llm uh is going to be written such that it's forced to respond in our structured format and this is a really nice feature of llms that we can force them to respond in a specific format so the Pros to this version are that we are virtually guaranteed structured output because we're always going to reach the second llm barring some really strange loop with the llm and tools and since this second llm is guaranteed to have the right response format our overall graph is guaranteed to have the right response format which is really nice however to guarantee this response format we're going to have to introduce another llm call which is going to mean higher latency and higher costs and in addition the first llm actually lacks the context of the response format and what I mean by that is this first LM here maybe we ask it a question like what's the weather and the response format say we want to respond with the wind speed perhaps that first LM doesn't know that though because we haven't provided it any context on the response format so when the user says what's the weather it only looks up the temperature or the precipitation or whatever and then it it finishes quering that information it thinks it's solved to the question and it goes to that second llm and now that second llm is trying to format this response with the wind speed but it doesn't find it anywhere so that's one of the cons uh in this format but now that we've ran through a quick overview of both of the formats let's dive into some code and see how these work in the real world all right so now we are in our code repo for the first option where we uh bind both of the the structured output and the tools to query the information to the same llm we only are going to have a single llm in this graph so we can start walking through the code first we're going to Define our structured output class in this case it's going to be a pantic model that mimics some sort of weather response this is a very simple vanilla example but you could make this as complicated as you want next we're going to define the input output and the state to our graph so the input is just going to be the messages State and that means it's just going to be a list of messages uh our output is just going to be our final response which is going to be of type weather response that structured output that we had above and then our overall state is going to contain both the input and the output state so it's both going to keep track of a list of messages by uh subclassing from messages State and then it's also going to keep track of the final response next we can Define our tools and our model so we're going to be using the python tool decorator here to define the get weather tool and the get weather tool can be written as a python function with the tool decorator right above which is really cool uh and makes turning any python function into a tool super simple um and a really cool thing here or at least something I think is cool is that we can bind both this get weather and this weather response uh to our model and the reason I think this is cool is because get weather was defined as a python function that we added a decorator to and weather response was defined as a pantic base model but we can pass both of those in and under the hood L chain is going to deal with converting those to the proper format for the llm to interact with which is really nice so now we have our single llm which both has the get weather tool and the weather response tool so it should not only be able to solve uh the user's query but it should also be able to respond to the user in the correct format so next we can Define our graph nodes and we're going to have two nodes we're going to have our call model node which is going to actually call this model and then we're going to to have our respond node so our respond node is just going to take the last tool call that was made from our model and that last tool call is going to be the weather response tool call so once we're at the respond node that means the weather response tool got called and we're going to take that the arguments from that tool call and we're just going to pass those to our weather response object and then we will return that as our final response next we can Define our routing function which is going to inform our graph what's step it should take next based on the output of the llm so if the llm outputs a single tool call and that tool call is the weather response tool call we're going to respond to the user and otherwise we are going to go use our other tools so in this case the get weather tools but any tools that are helping us answer the question and not respond to the user and note that we make that check for the tool call length being one because we want to ensure that the llm is only responding to the user it doesn't have any more information to get um and that is going to be the case that we respond so now that we've done all of that we can Define our graph structure which is fairly simple we're going to be defining a state graph with our uh afor mentioned agent state agent input and agent output we're going to add our three nodes the agent node the tools node and the respond node we're going to set our entry point as agent the graph is going to start by calling the llm uh and and then we're going to add our conditional edges from the llm to the should continue function which we just defined above and lastly we'll add it an edge from the tools to the agent so as soon as we call tools we're going to get that response right back to the llm and then we're going to add an edge from the respond node to the end of the graph once we hit the respond node we are done lastly we'll compile this graph so that we can use it in L graph studio so now that we're done with all that let's head over to the studio and see how this uh Works in action all right now we're in the Lang graph studio and on the left hand side you can see our graph visualization very nicely and we can input our question to our graph so we're going to ask it what is the weather in SF and we're going to see the graph running both visually on the leftand side and on the right hand side we're going to get a stream of the output so perfect we see that the AI first calls the get weather tool which then returns it its response and then it calls the weather response tool which rout it to the respond node and then our final response is exactly what we are hoping for this structured weather response object with the wind direction the temperature and the wind speed and now we can actually open this run in lsmith and verify that everything worked as expected so as it loads we can see that our rendered output was exactly what we expected which is great so the output from the studio matches the output in lsmith and then on the left hand hand side we can actually see the exact trace of what went on so we can see the nodes that got called so it called the agent node the tools node the agent node and then the respond node which is uh the flow that we were hoping to receive and then if you wanted to further debug what's going on you could dive into each of these llm calls looking at the input and the output uh we're not going to run through all of that right now but lsmith is a great place to do all of your debugging and it couples really nice ly with the front end UI that you get from langra Studio so now that we've taken a look at Option One let's go explore the second option where we use two different llms to structure our output all right so now I'm in the code for the second option where we're going to have two llms I'm going to skip over defining the weather response and our input output and graph State because those are the exact same as they were defined in option one so you can go back in the video to look at those if you want the first uh difference we're going to have between these two options is that we're defining two models here and so the first model is just going to have the get weather tool binded to it so as you can see our tools definition is now a list of length one and it just contains the get weather tool and then we're going to have a second model called the model with structured output and this model is going to use the dot with structured output method from Lang chain uh and we're going to pass the weather response and now this model is going to be guaranteed to always output uh a response in the format of the weather response pantic model so now we can Define our graph nodes again our Carl model node is basically identical to the first option except this time we're not using the model with uh the response format bounded as well it only has the the get weather tool we're going to invoke that and return whatever the result of that invocation is and now our respond node is a little more complicated so instead of just passing the output of a tool call we actually need to structure the output somehow so we're going to use that model with structured output and we're going to invoke it on the content of the second to last message so that second to last message is going to be the prior tool message in the conversation so if you remember the flow what's going to happen is it's going to go from agent to Tool back to agent and then to respond so once it's at respond that means that two steps ago it was at the tool node and so we can take that second to last message grab the content from that tool message so that's the output of the tool what did the get WEA tool respond with and then we just want to structure that response we have to make one little trick here by converting uh that tool message into a human message because you can't invoke an llm with just tool message uh the llm is going to be expected to be invoked with a human message then we can Define our routing function which is fairly similar to option one uh but in this case the check is a little simpler if there are no tool calls that means we're going to go to the with structured output node um and if there is a tool call we're going to use that tool and our graph definition is the exact same as option one so I'm going to skip over that and instead I'm going to head back to langra Studio to show you how this option looks on the front end all right now we are in langra studio and we can see again our graph nicely visualized on the left hand side and we can input a question to it and we're going to ask it this time what is the weather in New York and once we click submit we are going to see our graph running on the left hand side we can see the visualization of that and on the right hand side we can see the output getting streamed so we're going to see our AI is first going to call the get weather tool as we expected and then the second response it has there's no tool calls in this response and so that's going to Route us to the respond node which then calls our model with structured output and as we can see here it Returns the output in the exact format that we would expect so we can actually open this run in lsmith to verify that everything worked as expected and perfect on the left hand side again we see the exact uh sequence of nodes that we saw in the studio agent tools agent respond which is exactly what we expected and then we can verify that that output is uh in the proper format which it is in this case case and then if you wanted to do more debugging you could dive deeper into each of these traces and see what's going on that wraps it up for this video on structuring the output of react agents I hope you learned a little bit and I hope that you can use what you've learned in this video to build some cool projects with Lang graph have a great rest of your day