Single Step | LangSmith Evaluation - Part 25

Key Takeaways

hey this is l l chain we're continuing our lsmith evaluation series talking about agent evaluation so in the last video I kind of gave a deep dive into General structure of agents and Tool calling and um how agents can be laid out um so we talked about Lang graph agent and this is kind of the classical kind of langra agent layout where you have a tool calling llm with some bound tools input the llm makes a decision to use a tool or not if it uses a tool that goes to a different node this tool node that actually runs the tool Returns the tool output to the llm and this continues until LM just returns a text response and doesn't choose to call tool anymore so that's like a very basic agent architecture you can put together with Lang graph and we talked about three types of eval so one type of eval is evaluating the final response which is we talk through in detail now the second thing you can do is think about evaluating a single step in this agent so typically what this looks like is you might evaluate given an input do I call the right tool so let's show how to let kind of blid that out again when we look at tool use right really what you're doing is you have an llm you're binding a tool to it or so a set of tools typically and given an input you're just kind of checking whether or not the right tool is called so this is actually really useful especially because so many different llms right now have tool calling you may want to evaluate like how well is it actually working in my case with my tools so this is a really good eval to think about now this is The Notebook we already talked through so just in case you hadn't seen it yet this basically sets up uh a lang graph agent for SQL so I'm using chin chinuk DB um so this is kind of all the code we already talked through this is basically setting up my Lang graph agent this is kind of defining all the different tools I want so I'm using the SQL toolkit um and I'm defining two custom tools right here this query Checker uh and then this result Checker so those are my tools I Define my state I Define my agent and again we talked through this already so you know you can proceed the prior video if you wanted to see a little bit more explanation here lay out my graph and there you go you can see the graph we always showed how you can run this as a test now we just talk through response evaluation and for that we built a data set of question answer pairs okay so that's what we have now let's talk about this single tool evaluation so for this what's kind of nice is I don't even really need my agent all I need is basically just that llm tool invocation I can test that in isolations that's a nice little component I can I can test so this function basically just wraps that assistant runnable is basically you can go back and look that is basically just our SQL agent so or or sorry I should be more clear that is the llm that's part of our agent so that is this piece okay llm tool to use and you can see you you put it in our agent like this so you have the llm it gets a prompt it makes decisions whether or not to use a tool and Lang graph orchestrates this by basically connecting this llm piece to a tool to node so this is like kind of your assistant node tool node and this goes in a circle but the key thing is that llm node needs to be able to use choose the right tool at the right time so that's something you may absolutely want to evaluate okay so that's that's what this runnable is this is basically that llm node and isolation first I def find this very simple function that wraps my assistant runnable which we just talked through so again that assistant runnable uh is basically just going to uh invoke tool use or not so I'm going to basically pass example questions from my data set which we talked about previously so I have five different questions related to my database I'm going to pass each example into my runnable right here and I'll get some result out now again if you look at our diagram here that result is either going to be a tool call or just a response of some type okay and all I'm going to do is in my evaluation function I'm just going to go ahead and grab that response so that's what you see here and I'm going to extract the tool call if there is one now there may not be but that's fine I just try to catch it so if there's a tool call I get it and I compare the tool call to what I expected that's all I'm doing one if if it's correct else zero so this is a really simple way to build a custom evaluator that basically looks at whether or not the tool called by your llm matches the tool you expect and this is very generally useful you can use this with you know many different llms that offer tool calling um so I can go ahead and kick off an evaluation so again I'm just passing in my function here I'm passing in my data set name I'm passing in my custom evaluator that's all I need I go to my data set look at my experiments great and ran so I can look at all my examples and here I can look at this is just like the raw output of my of my uh llm and you can see here's the input the output is indeed a tool call and it's calling the SQL DB list tables which is exactly what we want so basically the way to think about this is for our particular agent for any input question the first thing we've instructed to do is hit the SQL DB to list the tables okay so for all cases we expect this is the tool that it'll call so that's just a really nice simple thing and I've hardcoded that in my evaluation function if I go back and look I can see one one one one so these are my single tool call scores and they're all correct so basically my agent or in this case it's just my llm I'm testing in isolation is correctly for every input question calling uh that SQL DB list tables you can see them all right here so this is a good example of a really simple uh single step evaluation you can do um in my particular case I always expected for any of input question that this tool will be called and so that's why I can hardcode it now you can also be more sophisticated here because you could put this tool name in your eval set as a reference and then for if for example if different questions or prompts basically elicited different tools you can encode all that and very simply run that evaluation as well so this is showing like the simplest case where you hardcode your tool call but you can absolutely do more sophisticated things like basically put this tool call in your uh in your data set as a reference you could run the eval that way so again really simple very useful eval and very useful for evaluating different llms that all offer tool calling thanks