Anthropic function calling for structured LLM outputs

Key Takeaways

hey this is Lance from Lang Shain so anthropic just released tool use their API and I've been really excited about this for a while so let me kind of walk through the implications and and how it works so first all you need to do to get started is just pip install Lang chain anthropic um and I want to talk through how this is really useful for structured outputs so let's just kind of talk through it how can we use tools to produce structured output I kind of think about tool use uh or function calling as interchangeable terms and the point is it basically invokes the LM to produce a payload and this payload is usually a Json string it can be passed to an API like web search for example it could also be passed to a parser to produce a structured output like we're going to show here so Lang chain has this with structured outputs method which basically binds um uh tool use for structured output generation to an llm um you pass in a schema and you get your your object out so let's actually walk through what happens under the hood um you have a schema let's say we're doing code generation we want the output to have three things a prefix a code solution and imports okay we can Define that as a pantic object with those with that particular schema we pass a question this object is converted into an internal function scheme which is basically like a Json which is then Plumb to the llm so the LM knows about this function or this tool that it can invoke so it gets the question it ideally invokes the tool that returns a Json string we also using this method take that object you define for your output schema and we create a parer from it we parse that string to the output that's the flow so let's just show how this works a couple easy Imports here's our data model right we talk about that code uh schema we want prefix Imports code all strings we kind of describe what it actually is now here's our llm we're going to use CLA three Opus and this little header just indicates that we want to enable tool use um so again we call the llm with structured outputs um and we're include or we Plum through the schema we defined right here as code I'm also including this include raw flag which will basically you'll see returns both like the raw model output so you can kind of think of it as this thing it returns that and it Returns the par output and it returns it returns any parsing errors that's very useful trick so let's kick this off this runs now may take a little bit we're using CLA claw 3 Opus so it's like the most performant model and we're just asking a simple question write a Python program that prints a string um uh hello world and tells me how it works in one sentence so in this case I can see kind of the raw output you'll see something interesting initially anthropic does this thinking step um which it kind of reasons about the question then anthropic you can see here this is actually the tool call and so if you look at this here's that Json string prefix code Imports like we had talked about above so that's pretty neat you can actually look at this raw intermediate object which is this thing and see that intermediate Json string that the model is returning then we go ahead and see hey were there any parsing errors we see no we can then fetch out that Parts result and then there it is so this is pretty cool there's our parse result now this is the pantic object we want out that's really it let's showing everything working end to end which is really cool and you can see we kind of did that all in like minutes let's look at more challenging examples this is something I've actually been working on quite a bit is code generation I have a bunch of different Lang graph code generation flows that typically involve some kind of inline or or in the loop unit unit tests so what I want to do is I have a code generation step that'll answer questions about some set of documentation like by context stuff typically could be L chain docs or other docs but what's interesting is I want the output to be this pantic object of like a prefix Imports and code just like we saw above and then I can apply unit test so this is like a real world motivation why I care about this and why I've been using it quite a bit um so I'm going to load those docs right here and um so this is just loading for example some documentation from Lang Cham and I'm going to show you something pretty cool here let me just kick this off and then I will uh go back and explain to you what's going on so here's a problem statement that's really common you see how above here we are counting on the llm to invoke this tool but that's not always guaranteed so we can do is note that in this uh output we get we get um this error so parsing error so if there's some problem with a tool invocation and this intermediate object that our parser attempts to parse is not parsable like for example what if it didn't call the tool so then it doesn't return to Json string it just returns like a blob of text it can't be parsed you'll see an error we want the ability to kind of catch that and retry so let's just show how to do that it's kind of a really nice little trick here um and we'll do it in a more challenging case so this is actually I'm loading a bunch of Lang chain expression language docs I have a code generation prompt that basically loads all those docs um into the context window and includes some stuff like your code assistant with expertise and Lang expression language um you know here's a bunch of documentation answer questions right so this stuff is same as before toine my output schema in this case I def I add a description this is the schema for code Solutions related to L transpression language which tells LM a little bit more about the tool U same as before same as before now here's where I can do something pretty cool remember that the model output has um is basically a dict and it has that parsing error field so what I'm going to add is a little function here so if the output has a parsing error I can fish out uh both the raw output and the error raise a value error um and then I can just like return the tool output so here's a trick I now build a chain which has our prompt our structured llm and this little check that's the new thing now what I can do Lang chain has this really convenient thing called with fallbacks so basically I can define a fallback chain which is basically just um this uh so basically this check occurs if this check throws an error I can go to this fallback chain which sucks in the error and basically automatically retries um and so I can Implement that all here where this is my code chain so we Define that up here um that code chain has this check at the end and I have this fallback chain if that check trips I go to the fallback and I basically I basically add messages to the LM saying hey um you're required to fix the parsing errors here's the error that you got back try again so it orchestrates that feedback so I can call code chain with fallbacks I add the fallback chain here we just ran that um and let's see what our output looks like so we can actually see um here's how to write uh a rag chain using L trans expression language um here's the Imports here's the code um hopefully the code executes um okay and so there's a pip install that I need to do that that is something that I would actually catch more in unit tests um but in any case you can kind of see that at least the pantic object was correctly formatted and returned um so that's really nice trick if you want to enforce the output schema um you can use this kind of fallback to catch errors in pantic validation or or and um or pantic parsing and to retry nice little trick there and again the arrow we saw down here is a really good example of what I see often when I'm doing kind of code execution in the wild that's something I would catch in my unit tests so I would attempt to execute the Imports if those fail I can retry the entire process or I can check um the code execution like we just did and retry there in this case that retry Loop would include hey your code's missing something you don't have this package pip install you know this guy and then my code execution would include that as an additional thing that is necessary um so so you know for example I was running if I was running this in a sandbox environment I could then run that pip install and it would it would update my environment and then it would run so you know nice example showing how to use anthropic with structured outputs like the vanilla case simple case up here and how to do it with this kind of retry fallback if there are validation errors um you can use what we talked about here this with fallbacks to basically catch any validation errors from Tool use and to loop back and to enforce that the tool is actually invoked um so that's about it hopefully this is useful thanks