Generating reliable structured outputs with Trustcall

Key Takeaways

going from unstructured text to a structured output is one of the most popular use cases for llms and often times we want to use llms to generate complex schemas so here's just a toy example of a semi- complicated ptic model that we want to extract from some unstructured conversation now if we can take an llm and bind that schema as a tool so this is a technique for extraction using tool calling and what's nice is it can be used with any model provider that supports tool calling and let's test this out by passing in this convers ation and ask the model to extract the conversation into the schema that we Define that finishes quickly we can very simply extract the tool call and test it to see if there's any validation errors and we do see a validation error here now the problem is that complex nested schemas are often difficult for models to extract directly via tool calling so that's the motivation for this Library called trust call which was developed by Will here at Lang chain so you basically can import an extractor from trust call pass in an LM just like we did before as well as the same name tools and Tool choice to enforce the tool call run it we can rerun validation we can see validation is now successful so I'll talk a little bit about this Library why it works and some of the clever things you can do with it so producing structured output is a very common and popular use case as mentioned but it has at least two problems populating complex or nested schemas as we just showed but also updating existing schemas without information loss turns out to be quite a challenge so you think about three separate problems that you want to solve with respect to structured outputs one is schema Generations so going from unstructured text to a structured schema another is schema updating so having existing schema adding information via unstructured text and updating it and the third is a combination of the two so generation and updating so imagine we have an existing schema unstructured text we can update that schema and also generate a new one now part of the motivation for this and where we've seen this play out quite a bit is in memory so if you use open AI for example you notice that it'll save memories sometimes it'll update existing memories some sometimes it'll create new memories and that's exactly this third scenario we're talking about here the ability to have a collection of things could be memories and given new information make targeted updates to existing items as well as create new ones if appropriate but in the context of maintaining memories it's also the case that we often want to just simply update schemas and occasionally we want to generate entirely new schema in isolation so all three of these scenarios are fairly common when working with llms so the central intuition behind this trust call library is of a Json patch so that's just a very simple way to update add and delete items within a Json structure but importantly without replacing the entire document that makes it cheaper also makes it less likely to have undesired deletions now let's see how this works in a few specific instances by walking through the code and looking at traces we previously looked at this first example of schema generation we showed that raw tool calling failed validation we showed that trust call was able to successfully generate this scheme but I want to show it happened a little bit under the hood so right now I'm looking at the trace for trust call which we ran now you can see initially we only have our schema bound this is the bound tool it's called as we expect here is the input conversation here is the attempted extraction into the schema now what's neat about trust call is it'll internally attempt a validation of our schema so you can see this validation is attempted and there are some errors we can look at the full Trace here and this is where the magic comes in so trust CLA and built-in tools that allow for Json patching you can see right here this patch function errors is called respond with all Json patch operations required to update previous invalid function call so you can see exactly what's happening right here so basically it's looking at that trace and attempting to produce a Json patch to resolve it so very intuitive attempt a validation and then perform Json patching to fix the validation errors so performs the update and then performs additional validation validations pass and we finish well that's why with the raw tool call we saw validation fa as shown here with trust call we very simply wrap a few useful things into this extractor initially we attempt to perform the extraction with our schema but we check for validation errors just like we do here we then look at the validation errors and use that to perform Json patching to fix them that process occurs iteratively within trust call and we get a successful output so let's talk about updating schemas here's an initial schema for a user profile and here is our initial user named Alex with some favorite shows movies books and a bunch of hobbies and so forth there's our schema we want to update the schema with information from a conversation between Alex and a friend so here's our conversation and the naive approach of course is just tool calling so we're going to go ahead and bind our schema to an llm and we'll pass in that initial schema as user info pass in the conversation and just say update the schema based on the conversation using tool calling that's it so here's the result now we can very simply compare those two just pass them into our chat here let CLA analyze them you'll see something kind of interesting the show Mandalorian was included in our conversation but it was not actually updated in the schema we can confirm that quickly by looking at the schema and look at shows and again we don't see the Mandalorian added we can look at the trace here and just dig into a little bit more what's happening so again we're just basically calling this user tool with this big prompt which has the existing schema as well as the conversation and we'll let the model attempt to one shot generate this new schema and as mentioned sometimes things can be missed like for example we missed the show the Mandalorian mentioned here I'm into this new series but it is not reflected anywhere else in the update now let's try trust call again we're creating extractors like before pass in our tool and Trustco allows for passing messages just like we saw before but also we can explicitly pass the existing schema with this existing keyword right here in that case we just pass in the user schema we can run that we can very quickly look at shows and there we see the Mandalorian which is the show that was missing previously and again we can do an analysis of this we can see that we indeed remove stranger things just as expected and we added the Mandalorian because in conversation Alex mentions lots of interest in stranger things and really enjoys the Mandalorian so it's kind of cool we can see here's the trust call Trace we can look what happens it actually calls this patch do tool now before we were actually calling the user schema tool and trying to update the schema in one shot with Pat talk we're actually selectively applying patches to the existing schema to update new information so you can see here's our conversation past explicitly and here is the prompt that trust call uses to generate patches generate Json patches to update existing schema and what's pretty neat is it generates these patches to update very specific things in the schema like remove certain Hobbies replace pets look at this that is our update on the show Mandalorian very cool and so forth so that's really the difference it's pretty intuitive what's happening is instead of trying to one shot regenerate the entire schema based on the conversation we actually use this patching tool to selectively update very specific parts of the schema that need to be changed now what's also kind of neat is these two ideas can be combined together so both generation and updating so here's an example of a schema person where we actually have a list so here's a list of people that were tracking here's a conversation now the only difference is we create our extractor but we allow for for this enable insert true and just like before we pass in our existing data this is our existing schema and the conversation and what's pretty neat is previously we had three people Emma Michael and Sarah and now based on conversation we add a new profile Livia with some information so we've basically created a new entry in our collection of schemas and we've also updated the existing ones and we can walk through all the various updates to each one we can see a number of things have been unchanged as expected added a few new points and importantly we added a new profile for Olivia now if we roll back through all I had to do for this particular case is just enable inserts that allows us to create a trust call extractor where we can actually add new elements to our existing schema as well as update existing entries alternative like we saw before with the user profile we don't pass that enable inserts in that case we just update the provided existing schema in place based on any new information and in the first example we again create a trust call extract just like the other cases but we didn't provide any existing schema in this case it will generate us a schema from scratch so we just covered really the three main use cases for trust call which is quite useful so schema generation particularly for cases of complex schemas using built-in validation and patching scheme updating again using patching as well as validation and combining the two together as we saw in that third case so we found this to be very useful particularly for things like memory where you want to maintain both things like like a fixed user profile as well as a collection of memories you can both update in place and add to over time so trust call is a very useful library and it's actually been pretty popular lately so we wanted to talk about a little bit more depth and encourage you to try it out if you're working at all on structured outputs so feel free to leave any questions below thanks