OpenAI Introduces Structured Outputs in the API (Tutorial)

Key Takeaways

now I'm going to show you here how to do it with the response format I have an example of that as well so this one is the one with structure output with response format and the way you use this well this is useful when the model needs to respond to the user in a specifi structured way so you know sometimes when you interact with the model it could be useful for the model to respond to you instead of like just plain text in a structured way and you will use that information wh however you want to use it right maybe you want to present that in a UI whatever you want to do with that information I think it's useful to have this particular capability asking it you're helpful mat guide the user to the solution step by step how can I solve the equation here and then I give it the response format and then I say strict equal true which enables the structured output feature and then I just gave it the Json schema so you can see that I've defined the steps then I've defined that this includes explanations and then output and then eventually I have a final answer which is of typ string so basically I've defined that Mony so this gives you a way to customize the schema itself and this is something that will be provided as part of the call to the model and the model will use this knowledge to basically go through the steps and so on so again this is just a neat way or useful way to allow the model to respond to the user in whatever structure way that the user wants and you define that structure here in the schema and here is the result of that so it says steps and then it gives me the steps here you can see that this is an an object and it has a list and then towards the end you will see that there is a final answer as well so I think this is pretty useful opening I just announced structured outputs which is now available in their apis so if you're a developer like me working with llms and so on I think this is an exciting update something that I've been discussing in the past few videos in my channel is this idea of structuring inputs for the model and this usually helps AIDS the model understanding better the task similarly with structure outputs what you're doing is you're essentially constraining the model outputs to a specific schema or structure that you want so rather than prompting the model a specific way on the output format that you want you would do this in different ways by defining what is a schema that you want and so on so it makes your application more robust more deterministic and more reliable so what we're going to do in this video is we're going to go through some of the details here in their announcement and also we're going to take it for a spin I'm going to show you some examples very simple examples of how to use this before continuing please consider leaving a like And subscribe to the channel that really helps the channel and allows me to keep doing some of these videos for you so thank you for that and let's get right into it so I mention here they're introducing structured outputs via the API right model outputs no reliability here to developers apply Json schemas Json schemas is really what it's all about here so the ability to define the structure that you want the model to Output whatever it is outputting right whether it be something that converts some structured data to some structured information whether it is in a conversation itself if you want something structured from the model you would use this particular feature and there are different ways how you can use it as I mentioned and that's what we are going to touch on today now I'm not going to go through all the details in the blog post is something you can do on your own I'm actually just going to be focusing on how to actually use it from like a very simple example to a little bit more advanced and interesting example that maybe might inspire you to start using this feature I believe most people that are using the apis today should use these features now there is a catch they have a component now that does the schema understanding and all of that for you to make the small as deterministic as possible so there is a latency penalty in the first call that you do and so after that call there is some caching mechanism that happens and then the model response um in the usual speed these are just the results they're pointing out here they mentioned that they're doing some evals and this is really important when you're doing structured outputs or experiments around this particular feature it's really important to even though they show that they can get like 100% on their evolves it's important to actually experiment and see what works for you because again you will notice that you are defining these schemas yourself and the way you define these schemas is you need to follow some sort of guideline or best practices and make sure a schema is as clear as possible for the model now this actually is neat because because you do less prompting and now you do more definition it feels like you're solving the problem in a more programmatic way which I really like anyway so these are the results this particular model structure output with strict equal true which just means that you're enabling the structured output feature and then you will see how it Compares with this one which is strict equal false we'll touch on that uh but you will see how compared to just prompting alone with GPT 40613 how it's even below 40% and this one is 100% you will see the differences here in performance now I don't think this is again too reliable or you shouldn't be reading too much into this they're just showing that it's very good on their evals that they're showing um evals in particular like complexation schema following type of evals but then for your use case you should be doing the same right because again the schema that you're specifying all the properties the all should be able to understand whatever properties you're defining in your schema sometimes when you're dealing with some IND domain type of use cases the model might struggle still with these ones so they say here how to use the structure up because there are two ways how you can do this from what I read there's a function calling method and then there's also using the response format uh they have a new parameter here called Json schema so this is where you specify your Json schema this is where you will do the work and this is what I said right you do less prompting and you focus more on specifying the schema and all the properties that you want in terms of the behavior that you want from from this particular model so let's get into some examples here just to kind of show you now I am reading through the documentation there are a couple of nice examples here I would highly recommend that you check that out as well I'll provide the link in the description just for you to go through this if you're looking for understanding better how to use this particular feature and what I've done is I've also prepared here a notebook this is a collab notebook I'll share also a link to this so that you can get started with this this is basically a getting started with the structure outputs I'll be adding more examples to this my own examples as well with my own use cases and I'll be building out this particular notebook so do stay tuned for that again there will be a part two to this because this is a topic I'm really deeply interested in I've been researching a bit on so this is the first example here now these are just like whatever libraries you need to install and then here you set your opening IPI key and once you have that set then you can do a first simple structured output example so the way you do this is quite simple actually you define the pantic object so we're using pantic here to define those objects that we're going to pass as part of the model request so here I'm defining calendar event and then I provideed a name then I provide a date then I provideed participants so these names are really important right so when you're defining these objects and schemas you'll realize that the best practice is to make sure that they are understandable for the model and the M can use this to accomplish a task that you're giving it to do so here we call the model and you will see that this one is GPD 40 2024 0806 which is today then this is the messages so we have the system role which is extract the event information this is a high level instruction then there's a user role Alice and Bob are going to ass science sphere on Friday this is not the instruction this is the input itself and so what we will expect from this model mod is that the model extract the invent information based on this particular object that we're giving it and you can can see that in this simple example we're giving it the response format which is this property here and we're passing it the calendar event okay so once we pass it the calendar event it's going to extract these pieces of information that are available in the input that we are providing it so if we run that you will see that the event itself holds that particular object so you can see that it name is science fair that looks okay date is Friday right and then it's going to look for participants and it out the participants which is Alice and Bob so that's a really simple example of the idea of structured outputs and this is really useful you're essentially helping and aiding the model to perform the task and you're doing it in a way that you're defining the object itself you're defining what you want the properties that you want instead of doing that using prompting which can be quite brittle and the second example here is really interesting so this one again uses the same concept now this one is a first they Define a class step and this step will Define the steps of this Chain of Thought reasoning uh process so the explanation and then we have the output and then we have another class right and you can see that we're using that particular um Step that we have defined here and the steps are going to be that and then finally we're going to have a final answer so what we expect from the model is the list of steps and then we're going to have this final answer which is a string type so once we do that then we can again call the same model here your helpful matth tutor guide the US user true the solution step by step and then I ask it how can I solve this particular problem here and now I can just request that from the model okay so what I'm expecting again is a list of steps and then a final answer so I've done that here uh here's the mat reasoning you will see that it has the steps so it list the steps in this array you'll see all the array and finally here towards the end you will see that it has the final answer this is the final answer which it extracted from the output here which is part of this last step okay so it does that in an intelligent way and again it prevents you from having to define the steps yourself and so on it just knows that it needs to call us test because you have defined this subject and that's what you're passing as the response format so you're structuring the outputs that you want from this model now I'm going to show you here how to do it with the response format I have an example of that as well so this one is the one with structure output with response format and the way you use this well this is useful when the model needs to respond to the user in a specific by structured way so you know sometimes when you interact with the model it could be useful for the model to respond to you instead of like just plain text in a structured way and you will use that information wh however you want to use it right maybe you want to present that in a UI whatever you want to do with that information I think it's useful to have this particular capability asking it your helpful mat guide user to the solution step by step how can I solve the equation here and then I give it the response format and then I say strict equal true which enable a structured output feature and then I just gave it the Json schema this so you can see that I've defined the steps then I've defined that this includes explanations and then output and then eventually I have a final answer which is of typ string so basically I've defined that manually so this gives you a way to customize the schema itself and this is something that will be provided as part of the call to the model and the model will use this knowledge to basically go through the steps and so on so again this is just a neat way or useful way to allow the all to respond to the user in whatever structure way that the user wants and you define that structure here in the schema and here is the result of that so it says steps and then it gives me the steps here you can see that this is an an object and it has a list and then towards the end you will see that there is a final answer as well so I think this is pretty useful now if you want to apply this with function calling I have also provided you an example here which I got from the documentation there's a lot of good examples here that you can track this one is useful when you're connect connecting M to tools as we know function calling is basically you are let's say having a conversation with a conversational agent and you ask it for something and then you have these available tools and the mod doesn't know how to respond to that but it can leverage these tools you can set up like these tools and Define these tools and the Mo knows when to call these tools collect the information needed and then you can pass that back to the mle and so the user that's interacting with this particular mole has upto-date information for instance if this is a conversation about the weather asking for the weather in a specific Place well the model needs to call a specific external tool and function calling allows all of that so the idea of using function calling with structure outputs is makes function calling even more precise and more accurate as we know function calling doesn't work so well like if you use it just a plain function calling feature and this has been a major complaint from a lot of developers that I know that I work with having this structured output and can guarantee you to some extent that you're getting you know reliable outputs and the structured outputs that you need I think this is going to be a super useful feature for a lot of developers that will unlock a lot of reliable applications with llms so here are just the tools that we Define again we give it the description and then we set some properties here the order ID and this is just string and the description of this is customer orders ID and then we can enable structure outputs by just telling it strict equal true and then once we have that then we can just set up our messages and then call the model we provide the tools and just by enabling strictly C true we are basically using the function calling capabilities along with the structure output capabilities of this model and so once we do that we get this response here it's more reliable as I mentioned I will do a part two to this otherwise this video gets a little bit too long and I'm going to go to some of their use cases as well compared to some of the things that we are doing ourselves there's a lot of interesting examples for you to take a look at so for instance these ones like stock price widget right you can create these UI you can create like a landing page for a gardener just by creating these schemas or defining these schemas on what properties and so on I think this is a really interesting application of structured outputs but the one that really CAU my attention is this Chain of Thought because this is something we use a lot so I actually demonstrated to you this particular example so by the way this model has been trained to perform this capability really well to understand these complicated schemas that you're defining right whether it be to the function calling whether it be to response format whoever you're doing that I don't know why it took so long they explain a bit here in the blog why this was challenging and again keep in mind this latest the penalty as I mentioned here which they most use because they preprocess the schema to generate this artifact that can use efficiently during sampling they explain that why and then they have like alternate approaches that they experimented with and finally here they have some kind of limitations and restrictions just read that and we'll go through this in the next part of this video one thing that they did here which I really like they acknowledge uh some of the community efforts around this for instance I've used instructor of used guidance as well and one of the popular ones is outlines pleas leave a comment if you have any questions on this I'll do again a part two to this just to go into more examples more details sort of best practices and again some of the ideas that we're using with some of these tools and how they might apply with the open a apis so thank you for listening and have a good one