ChatGPT API Announcement & Code Walkthrough with LangChain

Key Takeaways

hi everyone so earlier today openai announced the chat gbt API I thought it would take a quick look at what they announced and some interesting points about what they announced and then also have a look at how you can implement this in code with just the open AI package by itself and then also with Lang chain as well so they've announced the API and amazingly it's available straight away today which is fantastic they also announced some interesting things about whisper which I might cover in a different video but they gave some nice examples of customers who are using the chat GPT API starting to build apps with it there's some interesting things in the actual blog post here and this is really going into sort of what the model is and a little bit about it so one of the key things is that the model is actually 10 times cheaper than the previous GPT 3.5 models so that is fantastic right it suddenly means that a lot of things that were going to be too expensive to do with the open AI apis now suddenly become much more interesting to do you can now get you can now use a lot more tokens and do multiple calls to the API to be able to put something together which again give back to a user so the pricing is definitely one of the big things you can see they've also called this new API the gbt 3.5 turbo API and it lives up to its name so far in in testing I'm finding that it's much faster than the previous API and the text DaVinci 003 and it's able to process things much better and probably the quality is much better there as well one of the other interesting things that they announced along with this was this whole idea of the chat markup language so this is this is just let's just jump in here they've basically made available some of the internal token systems that the model uses so many of the large language models that are Transformers have what they call special tokens the most common ones just being like a start token and an end token but a lot of them use tokens for different things so you have things like the Macaw model which has tokens for contacts for questions for answers and they're kind of going down an interesting Direction with this but because this is all built for chat what they're actually doing is they're making this these sort of tokens represent either the assistant either their side of the API or your user or a system so you can see here when this is a simplified way of passing it in the basically you have the role is system and this is where your passing in the context the backstory of the actual bot to help it guide on every answer that it's going to give then you've got your user input in this case a question and then you've got the assistant sending the information back so this is quite a nice way of doing it that allows us to control they do mention that in future models they're planning to use these sort of tokens more going forward to allow you to steer where the model is going so it's definitely an interesting thing to get looking at now you'll see that also the response that you're going to get back is quite similar to before we've got a few different things one of the big things is that now the actual API endpoint has changed so let's have a look at that when we jump into some code they mentioned that instructing these chat models is basically the same as previously that you can play around with that but that may change it in the future they also mentioned some nice interesting stuff about chat versus completion so so that this model is set up to do chat but it can also do completion tasks so chat tasks is based on question and answer a call response kind of thing going on but the completion task is maybe where you start giving it something and then you want to just generate a bunch of content for you and they point out that they give you some sort of tips on how to do that with the tokens for this so basically just defining the context of what you're doing in the tokens and then passing things unfortunately as of now we can't fine tune this model so hopefully that will come in the future another interesting thing was that recently there was a leak about some of the things that were coming for chat gbt API and possibly the gbt4 API and we could see that this is from last week we can see that that certainly the first part is held true right that the chat gbt 3.5 turbo is this unfortunately the size of the span of tokens is still just 4096 so we haven't gotten the models yet that have the 8 000 span token or the 32 000 span token they are probably on the way in the near future anyway let's jump into the code and let's have a look at how you can implement this okay so here I've got my notebook and what I'm going to be doing is just walking through basically how you use the new chat GPT API with the standard open AI library and then with Lang chain so I've gone through I've set up my keys I've actually already run that cell and now the first one we're going to look at is just a really simple completion so here we've got a couple of things that we're going to put in so the start is rather than the openai dot completion endpoint we're using the openai chat completion endpoint so this is how you set that up with the open AI SDK directly we need to tell it the model that we're using and then we need to pass in this new sort of token system of messages so you can see here the first one we've got is basically we're setting up the role the system again is where I talked about earlier in the video of where we Define what the overall context of the bot is going to be so here I'm just putting your helpful assistant and the next thing that we want to do is we're passing some user input here so if we just run this right you'll see that okay sure enough we've got our response we can look at the response back that we're getting and we consider the answer that we got back was to hello what kind of assistant are you I'm a virtual assistant equipped with AI technology to assist you with various tasks and answer your question as best as I can how may I assist you today and the role that it passed back was the assistant too so we would need to take this and feed this into the messages as we're going to do this we can also see that we've got some nice information about the token number of tokens used Etc so this whole sort of chat markup language is going to become more prominent over time it seems so this is the token system that's actually going into the language model but to get this kind of thing this is from their documentation we're basically just going to run it in as messages like this so here you can see I'm gonna just set up my my special tokens with their actual content that relates to them and I'm just going to set up a simple while loop here of where we're going to have an input and let me just go through the code while this is running we're going to have an input as long as if the user types out exit we will get the total number of tokens used for the conversation and print that out and then break out of this Loop otherwise we're going to basically just add whatever the user said here and then we're going to keep adding the user's response their reply and then what it is that we're talking about so if I come in here and just say Let's do let's go let's show so when was Marcus Aurelius Emperor all right you can see here that because we basically defined up here that you're a helpful assistant named Kate so it actually introduces itself in this case Hello I'm Kate the Roman Emperor Marcus really is ruled from 161 to his death in 188.d I think it's a little bit more information in there we also then are printing out the total total tokens that were used up to this part so now if we go in and we put in something like what is wife we can see that okay now we're using a lot more tokens right because now we're feeding this messages part is now getting a lot longer it's got the first part and the second part in here so we've got our full sort of context is getting a lot bigger there okay so we can see that it's giving us information about the children and we can see now that our tokens are actually quite up there for this particular call not for the entire course so if we look at the entire calls so if we exit out of this we're going to see that we used 540 tokens in that small conversation so you can see that it's great that it's 10 times cheaper than before but you still can run up a lot of tokens if someone just sits there chatting away with it the next thing is using it with Lang chains so the link chain implementation I think is amazingly fast that the guys have done this so quickly in a matter of hours they already had some basic stuff going I thought I'd just show you just some simple things in here so the version of Lang chain I'm using is a release candidate so my guess is that within 24 hours they will have a new version out for this but you can see here that okay I'm basically bringing in so we've got these messages just like we had before which are going to go at the front of the prop prompt the thing that we've got little bit different so in the past we would only just bring in open Ai and now we're bringing an open AI chat and you'll see that if we and in the past we would basically Define the model as openai we might pass in The DaVinci passing temperature Etc the new one we're defining the model we're also passing in these prefix messages that are in there so now I've basically got this going so it's going to be basically take the the following user input and throw it in a informative and interesting but concise way for someone who is new to the topic right so this is just setting up our prompt template which is standard stuff for Lang chain and then now we can basically use this in here so we can see that we can pass in now when was Michael cerealis improved from you can see that this time because our our messaging and stuff like that we made a little bit more formal that you're a history Professor named Kate and we've defined a lot more now of the context that's going on in here so it's able to now give us an answer here zimpra and give is a little bit more information as if it was a history Professor trying to tell us some things about him and in the next one it's very interesting that okay we can do the same question again we're getting longer answers this time back another thing though that we should realize is that we haven't actually persisted any state in each one of these is an individual call to the language model so we would have to look at using the memory module in langchang to be able to constantly pass something back into it like we did with the example up here and I mentioned you'll see that in Lange over the next day or two so anyway this was a short video just to show you what some of the cool things that have come out with the new Chachi BT API and to show you how you can use it both in just the normal opening eye package and in the Lang chain package as well