PaLM2 from Scratch

Key Takeaways

okay so in this video I'm gonna be talking about Palm 2 so I'm going to tell you a little bit about the models a series of models I'm going to talk a little bit about how they're structured and how you can use them and then we'll look at actually coding something from scratch that makes use of palm without the use of Lang chain or any other framework like that so first off the the Palm 2 Series of models was officially released on May 10 at Google I O this year and there are a number of variations of these models so one of the first things to understand is that there are four main sizes of the Palm 2 models there's gecko which is aimed at mobile devices and running a palm 2 model on a mobile phone Etc there's an otter which is a bigger model but still a reasonably small model there's bison the one that we're going to be using today and there's unicorn which Google hasn't really made public so far so it's important to understand that when we're comparing these models to other models what are we actually comparing when people talk about Palm 2 often they're actually talking about the Bison model that we're going to be using here and I would say that the Bison model is probably on par or similar to something like chat GPT 3.5 turbo model it's not a gpt4 level model the unicorn model is obviously a lot more powerful but that hasn't been made public probably for a variety of reasons including that just serving it is going to be a challenge with all of these models so we know that Palm 2 is driving a lot of the new changes and improvements in Bard the experiment for conversational AI from Google and one of the main reasons for this is that Palm 2 is actually a multilingual model and that can handle over a hundred different languages now I talked about in the the recent video about bad that one of the things that they've done is that they've often masked out certain languages until they are very confident that they're not going to be offensive or the accuracy is not going to be bad for those models so today we probably haven't seen anywhere near the amount of languages that Palm 2 can actually do Palm 2 also has a whole bunch of coding in its pre-training so this is something where there are specialized versions of palm 2 that actually Focus much more on code and we can look at one of those in this as well as we're going through so on top of this Palm 2 is actually driving it's like the foundation large language model for Google for over 25 products and it's their sort of go-to model at the moment and probably until the Gemini models come along you're going to be seeing this in most the Google products there are also some very specialized versions of the Palm models so Med Palm 2 is one that's trained specifically for medical uses and a whole bunch of different things like that while I'm not going to go into depth with that it's pretty amazing that this model is able to get expert level on a number of sort of U.S medical licensing exams and speaking to some people inside Google they've been so impressed by it they feel like if if they were sick with some sort of disease they would probably aim to access this I think this is definitely in the future for this on top of Med Palm 2 there's also a model called SEC Palm which is a specialized version of palm 2 that's aimed for security and cyber security at uses which to date hasn't been made public another use that they're using this for is a thing called Duet AI for Google Cloud so I think you will see a lot more coming out about this in the next few weeks but this is basically a tool that helps you with coding and helps you with being able to set up things for Google Cloud Etc okay so your access on the Google Cloud platform to the Palm models is going to be through vertex AI so here you can see I basically just brought up vertex AI you will need a gcp account to do this if you've never set one up before you will get 300 credit the first time that you set it up and you'll be able to use this to access not only the Palm models but also a variety of other models in vertex AI you can see in here they've got a model garden with a bunch of different models both from Google and other open source models and then the plan is as I understand this is to add to this over time so it will allow you to basically use these models for a variety of different language and vision and even audio tasks going forward so what we're interested in here is the generative AI studio and we're going to be interested in language so I'm just going to open this up and you'll see that in here they've got a whole bunch of things around prompting and prompting the models they've got a whole bunch of things around chat with the models and then tuning the models as well so I'm not going to go through the tuning of models today we're going to look at the three different ways you can do chat and prompting in here okay so the first thing to understand you'll see these colored sort of guides here that there are three different ways that you can use the Palm 2 models the first would be freeform so the idea here is that if we wanted to do something like summary let's just open this up and we can have a look at the thing we can see that basically all we're doing is putting in a prompt and then we can choose our model here we're basically choosing the text bison model and we can submit it and it will basically then generate response back so in this case we've basically asked it to make a summary it's made us summary from this back and this one is just straight up texting text out no fancy system prompt nothing like that now obviously we can set the temperature we can set the token limit we can set our top K we can set our top P like this they have safety thresholds as well so you can pick how safe you want the model to be now I know some people are very anti the whole censorship of models and stuff like that for business cases this often you know makes a lot of sense so this is the first form of what they call uh free form so we're just gonna Not Gonna Save that prompt the next one we're going to come in and have a look at is the chat style of a prompt so you can see here we've got a context which is very similar to how we would have the system prompt with chatgpt or with the open AI models in fact often we would call this like the system prompt so this basically just sets the context of here where you are a customer service representative and it's basically got some policy and stuff and then you can basically have some examples so you can have like in context learning examples in here and then here you can see that the conversation is going to be user AI user AI Etc so whatever I type in here will just become part of the responses of the conversation and it's going to basically feed all of this in each time as our to grow our memory so while we're still putting in just a string and getting a string out we've now got this formatted a lot more so that we've got this context up here we've got this now you'll see that they use a different model to do this they're using the chat bison model rather than just the plain text bison model for this all right so the third type of prompt that we can do is actually what we call a structured prompt so let's have a look at this classify text here here you'll see that we're actually setting up we have a context like before we're actually using the text bison model like before but we're setting up a number of examples of sort of input text output text input text output text and this is a good thing for doing sort of classification or that kind of thing and then we would basically pass in what we want for something like this and it will generate an output so you can think of this as like we've got our context or our system prompt up the top we've got our in-context learning examples here and then we've got the one that we're actually going to feed through again like with all of them we can set our temperature we can set our token limit we can set you know the other variables there so at any point with any of these we can basically just come up here and click view code and so if we click view code we can either look at the python the sort of raw python code for this we can look at getting a collab for this that we could export out or we can get just the actual curl commands that we would send in for this so what I'm going to do is we're going to go back and have a look at a chat example and we're going to go through and do their company chat bot and look at going through this and exporting this out we're going to change it up a little bit okay let's jump into the code okay so in this notebook I'm going to go through three examples of code using the Palm to bison models to generate output so the first thing you need to do is install Google Cloud AI platform also known as vertex Ai and then it's very important that once you've got this installed that you restart your runtime if you don't do that you will get errors later on going through it so just restart your runtime and then you'll be fine everything will be set up the next thing that you need to do is basically go through what I call the the Google author authentication dance so if you're using collab on the same email that your Google Cloud account is on you will be able to basically authenticate so your collab can access everything inside of Google Cloud easily this is really going to help you a lot with running this kind of thing without it you need to create a Json file and bring that into here and authenticate that way but doing it this way it'll just go through the dance of asking you to allow it to access Google Cloud platform and then once you authorize it it's like collab is now hooked up to your Google Cloud platform okay so first up we're going to look at chat model we're going to be using this in the style of a chat so you need to basically set up vertex AI for your project so you will need to change this out to be the project your project name on Google collab and you want to set the the location this is not so important but generally you just go for U.S Central Us West one of those and then you can see here I'm next setting up the actual model for this so the chat model that we're going to be using is the chat bison001 model so this is their standard chat model so this is a version of palm 2 size being bison remember bison is the second biggest one and it's probably being fine-tuned for chat that's why they're calling it a chat model in this case we can see we've got our standard parameters so this is what we could set before in the UI or the temperature max output tokens top P top key in there okay now I'm just setting up the context of the chat so I'm using their example in here and this is basically a sort of a company chat bot support chatbot that kind of thing and so it's just got some information in here about the company that okay they're headquartered in Chicago that they make close we can see you know the typical price between 50 and 300 put a bunch of things that are in there just in the context in there so once we've got that context set up we can now just ping it with sending a message so you can see here when I ask it okay what does symbol sell the response back a symbol direct is an online director consumer Footwear and apparel retailer headquartered in Chicago so it's just getting that information from here and putting it together so this is standard llm chat bot kind of thing when was the company founded again getting the founded information from in there and when we can so each of these is basically getting this app now we don't realize it but as we've been going through this this is actually created a message history in here so if we want to look at the message history we can basically look at just do chat.messageistory and we can see we've just got a list of messages here so the first one is a chat message it's come from author user so we've got the model author and then the user author we could assign different names if we wanted to here well we can see that okay we've got that the user asking this and then the bot answering this and then the user asking this the bot answering this back so this is like what you you would see in a lang chain conversational history kind of thing so you would have the the different messages going through and this is done all for you in the Palm chat API so remember this is the sort of difference that makes chat different just from sort of text generation that kind of thing so that that's the first one to look at next one I thought we'd look at is uh using a text generation model but this is actually the code generation model and we're going to use a version of Bison which is built for code here so here we've got code generation model from pre-trained and what I'm doing is just giving it like the standard or what one of the similar kinds of things to Pal so where I'm basically asking it to write a python function based on this sentence so the sentence is Arjun started with 17 apples he gave Haruka 11 apples then and Rita gave gen seven more Force how many does ajan have now so think about it we've got 17 minus 11 and then adding 7 there so let's look at the output for this we can see showing off it basically generates three back ticks so this is markdown for showing us that it's going to be code and that it's going to be python code and then it generates the function and you can see that in there here it's basically got a doc string of kind of what we just said above there and then it basically sets each of these up with a comment for each of these so I just started with 17 apples you have 11 apples and you can see down here we've got start apples minus Haruka apples plus Emery temples and that's going to give us the right result so this is in fact just an example of pal done from scratch in the using the code generation model from Palm 2 here last one I thought that I would look at is using the sort of structured problem and I just wanted to show you that there's nothing super special about the structured prompt it's just a nice way for perhaps people who are not used to in context learning thinking about it when they look at the UI if we actually look at the code here what we're doing is hashtag generation and we're going to ask it to generate hashtags of tweets and then we're passing in some different tweets and then we're giving it the hashtags in this tweet up and then it's going to generate out and sure enough this is basically the the sort of in context learning as it goes through for these and then finally the last one that we're going to put in is going to be this one launching in the next video using Palm 2 with langjang creating a palm 2 composition agent using pump 2 with tools and sure enough can generate the hashtags out for these giving us Palm 2 Lang chain Palm 2 composition agent Palm 2 tools for these so in traditional way of thinking this is just conditional generation with this being the input the way that they show it in their UI is as a structured thing and really the structured part is just the in-context learning that's going on in there so hopefully this should give you a good example of how to use Palm 2 without any need for things like Lang chain or something in the next video I'm going to go through using it with Lang chain and we'll look at some of the things that you can do with Lang chain and and palm 2 as well as always if you've got any questions please put them in the comments below if you enjoy videos like this and you want to see a bunch more please subscribe and click like as well and I will see you in the next video bye for now