ChatGPT: Zero to Hero

Key Takeaways

in this video we are going to talk about chat gbt and all of its technical details so at chat gbt you ask a question and it gives a wonderful response the way I want to structure this video is we will first talk about some fundamental concepts that are required to understand chat GPT and then go through each and every single detailed step in which helps you at least understand how chat GPT constructs the answers that it does so that it's safe and non-t and also quite factual so let's get to it also thank you all for 100,000 subscribers now if you can get this channel to 150,000 subscribers that would be absolutely amazing and we'll be posting more machine learning deep learning and AI content in general just like this so with that let's get started with some chat GPT lore to understand chat GPT we kind of need to understand some more fundamental concepts so chat GPT is built on top of GPT as well as the entire par Paradigm of reinforcement learning and the GPT models themselves are essentially language models and they are also built on top of Transformer neural networks so let's take a look at each individually so language models language models are models that have some inherent understanding of language in a mathematical sense and I say in a mathematical sense because they understand a probability distribution of a sequence of words words so given some context or words that have preceded it these language models can determine what is the most appropriate word or word token to generate next and depending on the type of data that is used to train this language model and also the architecture of the language model itself we can get different types of probability distributions of these word sequences which means that these language models will try to generate different kinds of words depending on these different circumstances and because of this we can actually generate language models to handle very specific tasks like question answering text summarization as well as language translation among others now let's talk about Transformer neural networks so Transformer neuron networks are a sequence to sequence architecture that takes in a sequence and outputs another sequence sequences in this case can be sequence of words for language the transform architecture consists of two parts an encoder and a decoder and so in order to do this translation from English to France it will take all the words of the English sentence simultaneously it'll generate word vectors over here for every word simultaneously these word vectors will then be passed into the decoder and then the decoder will generate the French translation one word at a time time and every time that there's one word generated it's going to be provided as context to the decoder itself now I've explained much more in detail how this training Works in my video on Transformer neural networks so please do check it out for more information now what's really cool about this architecture is that we have now two components an encoder and a decoder that do have some sort of contextual understanding of language and can actually be used as a base for language models and so if we stock the encoders we'll get a bidirectional encoder representation of Transformers or Bert and if we take the decoder parts and stack them together we get a generative pre-trained Transformer these are popular language models which are typically pre-trained on just general language data and then they are fine tuned by us depending on the task that we want to solve chat GPT is a GPT model that is fine-tuned to respond to a user's request and then it is also fine-tuned further by using reinforcement learning reinforcement learning is a method of achieving some goal via rewards I'm going to explain reinforcement learning in general by using a typical example and explaining these Concepts here but I'm also going to explain just after how these Concepts relate to chat GPT so at first we have an agent over here here and the goal is to make the agent go to this end State and in order to entice this agent to make certain moves we use rewards rewards are these scalar values that you see in these squares a high reward is used to entice it towards the goal and every other place has a lower reward so that we and make sure that the agent actually goes as fast as possible and taking as few steps as possible the state here is going to be the representation of the current step so for example this agent is in a position one one so that position is the state action is what action is taken by the agent for example left right up or down within the boundaries here in order to get to the final goal and the policy thus becomes the sequence or one sequence of actions that the agent will take in order to try to achieve the goal so one such policy for example could be the sequence of actions down down right right right and in this case the reward given total reward would be 10 - 1 - 1 - 1 - 1 which is 6 another policy could be the sequence of actions write down right up right down down and in this case the total reward would be 10 - 6 which is 4 and now we have two policies each with its own reward and so we can tell which policy was better than which now relating this to chat GPT the agent is the model itself the reward depends on the response that's given by chat GPT if the entire response is a good response then it's going to get a high reward if it's not a good response it's going to get a negative reward now in order to talk about state every single action taken by an agent is essentially a Time step now in the context of chat GPT a Time step occurs when every single word or word token is generated and so we can define a state as a combination of the user input prompt as well as every single word that has been generated until this point and this can be used to infer what action we need to take which is what word we should should generate next because this is inherently a language model and so the overall policy would be the sequence of actions taken which is what is the sequence of words that are generated so different policies would entail different responses and if we have multiple responses each of these are policies that means that each of them will have their own rewards and we can then start comparing them to see which was the better response and which was a worse response and then we would fine-tune the model in order to help it generate these better responses now that we have like a holistic idea of some of these foundational pieces how do they all come together with chat GPT so I took a screenshot of this from the open AI blog and let's actually walk through each of these so the entire process can be divided into three major steps in the first step we take a GPT model that has been pre-trained on understanding language itself and now we will fine-tune it in order to take in a user prompt and actually generate a response according to that prompt and we get the data with labelers so we'll have essentially a few labelers that will write a prompt and also write a response to how they want to see that prompt answered and because we have the input and the output this becomes a supervised fine-tuning of the GPT model hence sft next we now take this supervised finetune model and will take a single prompt pass it through the model and we generate a few responses over here and now the labeler is going to be ranking how well these responses are so for each of the responses that are generated the labeler will assign some reward and this is going to be used to train another gbt model which is called the rewards model and because this is a model it is a function which takes some input and generates some output the input to this model is going to be an initial prompt as well as one of the responses and the output is going to be the reward that quantifies how good was this response and now in step three we'll take an unseen prompt pass it through a copy of the supervised fine train model and then we are going to generate a response and this response is passed to a rewards model to get some rank that quantifies how good was this response and now this rank is actually going to be used used to further fine-tune our fine-tuned model and the way that it's done is that this rank is going to be used in the loss function of this model to back propagate some updates to the parameters now what's really interesting is that this process actually helps the model incorporate non-toxic Behavior as well as also create factual responses and this is because that's how the reward was generated itself the responses which were non-toxic and and also factual we're given a higher reward and so incorporating the reward into the model in this way is going to help the model generate responses that are less toxic and also more coherent and factual and that's kind of the overview of this entire process of how Chad GPT works now that we took a look at this full infographic in the next three sections we will dive into these three steps so let's start with step one in identifying what is the GPT in chat GPT why do we use it and how was it trained so the source of like GPT kind of comes from the Transformer neural network architecture which was introduced in 2017 which was a sequence to sequence architecture so it would take an as an input some sequence and it would output another sequence now in the field of natural language processing this can actually be super useful because sentences are a sequence of words and so we started to use these Transformers for well NLP problems like translation and I'm going to walk through exactly how that goes here so the Transformer architecture has two parts it has an encoder and a decoder the encoder is going to take all the inputs simultaneously and within this it's going to actually create some vectors for each of these words or I should say word pieces since there's going to be four that we have four vectors over here now these four vectors are going to passed in simultaneous iously into this decoder architecture and we're probably going to have when starting out we'll have like a start token and then it's going to now one through the decoder and it's going to Output one word at a time in this case let's say that the the problem that we're trying to do is translation from English to an Indian language specifically a South Indian language called Canada so this is going to be fun I'm going to teach you a few things here okay about Canada so my name is a j so the first word that's probably going to be generated by this hyp hypthetically if it's completely trained well and it's working properly is going to be let's see n which is translation the can word for my and then name is a j so after this first word is generated this word is now going to be the input for the next pass and so we input it here and then in this next pass we'll generate the next word here which is supposed to be hu this this actually means name and then for the third pass it's going to generate the next word which is a j now what this actually kind of shows is that this overall architecture has some semblance or understanding of language and in fact we figured out that simply the encoder part up here and the decoder part individually also have some understanding of language and so we can pick them apart stack them up in order to have them understand more and more parts and intricacies of language so if you stack just the encoder pieces together you get a bidirectional encoder representation of Transformers which is Burt and hence that entire research field that's gone into it and if you take the decoder part and stack them up you'll get generative pre-trained Transformers which is GPT and we're going to be focusing more on these GPT architectures moving forward so I hope the what is GPT over here kind of makes more sense right now now that we have the what out of the way why exactly are we using GPT architectures over let's say I don't know recurrent neural networks or any other typical modeling strategy that we used in the past well if we wanted to typically train a supervised model we would need to train all of these model parameters from scratch and we do this by collecting a lot of labeled data unfortunately though with each and every single one of these tasks mentioned you need to create a lot of get a lot of labeled data data feed it into your model to train it and to learn those parameters but this is actually going to be extremely hard to find in very large quantities and even if it does find the plethora the huge vast amounts of stores of labeled data it'll probably only be able to answer one of these major domains at a time and so to solve this issue we want to adopt a modeling strategy where we're doing some generative pre-training followed by some discriminative fine tun in Phase now we'll have a model where we want to do some generative pre-training which is the unsupervised approach to learn about language modeling language modeling here is the type of problem that we're actually optimizing for then we have discriminative fine-tuning which is a supervised approach to learn very specific tasks so in this case it could be like question answering or document classification or simply user response generation like a chatbot and so now that we have some understanding of why we need a GPT architecture let's actually play out what generative pre-training and discriminative F tuning actually mean in practice in order to do generative pre-training the goal is to optimize for the problem of language modeling now if we want to make GPT a language model language models have an understanding of word sequences its main objective is to predict what word is going to come next given the context of all the previous words that have come before it mathematically represented that's exactly what this is but we'll get to that so let's say that we have one training example where we just basically scour the internet for random sentences one of those sentences is today I want to play now we have a start token which we now input to this untrained GPT architecture when we start we want to say okay right here we wanted to generate the word today once we generate today we're going to now put that as the input for the next time step where we pass it into GPT and it should do its little magic in here and try to predict I now we're going to every single time we do this we want to CH tune the parameter such that it is more likely to produce this word I in the next time step we want to generate the word want then in the next time step two and then in the next time step play now mathematically speaking we want GPT to be optimizing some objective this is typical of all of machine learning and deep learning and the objective we're trying to optimize is that of a language modeling objective where we want to try to predict the next word which is wi in this case let's say this entire sentence is W so wi would be play using all of the previous words that came before it using the words today I want to and Theta here are the parameters of this GPT architecture and this overall statement across all of the words is something that we want to maximize and so Theta will have the parameters that will maximize this objective and so I hope this relationship between what's happening in intuitively and also mathematically makes a lot more sense now at the end of this generative pre-training phase we're going to get a model that has some inherent understanding of language and more practically speaking it's going to be able to generate well given a word sequence it will be able to figure out what word to generate next now in the discriminative fine tuning phase we now have a general model but we want to make it satisfy or solve a very specific problem for example in document classification simply giving the next word is not enough we actually want to understand the overall sentiment of the document of what it represents or we want to understand a categorization is it a sports document is it as a news document is it some other type of document and so typically we would have this pre-trained architecture and add probably like a very simple linear layer randomly initialized and then have a very small amount of actual training data for document classification to basically learn these small sets of parameters and further find tune the rest of this model now because there's only a small amount of new parameters and also like most of these parameters in the GPT model already have a good understanding of language you don't need too many of these these pairs or these supervised pairs of um examples for document classification and this is what makes it so much easier to actually just get started with so many of these facets of natural language processing using just simply a pre-training and fine-tuning um kind of argument or phase now with chatbot and chat GPT the input is some user prompt and the output is a response and that's already kind of the format in which GPT architectures are already fine-tuned and so all we would need to do is we don't even need to add very new parameters as we did you know for document classification but we can get extra examples of user prompts and their corresponding responses that are required in order to further just fine-tune this specific model where we're just going to be tuning these parameters and it's actually this that we kind of see where in the first step of chat GPT the labeler demonstrates the desired output behavior and it fine-tunes via supervised learning now to get a more concrete idea of exactly what's going on over here let's look at this figure in normal GPT U we would typically pass in um let's say one token and GPT will generate one token at a time so typically it'll be let's say that we generated three words today I will now this is going to be passed to GPT in this fourth time step and eventually it's going to generate uh a vector which is going to be of the size vocab size cross one and I guess like in the later versions of GPT this vocab size is the number of possible tokens that we could possibly generate in this specific language by GPT these tokens are not exactly words but they are word pieces so that this vocab size doesn't doesn't Skyrock it to be of the order of infinite number of values because there's just that many words the vocab size is like around 50,000 or something like that and this is actually going to be applied to a soft Max and the reason we do this is we want to make this entire thing thing a probability distribution so the total value is going to sum to one and when it's a probability distribution it's going to signify what is the probability that each of these tokens is going to be used as the next word now obviously we can only choose one of them and we don't typically choose the top one with the highest probability because that just sounds less natural and not superum instead we use a sampling technique this can include like temperature sampling or nucleus sampling or top case samp and you can sample basically from this softmax in order to determine what the next word should be let's say that the word that we generated or the word that we kind of picked was the word play and so play is going to be the next word that's generated and hence you'll get a response today I will play by chat gbt and this is also why you kind of see chat GPT generate one word at a time the in this ability actually comes from the underlying GPT architecture and in fact just to make it even more concrete I'm going to do a direct comparison with chat GPT and so let's say that you you ask a question what will you do today chat GPT might respond with today I will um as the first three words and maybe during the fourth word again it's going to go through the GPT architecture create this huge Vector of the vocab size cross one it's then going to sample from this probability distribution it's converted into a probability distribution because of softmax and then we'll sample from that to get the next word and so I hope now this creates an even better picture of what is going on behind chat GPT and how big of a deal GPT is within chat GPT I also want to hammer home the point that like chat GPT didn't just come out of th a it's very clearly based on many concepts of language modeling of Transformer neural networks of GPT architectures that have come before it and so much more let's now move on to the step two of this infographic to get a much more detailed picture so just to reiterate this at a very high level we have a supervised fine-tune model that can take one question and generate multiple responses and these responses might be slightly different from each other and so we have a person who will human label decide which of these responses is actually better for this specific question and so they are ranked and then we use this data as training data to train a rewards model and the rewards model is of a similar architecture now this is the gist of it but there are some questions that we want to answer here so my big question here was why can GPD generate different outputs for just one input so let's say here that we have a GPT model and in this case it's taking an input with we'll just call this some you and it's going to be what's for breakfast and the output of this model is going to be for every single time stamp it will generate one word at a time so it first generates today then it generates we then will then have then French and now we are at this stage to generate the sixth word let's call this entire response W now what GPT here is already trained and already fine tuned as well and so it already has a notion of language just like a language model and what that means is that it has an understanding of the probability distribution of word sequences and so what it's going to predict here is well this is the six word so we'll call that W5 given all the words that have come before it that's 0 to 4 and this entire input context so that's you and so GPT is going to try to determine what this value is and which word this corresponds to and then it would output that corresponding word typically in machine learning models if there is an output with the highest probability value it will just output whatever that value is but that's not exactly what we want in a language model because had we done that in a language model for the same input that means we would always generate the same exact output every single time because in this case let's say toast is the highest prob ility word we will always say today we will have french toast for this same input of what's for breakfast but this isn't exactly human behavior and humans tend to say words that are not the most optimal at every single word that we speak and so in order to circumvent this we kind of use decoding strategies to make the decisions more stochastic and more humanlike so when we pass in the input context what's for breakfast to a GPT model now GPT will now go through a decoding strategy and this decoding strategy will then determine what word we generate now there are many kinds of decoding strategies there's for example there's nucleus sampling temperature sampling there's top K sampling where the main goal here is not just to take the top word but to sample from some top few distribution words here in order to generate this next word and so it gives it some element of stochasticity so let's take an example that today we will have French blank and GPT is supposed to determine what's supposed to go here now GPT is already trained like I mentioned before so it has this knowledge of word sequences and probabilities and so GPT determines that at this stage this is the probability distribution of words that can go into this spot so there's a 31% probability that this word should be toast then there's a 19% bread 7% fries and it's in descending order this distribution now with just a GRE sampling which is the traditional case of like oh just pick the highest probable word it's just going to predict the toast all the time and every single time however if we use something like top K sampling let's say that K is 10 for every single word like in this case it's going to use the top 10 of these words with the highest probabilities in order to generate the word so it takes the top 10 here and then it will sample from these top 10 and then use that sampling as a part of this word next and in this case the top K is well let's just say we picked fries so today we will have french fries would be the prediction if we' used the decoding strategy top 10 so with nuclear sampling it's kind of very similar but instead of for every single case like every word we always pick a fixed amount like 10 we'll pick a variable amount depending on the probability distributions that we have at that point so for example p is equal to 0.9 would mean that we will get the all the words such such that the top words that correspond to the up to 90% of the total probability so if you add these numbers you get 0.57 so hypothetically if this P was 0.57 we would have only taken these top three sampled from it and then used that as a French word and in this case let's say that we could have gotten something like bread today we will have french bread would have been the case with nucleus sampling and then we have temperature sampling where we kind of change the overall distribution itself and we skew it depending on a temperature value so this temperature value for example can range from 0 to one if it's zero all of the the the highest probabilities will be skewed much higher and the lower probabilities would be skewed much lower and in this way you will have like toast tend to 100% everything else tend to 0% and if you were to then perform this temperature sampling it would be the same as the greedy approach cuz you'll always get toast anyway ways however as you increase this temperature to something like 0.7 this value of the probabilities will go decreasing for toast and it'll probably increase for the smaller probabilities so that when you start sampling you get a higher chance of variability for this next word and so you can see that as you get closer and closer to one the randomness and variability of the word generated increases in this case let's just say it was toast it could be toast it could be something else but the greedy will always have toast like I mentioned before and so if we had done this in another world in the same way we might have gotten something like this this could have also been a very possible output where the topk nucleus and temperature samplings gave different possible words here but the greedy will always give you toast you can actually see all of this math in action by going to playground for open AI it's like a beta version and you can just type in a specific prompt and you will get a response and this response can can be different depending on how you set the temperature or if you want to use like instead of temperature sampling you want to use like some nucleus sampling over here you can set up the top P value or anything else too so I'd highly recommend just checking this out and so I hope that how GPT can just take the same prompt and yet generate multiple responses makes more sense now now in this next phase here we have labelers that need to rank different responses that we get here and by ranking they also have to assign some actual reward value because this reward is going to be used quantitatively in a loss function so it has to be a number and not just like an arrangement that they've shown here but how exactly do we correctly quantify the quality of a response here so for every single labeler they'll be given a screen that looks something like this where they have you know they have the input prompt over here they have the output of that prompt over here and then they'll be asked to just rate this on a scale of 1 to S and then they're asked like a bunch of questions that are binary choiced over here so one of my first thoughts and looking at this screen was like why are we asking them so many extraneous questions don't we just care about the rating itself and this itself is just going to be used as the reward well that is partially true actually it can be used as a reward but let's say that I a labeler and I choose three for a specific user prompt and an output but how good is my rating of three to get more meta into this how high quality is my rating of three you can't really determin that so well because what's three for me might be two for you or someone else and because of that it becomes harder to get very high quality ratings too so to combat that issue we use something called a scale now a scale is essentially just a set of questions with categorical responses that we ask all of these question questions are made to ascertain how well the labeler is sensitive to the issues that are being presented after all we want chat gbt to have some understanding of nuance of language as well as understand sensitive topics so there's a bunch of labelers and they all fill this out for the same question we can then aggregate all of these responses for the specific uh instruction output pair below and then we can just say oh so it looks like this labeler labeled a three but they didn't answer the questionnaire similar to how other people answered it and so I'm not going to really consider their label three to be of high value and hence I'm only going to use the other responses that correspond to like the people who have labeled this in a very similar way and so by filling out this questionnaire and only using responses that correspond to where the bulk of people had filled out this questionnaire we can only get the ratings that are good ratings and are can be used in order to train a rewards model also just a little tidbit here the typical type of scale that's used here is called a lurt scale lurt scale is a common type of scale that typically corresponds to questions of psychological nature and so like I mentioned before we have good labels here now that's our first step to actually training a good rewards model and so a followup question is here how do we train this rewards model our rewards model is the same supervised fine-tuned model but with a scaler output and hence I've kind of connected all the neurons to just like a single output neuron here the input is going to be a prompt and the corresponding response and the output is going to be a reward that tells us how high quality this response prompt combination is this architecture here is the duplicate of this rewards model architecture here and so you can kind of treat this as like a Siamese Network where we have a prompt response one here prompt response 2 here they have their own rewards and we just compared this to the actual labels that we just generated like which response was better now we generated rewards here we can then use it in a loss function which can then be used to back propagate some values and hence further tune this rewards model now this loss function here assumes that the response one is always better than the response two as a label and it's also a very interesting function here that we'll try to get some more intuition on so here is the loss function and the R1 is the reward for the First Response R2 is the reward for the second response and this loss function will assume that the true label was that the first response was definitely greater than the second response now if our model however continued to predict that response one was greater and greater than response 2 that's a good thing and that good thing is reflected by our loss it means that the model is actually getting it correct and that's why we have lower and lower loss on the other hand if the model predict that the second response was greater than the first this is wrong and so you can see that as you increase the score by the model if it was like much higher and higher then you can see that the loss only increases here and so this loss function is actually quite effective in training our rewards model if you're kind of wondering like why we have a sigmoid function here it's because that this loss is proportional to the log odds that the first response is greater than the second response I've taken a screenshot of this exact loss function from the paper of instruct GPT and this kind of just shows exactly how the training is happening you would think that like in normal training we would just take all of these pairs we have a loss value for every single pair and we just randomly like start updating our network but the problem is that like there are four responses for example example from the same prompt and if we take pairs of those we have 4 C2 different prompts which is like six different prompts in this case and each of these prompts if we keep passing them into the model directly and shuffling it may lead to overfitting and so what we would do instead is we start batching all of the the single prompts with all of their response pairs together and so all of the six kind of losses that we kind of get from the single prompt they are all used together to make only one update to the model instead of six updates to the model and this has the benefit of one decreasing the computation time because there's just less back propagation updates we need to make and two it also helps us prevent the model from overfitting especially as the number of responses that we get for a single prompt is higher K corresponds to that number of responses I take it as four but it could be as high as 9 or 10 or anything that you decide and so that's how we train our rewards model and then we can use it as I mentioned before here to truly understand the quality of an unseen response and then use this output reward to further fine-tune our fine-tune model in order to have it generate more human responses that are factual and non-toxic and so I hope that you all have an better understanding of how this step two truly works about how GPT generates multiple responses with a single input how labelers actually Rank and also how to get rankings that are of high quality and then creating and training a rewards model along with some very cool loss function ideas let's now dive into the details of step three and this is kind of where the reinforcement learning piece is really tied in we have an unseen prompt that we passed through the supervised fine-tune model which generates a response now I want to actually go through exactly the mechanism and how the GPT model will generate this response so what does a response look like from GPT so we have the supervis fine tune model of GPT specifically GPT 3.5 now the input to this is going to be a user prompt and also all the words that were generated prior for this specific prompt in this case this is the beginning so we'll have like a start token we input this to GPT in our first time step and this is going to generate our actual first word or the next word which is today in this case for what is for breakfast now in the next time step today will now be a part of the input along with everything that came before it that we mentioned before and it will generate now the next word in the sequence which is say I and then this process repeats where we take now the word that was generated as the input put for the next time step and we generate the next word which is will then have then French and toast and so you can see that the GPT model over here is going to generate one word at a time using all of the previous words that it had come before it as an input context and so we have a user prompt we pass it into the supervisor fine tune model to generate one word at a time until all of the words have been generated for that response then this entire response is going to be passed along with the input prompt to the rewards model that is already trained and so what it's going to do is it's going to tell us how good was this response for this input prompt we'll get a reward and we now use this reward to finetune our original supervised fine tune model of GPT now in order to make updates to parameters it has to be used somehow in the loss function and this is exactly what we do with this poo model or technique how is the GPT model updated it is updated via proximal policy optimization policy optimization techniques in general are a class of techniques that try to maximize the total ru reward scene and specifically in the proximal policy optimization case the way it accomplishes this is using the reward in the loss function itself this function you can consider is like the negative of the loss function and so we want to kind of maximize whatever this value here is this product of this R function as well as this Advantage function a it is going to be proportional to the reward and so higher the reward we're going to have like a much higher value for this entire function and that is going to influence the direction of the parameters in our GPT Network these Theta are the parameters of our original GPT model I'm going to explain the individual terms here in my next pass but for now I hope you kind of just get to understand what proximal policy optimization is trying to do in general it is trying to optimize or maximize the total reward seen by our Network now let's get to the pass three where we dive into further details for our pass three I want to actually take a look a little closer into this model itself and just reson past to how it generates one word at a time but how exactly does it select which word to generate at every single time step so let's consider now our example for we have our original GPT model we have an input prompt and it has been generating words one at a time like it generated first today then I then will then have then French and now it is at the step where it should try to generate what should come next based on this input as well as everything you know everything that's come before it will also be an input here as well every single time that it needs to make this decision it's going to have like a table in its brain and this table is of a probability distribution it's a table of words as well as the probability that they will occur next for this given input prompt and given response that has been generated until now so in this case it's going to say okay there's like a 38% chance that toast is going to be coming and filling in this there's 27% chance it's going to be fries and there's an 8% chance it's going to be bread and maybe like there's some other words that come after this in the table but to actually decide the exact word that's going to be done G GPT is not going to just choose the top one and be like oh this toast is definitely going to be it because in general in language specifically we don't always as humans say the most optimal word that's going to come next it's just not going to sound very human and so what's going to happen instead is we have this probability distribution and according to this distribution we're going to just sample a word so there's a higher probability that toast is going to be selected but it's not guaranteed that toast is going to be selected and let's say that in this case if I pick a pick some word it just happens to be toast that's great then toast will come and fill this blank but it could have very well been fries or bread at any point of time and because this table is generated a new every single time we get a new input and these inputs can be different every single time we use even the same R request prompt this is kind of why GP t as you've seen probably in this initial case that it can it can actually create different responses despite having the same input prompt and because this response can be different every single time even for the same prompt we want to pass it into the rewards model just to check how good it actually was as a response and like I mentioned in previous passes we use this reward in our fine tune model to actually change the gradients and make some gradient update every every single time we have an input prompt now let's actually get into details about the loss function that's used to make the gradient updates so like I mentioned before this is the loss function or rather the negative of that loss function that we want to maximize Theta is the parameters of our original GPT architecture and is going to be the parameters of our chat GPT model to come now T over here is going to be every single time we have one complete response that is one time step R is going to be a ratio or a rewards ratio where it is the ratio of rewards with the new parameters for the given input prompt divided by the reward of the old parameters with the same given input prompt and so if this is a very high number that's way over one that means that whatever parameter updates we are thinking to make now they're actually going to be better at least for this specific input prompt than what it when the old GPT was for the previous time step and so ideally we would want to make these gradient updates in general now a is going to be called the advantage function and in reinforcement learning the advantage function is a value that assesses how high quality the output was with respect to the input which in our case is the same thing as a number that's proportional to the the reward and so overall this product of R and A is going to be very high if the response is very good and it's going to be very low that is either like closer to even negative because this Advantage function can be negative if the response is very bad at least for this given input and so we want this product to be as high as possible and that's kind of exactly what these policy optimization techniques do they try to maximize the total reward now this is this is great but we also don't want to make the gradient update a little too large because typically in learning we want to learn step by step and so in order to make sure that the gradient update is not too large we will try to clip the upper and lower bounds of this ratio and in this case it we can clip it by an arbitrary value of Epsilon we choose one as the center because if like the rewards ratio is one that means that the new parameters reward is equal to the old parameters reward which means that in Downstream like the parameters themselves don't change how large Epsilon is Will quantify how much we are allowing the gradient updates to change after looking at like a single example user prompt for example if it was between if it was like 0.1 Epsilon that means that this rewards ratio be confined between 0.9 and 1.1 which means that we are only going to make minor GR updates overall and in fact that's also why we're taking the minimum of this of these two values so that we have literally the smallest update that we will be able to make now why do we have an expectation over here well this kind of goes to the fact that I mentioned before how we can generate for the same input multiple kinds of responses and so we would want to simulate this input the same input into GPT like multiple times over and then we'll take an average of those values and so this entire thing won't just be reliant on a simple like arbitrary output that chap GPT happened to produce and so we take this final value and we then make grading updates via gradient Ascend because now this is not a loss function it's like the negative of the loss function something that we are trying to maximize and so we're going to keep performing this update for every time step T that is every single time we see a user input response pair and over time this value of theta is just going to get better and better until eventually we'll get a model that is just non-toxic to a very high degree factual to a very high degree sounds more human to a very high degree and that is chat GPT that we see today now that's going to do it for the video I hope that this infographic and chat GPT is demystified and while there's an explosion of language models on the scene today they are still based on the same fundamental principles so please do like subscribe thanks again for 100,000 subscribers love you all and we will see you very soon for another video bye-bye