Reinforcement Learning From Human Feedback (RLHF) | Direct Preference Optimization (DPO) | Explained

Hello and welcome. I hope you are having a great day. In this video, we are going to understand a very important technique in LLM training or large language model training called RLHF or reinforcement learning from human feedback. So this stage comes after the supervised fine-tuning stage or SFT stage. So once we have trained the model to follow instructions, all the instruction following like question answering, code generation etc is done. Now we want the model to be more aligned with human values and safety. So the model becomes more close to human values. Now it has just the raw intelligence and we want the model to be more safe, more helpful and aligned to human values. So how it's done? So that is the objective. Why do we do RHF? How it's done? So current method that we used is called DPO, direct preference optimization. So we'll discuss about this method also in detail and also we'll discuss the key use cases of RLF. Why do we do that? So let's begin. So what are the LLM training phases? Let's do a quick recap. So we have seen this. So initially the whole internet data is used as the corpus that is all the wiki reddit stack overflow all the social media sites all that data available on the internet is used as the input the text is taken out and it is fed to the large language model and during the pre-training stage that is the first stage it learns all the language modeling all all the grammar, syntax, meanings, etc. Everything listening capability, everything is learned here. So now the model has a a fair understanding of the language. So it knows how to how the human language is. Now it can generate if you give a prompt then it can keep on generating a text from there. Okay. So that is your pre-training phase. Then we want the model. So now it has language capabilities. Now we want the model to follow instructions. Say for example, I want to do the model question answering. It has the language ability. But I want to chat with the model as we are as we do with chat GPT. So that ability doesn't have it. It has not learned it in in the text that it has seen. So that is done during supervised finding finetuning stage. Also code generation, code interpretation, tool calling, tool calling or function calling. So this ability is added during the supervised fine-tuning stage. So this is a very important stage where all the intelligence of instruction following is added in this stage. And this tool calling is the main thing which which is done during supervised supervised fine tuning or SF which is powering the AI agents because what are AI agents? They're just calling external tools or making API calls. Right? For example, the LM may not have all the information about uh about whatever we are asking for. For example, if I ask the model anything which has to do with the current context or it has to do with web search. For example, if I if I'm asking what is the share price of Tesla today or what is the weather today in certain city that information it may not have for that it has to make a web pole. So that feature was added in the tool calling. So this was this came in around 2023. LM first GPT came in 2018. LHF was in 2022 I guess I remember we'll see that. Okay. So this was 2023. So this was uh as a part of a paper called tool former which introduced this concept. So all these features came in one by one that is making it more and more powerful and on top of this this tool calling feature all these agents are built cuz this is nothing but calling APIs. The LLM can now make API calls to the external systems and get it. Okay. And then these two we have discussed in detail. If not I'll put the all the links. So this was discussed in the LM and we have SFT video also. So I'll put the link of those videos in the description. And now this is our focus. So let's focus here RLHF. And after all these phases, all the three phases then what we get as output is our final LLM. Okay. So let's see what happens in RHF and why it cannot be done in SFT. What is the limitation? So what is the limitation SFT? So what we do in SFT that we are feeding the model with lot of question answering. So say for example if I if I have to solve a problem some mathematical kind of problem. Okay say how to find a prime number. So there are so many techniques and we ask the developer to write code for that find a prime number. So there are many developers they will say five developers give five solutions for that maths problem with the code also and what should be the output. So all of the solutions may not be the best right or the best quality. Some of them will be good and some of them will not be good. But finally the model will be trained on all the five right. So the model will learn from everything and it will average out all the effect. Okay. So it doesn't know which is better inherently. The model doesn't know it is just a data for it. It will learn from everything. Whatever you give it, it doesn't know which one is good, which one is bad. So that is one problem. So the average problem and then the safety this is this is more important with respect to the use case with respect to humans. So first thing is that we are giving it more this options to learn that these are the correct answers. So it has learned so many inputs have been given. So all these answers tells them that what is the correct answer but it doesn't know what are the all the possible wrong answers for this question. There can be many possible wrong answers for this question and we when we start asking the model follow-up questions it can start making mistakes. So it is very hard or humanly kind of impossible to tell the model these are all the possible wrong answers for certain question. So there we give so this is also this cannot be captured. So we are just kind of teaching it to do question answering but I cannot tell it all the wrong answers. So this is one of the limitations of SFT. So we are learning what are the limitations and how this will be handled in RLF. And one more thing is that when it is learning from the all the text that it has seen on the internet in the pre-training phase it has all the information right wrong everything that was posted on internet it has learned that. So it and it will generate text based on that only. So if there is some harmful content on the internet so it has the ability to generate that also. So if I ask the model to tell me how to make a bomb it will give me the answer. If I ask the model how to do a suicide it will give me all the steps in detail steps but this should not happen and this is harmful for humans. So such kind of behavior which is a risk to human safety these all things which is not allow allowed to name not not aligned to human values and not safe for humans. So such things are not trained during the pre-training or the or the SFT phase. These things are again fine-tuned during the RLHF phase and that's what leads to our RLHF phase. So let's see. So RLHF is nothing but reinforcement learning from human feedback. So what do we do? We polish is the polishing final polishing is given. So whatever remains so we we are just polishing the model final polishing that aligns an LLM's raw intelligence with human values what is more valuable to us so safety it should not be racist or it should not be biased against some gender so such kind of values we give it during the RLF phase helpfulness how should the answer be okay if I tell you to give me answer in 50 uh words only not more then it should be able it should comply with that it should not keep on generating the text of whatever it has seen in the past okay so this is and what um technique we'll see so we'll see here that this in detail that it uses direct preference optimization technique or DPU prior to this it used PPU which was a two-stage approach but DPU is more efficient because that's why the name is also there direct preference optimization so we'll see this in detail and This RLHF was introduced in the research paper called instruct GPT that came in 2022 and this was led by a team of researchers led by OAN excuse me if I pronounce his name wrong okay so very difficult name so what was the paper training language models to follow instructions with human feedback and this is the paper link so let's see now what is DPU how it's how RLF is done. Even you are a part of RLHF. So even we are asked so generally when you use GPT or Gemini right so sometimes you will see that you will get two options A and B and the LLM will ask you to rate that which one is better okay just rate so this is nothing but you say say for example if you choose B then it will take that and it will eliminate other option so this is how you it is getting that which answer is more helpful to humans it will collect such information from all the users and will use to train the model. So that's what it is doing. It is when it is asking you, it is collecting data for its RLHF and that's what is being done in RLF reinforcement learning from human feedback. We are giving feedback. So what happens in this? The first stage is pair wise comparison. So the safety model takes a prompt. So a user prompt comes in and it generates two different responses. Why winner and why loser? So two responses, a winner response and a loser response. It doesn't know which one it we have given this label winner and loser Y, W and L because one will be preferred by the human, other will not be preferred by the human. So one the human will choose as you chose B and then other one will be rejected. So this is the winner, this is the loser like that. So that's what is happening and the which one is better response a human or other LLM. So during training time we sometimes use other LM also to rate. So that is another thing that can another LLM can also a good LLM which is better than that can also be used to rate the output. So that is also used but generally this that is humans do it. So when we were touring we used to do our ledger. So there was team which will have all the options A and B. But when we are using this as a user you just give which one is better right? But when the humans are training it so that time what we do we are rating it on multiple parameters and also giving reasons okay correctness accuracy and then conciseness all these there are multiple parameters okay helpfulness so there are different and each one each parameter is measured and each one you have to um let me redraw so each parameter you have to rate on a scale of 0 to 5 And also give reasons why you have chosen a certain option. If you're choosing A or B and you have to rate them and tell why are you choosing which option finally and that will be again reviewed also but that is how it is done when during the training phase the actual data that is fed into the model and such tasks are created and multiple such thousands and thousands of tasks for certain category are created and that are fed to the model and humans are doing this. So touring trains all the models for many frontier AI labs including when I was there uh I was doing for Apple bite dance by danceance is the parent company of Tik Tok then there are other projects also mistrol open AI Google all of them they train in their models at Turing does a lot of it there are other companies also but Turing does a lot of it okay so that's what was happening there and people around the world they generate the data for training the model because this is human feedback Okay. And then what is the result? A triplet. So the prompt and the chosen response. This is your winner and this is your rejected response. This is your loser. So this triplet is generated and this will be used to train the model. Now for this prompt, this is the correct answer. This is the wrong answer. So the model will know okay right wrong right wrong what is wrong. So this is how a model is trained using human feedback. So the next step will be that optimization. So this is the optimization where optimization is happening. So you're optimizing the model. So you have the input is there and now this input will be used to optimize the new model. So there are two model the policy model the model which is being trained currently trained where the RLF is happening and this is your the frozen copy. This is not changing. This is the original model with SFT your reference model. So there are two models. So one model is the reference and the other one is being trained. So which is this is being trained. So two versions of the model are compared and we have loss function so that the two parts of the loss function. So first of all the goal is that to increase the likelihood of chosen response and decrease the likelihood of rejected one. So there are the the loss function is a comparison. So it's a complex loss function. So first you have to choose the right the winner the winner response should be uh should have more likelihood to be choose chosen and then we have to decrease the likelihood of choosing the rejected response. So here we are comparing two responses. So there they use binary cross entropy loss. Okay. And on top of that it has to stay close to the original model also. Okay. So we are using binary cross entropy because there are two options. right wrong right wrong right wrong wrong one zero whatever state so for that we use binary cross entropy loss for training the model and then on top of that I have to stay close to the original model also so some form of kale divergence is used now kale divergence means when we're comparing the distribution of two models right that time we use scale divergence so my model should be the distribution output distribution whatever it is producing should not drift away from the original model for that we use scale divergence so the fin of the loss function is a comparison of some form of cross entropy loss, binary cross entropy loss plus scale divergence. Okay, so that's our loss function finally and that is used to train my policy model which is this which this is for our DPO. Earlier in PPU we used to separate uh train a separate reward model and then that was used. So that was a again a tricky and kind of convoluted process. Okay. So I won't discuss that further. PPU is nothing but proximal policy optimization. So I'm just mentioning it so that you should know that prior to DPU, PPO existed but that is not used now. Okay. So this is a DPO. I hope DPO is clear. So this is the main thing the algorithm of RL chip that is doing it and the key use cases of RL shift. We have discussed this in the past but let's revise. So the instruction following so exact instruction following and whatever we want to do further finetuning or align it with the human taste that how do I want because it is asking which is more helpful. So this is for helpfulness. Which kind of response is more helpful for you? So when I ask the model to give me the answer in 50 words only then it should give me the answer in 50 words only. So that's what I'm increasing the helpfulness. Then safety guard guarders. So this is around safety that it has to decline the request for hate speech or some dangerous instructions like give me the instruction steps to create a bomb, nuclear bomb etc etc whatever. So that is your safety. Then subjective nuance also this is again again around helpfulness. So how do I want the model to have the tone of the model whether it should be professional or casual. You can instruct the model and it will follow. So that is kind of role playing right? We give a role to the model that is helpful. So that is being learned during RHF. So this is for helpfulness. It will become more helpful that you are such and such. You are an you are a software engineer at top AI company. Now give the answer. Or you are a researcher at certain company. Give me the answer like that. factuality. So it if the model doesn't know any answers instead of making random guesses it should say I don't know gracefully it may not have all the answers. So this is being trained in narf with human feedback. Okay. And then coding style for example if I'm coding questions so coding also so there will be some ways of coding which will be better. So that's what we judge if you're having code interpretation or code generation task. So the developers will see which kind of coding response is better. the code that the model has generated which is better which is more pythonic if we are dealing with Python codes. So whichever is the more pythonic way of doing it so that will be chosen during training that data will f fed to this so that when it generates the output the output should be more aligned to the guidelines of that particular programming language. So these all things are done during RLHF. I hope RLF is clear now. Okay. So that's all for RLHF and that's all with respect to LLM training. So all the stages of LLM training pre-training, supervised finetuning and RHF is done including parameter efficient finetuning where we saw low rank adaptation. So all of the that is done. If you have not watched I'll put all the links in the video description. Kindly watch it. And if you have watched till here I hope RHF would be clear. Okay. And all the LLM training phases must be clear. And if that is clear, very good. Good job. Well done. And if something is not clear, you can write me an email to this. This is very important. These concepts are very important. So if something is not clear, feel free to reach out to us. So I'll write the email, write an email. And next we'll see some other forms of transformer architecture which is used. So we used B. So B is used some forms of B but are used a lot. Okay. And there is something called sentence B that is used in rag. So we'll see that also. So we'll understand all of this. So we'll only see B and then in the subsequent video we'll see sentence first. Sentence B or S Bert also it is called. But these two in the next two videos we'll see this. So that's all for this video. Bye for now.