Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!

Key Takeaways

If you tell me what you like and what you don't like, I'll use that to train my model. Stack Quest. Hello, I'm Josh Starmer and welcome to Stack Quest. Today we're going to talk about reinforcement learning with human feedback and it's going to be clearly explained. This Stack Quest is brought to you by the letters A, B, and C. A always B Curious. Always B curious. Note before we begin, I want to mention that reinforcement learning with human feedback is almost always used to help train large language models like chat GPT and DeepSeek. So in order to provide context for understanding reinforcement learning with human feedback, we'll also talk about how to train a large language model from scratch. If you just want to learn about reinforcement learning with human feedback without the context, you can skip ahead. Either way, you need to already be familiar with decoderonly transformers and back propagation. If not, check out the quests or my book on AI. And you should also be familiar with the essential concepts of how reinforcement learning works with neural networks. If not, check out the quest. Now imagine we wanted to build and train the next chat GBT or DeepSeek from scratch so that we could ask it a really important question. What is Stat Quest? And it would generate a polite and helpful response. Stat Quest is an awesome YouTube channel for learning about AI. That means we need to learn how to train this untrained decoderonly transformer model. By untrained, we mean that all of the weights and biases in the model are just random initialization values. For example, if we gave this untrained model the prompt, what is stat quest? Then because the model is untrained, all it will generate is blah blah blah blah blah. Now, since blah blah blah blah blah is not a particularly polite response, and it certainly isn't helpful, we need to train the model. The first step in training a large language model is to train it to predict the next token in a really large body of text like the entire Wikipedia. For example, we could start with the article on ah and use the tokens for a was a Japanese to train the model to predict the next token girl using normal back propagation. And then we might use was a Japanese girl to train the model to predict the next token group. Likewise, we would just keep on using fragments from Wikipedia articles to train the model until we got to the end where we would use of horizontal points has been to train the model to predict the next token changed. Oh no, it's the dreaded terminology alert. Even though we are training the model to predict the next token in terms of large language models, this is actually called pre-training. Thus, by pre-training our untrained model to predict the next word with a huge body of text, we end up with a pre-trained model. Bam. Pre-training a large language model results in a model that is pretty good at taking bits of Wikipedia articles and predicting the next token. However, this isn't how anyone actually uses chat GPT or DeepSeek or whatever. For example, Squatch has never prompted a large language model with of horizontal points has been and expected to get the response changed. Instead, we usually enter prompts like, "What is a good YouTube channel for learning about AI?" and expect a polite and helpful response like, "Stack Quest is a good channel to learn AI." Bam. In other words, when we prompt a large language model with a question, we expect a response that is both polite and helpful. So our model which so far has only been pre-trained to predict the next token is said to be unaligned from how we actually want to use it. And that means that we need to align the pre-trained model to the way we actually expect to use it. Aligning a pre-trained model usually takes two steps. Supervised fine-tuning and reinforcement learning with human feedback. Supervised fine-tuning uses a data set with pairs of human created prompts and human created responses. For example, someone might write out a prompt like this. What is a good YouTube channel for learning about AI? And that same person or possibly somebody else might write out a response to that prompt like this. Stack Quest is a good channel to learn AI. Bam. And given a prompt paired with response, we could use normal back propagation to train the model to generate the response. As a result, supervised fine-tuning turns our unaligned pre-trained model into a model that has started to be aligned and can generate polite and helpful responses to specific natural language prompts. Note, because creating these prompts and responses is done by real people that need to be paid for their work and takes a long time, the data set for supervised fine-tuning is relatively small compared to the data set used for pre-training. As a result, supervised fine-tuning can easily train a model to overfit the data. In other words, our model might respond appropriately to this specific prompt, but it probably does not generalize well to new prompts that were not part of the supervised fine-tuning data set to begin with. So, after pre-training followed by supervised fine-tuning, we have a model that is somewhat aligned to the way we plan to use it. It can respond appropriately to prompts it was specifically trained on, but not to new prompts. Hey Josh, how do we train the model to respond to new prompts? Well, the ideal way to deal with overfitting is to just use a super huge fine-tuning data set. Unfortunately, creating a super huge fine-tuning data set would cost a super huge amount of money. So instead of spending a super huge amount of money on building a super huge fine-tuning data set, we can use reinforcement learning with human feedback to train our model to generate appropriate responses to prompts it wasn't initially trained on. Bam. Note, before we dive into the details of how reinforcement learning with human feedback can be done, I want to reiterate that the main idea is to create a much larger data set than we used for supervised fine-tuning while simultaneously minimizing the cost required to obtain human preferences in how the output should be polite and helpful. In other words, if it was super cheap to have people create a huge data set for supervised fine-tuning, then we would probably do that instead. So, with that said, the first step in understanding reinforcement learning with human feedback is to understand that given a specific prompt like this, what is a good YouTube channel for learning about AI? We can configure a model to always select the output with the highest value for each token and as a result generate a specific answer every single time like this. Stack Quest is a good channel to learn AI. Bam. Alternatively, we can use the outputs from the soft max function as the probabilities that each token is selected. And when we use the probabilities to select the output token, the token with the highest value will be selected more frequently than the other tokens, but the other tokens still have a chance at being selected. And as a result, we can get different outputs like this. Stack Quest is a great channel for learning AI, double bam, or this. Stack Quest is an awesome channel for learning AI, triple bam. So given these three generated responses to the original prompt, we can list all possible pairs of responses. And for each pair of responses, ask people to tell us which one they prefer. Asking people to list their preferences is much faster than asking people to write out the response that they think is best. And these preferences are the human feedback bit in reinforcement learning with human feedback. Bam. Now that we have the preference data, we can use it to train a model to give relatively high scores to the preferred options. Hey Josh, I'm not sure I understand what you just said. Can you give me an example? Yes. To train a model to give relatively high scores to preferred options, we first make a copy of our model that has already been through supervised fine-tuning. Then we modify the model by removing the unmbedding layer and replacing it with a single output and the result is a new model that we will train to calculate rewards for reinforcement learning. Now given the human feedback preference data that we collected earlier, we can train this model to generate a positive value or positive reward for the preferred response to the prompt. And we can train the model to generate a negative value or negative reward for the response to the prompt that was not preferred. Note to be clear, when we do this training, we pass the prompt and the response to the new model that calculates the rewards. The context of the prompt is needed because different prompts will need different responses in order to be polite and helpful. Anyway, one cool thing about this process is that we don't have to define the ideal output values or rewards in advance. In other words, the model can figure out the appropriate output values or rewards on its own. For example, this loss function, which was used by Open AI in their 2022 manuscript, training language models to follow instructions with human feedback, lets the model figure out the best rewards without us having to define them in advance. This loss function might look super fancy, but it's really just a few relatively simple things multiplied together. We'll start by focusing on this difference. The first term, reward better, corresponds to the reward the model calculates for the better response. The second term, reward worse, corresponds to the reward calculated for the worse response. Now, ideally, we'd like reward better to be a positive number, and we'd like reward worse to be a negative number. And thus, ideally, we want the difference between reward better and reward worse to give us a relatively large positive number. And that means ideally we want to plug a relatively large positive number into the sigmoid function. For any input value or x-axis coordinate, the sigmoid function gives us a y-axis coordinate between 0 and 1 for this sshaped curve. For example, if we plug in a relatively large positive number like 10, then the output or corresponding y-axis coordinate is very close to one. In other words, if in an ideal situation, reward better is positive and reward worse is negative, then the difference will be a relatively large positive number. And the sigmoid function should give us a value close to one. In contrast, if we plug in a relatively large negative number like -10, then the output or corresponding y-axis coordinate is very close to zero. The output from the sigmoid function values from 0 to 1 are then plugged into the log function. And plugging in xaxis values from 0 to 1 into the log function gives us the corresponding y-axis coordinates for this curved shape. Now remember ideally the difference between reward better and reward worse will be a relatively large positive number and thus the output from the sigmoid function should be close to one. So in an ideal situation the log function will output a relatively high value. However, if we use gradient descent to optimize the parameters in our model, then gradient descent, which tries to find the lowest point on a curve, we'll try to find this lowest point. And that low point is the opposite of what we want. So, since we want to converge on this high point, we flip the curve over by multiplying everything by -1. And now without having to tell the model that we want reward better to be positive and we want reward worse to be negative, we have a loss function that can train this model to calculate the rewards correctly. Double bam. Once we're done training this new model, which is called the reward model, we can use it to train the original model that only had supervised fine-tuning. To train the original model with the reward model, we start with new prompts that were not part of the supervised fine-tuning data set and we don't have humanprovided responses for like this. What's another good YouTube channel for learning about AI and have the original model generate a response like this. YouTube is special. Now we run the prompt and generated response through the reward model to calculate a reward. And because the response is not helpful, it will get a negative reward from the reward model. And we can use the reward model to train the original model. Then after training with reinforcement learning, our prompt, what's another good YouTube channel for learning about AI? results in a polite and helpful response. Serrano Academy is a great YouTube channel for AI. And because this response is both helpful and polite, it gets a relatively large and positive reward. In other words, by training a reward model to score responses like a human would, we can use the reward model to train the original model to generate polite and helpful responses to new prompts without requiring us to create a huge and super expensive supervised fine-tuning data set to avoid overfitting. Bam. Once we have finished training the original model with reinforcement learning with human feedback using our reward model, we end up with the final model that is both trained and aligned to the way people actually want to use it. Now when we give our model the prompt, what is stack quest? Instead of just getting blah blah blah blah blah as output, we get Stat Quest with Josh Starmer is an awesome YouTube channel for learning about statistics, machine learning, and AI. Triple BAM. And now it's time for some shameless self-promotion. If you want to review statistics, machine learning, and AI offline, check out the Stat Quest PDF study guides, and my books, the StatQuest Illustrated Guide to Machine Learning and The StatQuest Illustrated Guide to Neural Networks and AI at stackquest.org. There's something for everyone. Hooray! We've made it to the end of another exciting Stack Quest. If you like this Stack Quest and want to see more, please subscribe. And if you want to support Stack Quest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs or a t-shirt or a hoodie, or just donate. The links are in the description below. All right, until next time. Quest on.