Reinforcement Learning from Human Feedback (RLHF) Explained

Key Takeaways

rhf is a technique that involves fine-tuning llms based on feedback provided by humans on the labels human reviewers rate the output of the llm on prompts and these ratings act as signals to fine tune the model to generate High rating output rhf brings humans into the loop to steer the llm in the right direction and ensure that the generated text aligns with user or application requirements reinforcement learning from Human Fe feedback is an approach in artificial intelligence that combines reinforcement learning techniques with human guidance to improve the learning process rhf involves training an agent or model to make decisions and take action in an environment while receiving feedback from Human experts the goal is to optimize the model's performance by leveraging human feedback as a measure of performance or even as a loss to optimize the model rhf comes with three stages in order to get the final model in the pre-training phase days the model is trained on a large Corpus of publicly available text from the internet this helps the model learn grammar facts and some reasoning abilities however it doesn't have specific knowledge about the prompts it will later receive during fine-tuning the output from this stage is pre-trained llm after pre-training the model goes through supervised fine-tuning in this stage the engineers perform fine-tuning on the pre-trained model using a data set that consists of human generated responses paired with high quality prompts this data set is created by human labelers who write tens of thousands of prompts and example responses the model is trained to generate responses that align with the provided examples the goal is to make the model more intuitive and better at following instructions the third stage of training is rhf where the model is fine-tuned further using reinforcement learning techniques and human feedback rhf incorporates human feedback into the rewards function of the model allowing it to optimize its decision-making process based on the feedback received the reward model is trained using comparisons where response a is deemed better than response B without specifying the degree of improvement or the reasons behind it rhf helps address issues like hallucination where the model generates responses that are not accurate or reliable by having a better reward function the model can be punished more for making things up reducing the occurrence of hallucination the process of training the reward model in rhf involves several steps first a large data set of prompt response pairs is collected human annotators are then tasked with ranking these responses based on their quality these rankings serve as the ground Truth for training the reward model to train the reward Model A supervised learning approach is employed the fine-tuned model generates a set of responses for each prompt in the data set the human evalu valuators then rank these responses providing a numerical score for each one these scores act as labels for the reward model the reward model is trained to predict the reward score given a prompt response pair this is achieved by minimizing a loss function that measures the difference between the predicted reward score and the true score provided by the human evaluators the model is trained using gradient-based optimization techniques such as stochastic gradient descent to iteratively update its parameters and improve its performance by maximizing the gap between the reward scores of Chosen and rejected responses the reward model learns to distinguish good responses from bad ones this allows the rhf system to generate highquality responses by optimizing the policy based on the predictions of the reward model during this process random prompts are chosen from a distribution and feeding input into the llm model to generate a response the response is then scored by the reward model it is important to impose a constraint during this phase the result from this model should not deviate too far from both the models obtained in the supervised fine-tuned model and the pre-trained model it is important to strike a balance between generating high-scoring responses and maintaining the consistency with the sft phase and the original pre-training model this is achieved by introducing a constraint in the training process the constraint ensures that the model does not deviate too far from the previous phases preventing biases towards responses with extremely high scores that may not necessarily be good responses reinforcement learning from Human feedback is an approach that combines reinforcement learning techniques with human guidance to improve the learning process it involves training an agent or model to make decisions and take actions in an environment while receiving feedback from Human experts