OpenAI releases CriticGPT to correct GPT-4's mistakes | Read the paper with me

Key Takeaways

hey folks welcome to this new series called reading papers with herid where we simply grab a cup of tea or coffee whatever you like and I will walk you through the important details Concepts explained in research papers so get your cup ready as today we're going to talk about the paper that was released by openai last week called llm critics help catch llm bugs where they released their new model called critic GPD now what is critic GPD why is it important what are the methods that they have followed What are the kind of results that they have seen we're going to talk about everything so let's dive in so critic GPD represents a novel approach to enhancing the reliability of air generated content now this Innovative model which is part of the gbd4 family this is specifically designed to help human reviewers in detecting and critiquing errors in the code which is produced by Char GPT and here they have mentioned that on code containing naturally occurring llm errors the model written critiques are preferred over human critiques in 63% of cases and here further you can see that human evaluation finds that models catch more bugs than human contractors which are paid for code reviews so this has Pro provided that method which basically solves this growing challenge of evaluating increasingly sophisticated AI outputs particularly as large language models become more and more complex so heading over to the introduction section what makes the most capable AI systems effective today we know that they are all trained with reinforcement learning from Human feedback RL HF now this method leverages the fact that evaluating AI output is is usually faster and easier for humans than demonstrating the perfect output themselves so as AI models become more and more advanced now what is happening is even seasoned experts are struggling to reliably assess their outputs now this limitation of human evaluation is a fundamental issue with rlf and the field of scalable oversight aims to solve this by training models to help humans evaluate AI output more effectively and previous research has shown that methods like debate can help humans better assess answers to reading comprehension questions but now it's time to assess these models in more realistic settings so how can scalable oversight help humans assess model written solutions to real world tasks and for the first time the research demonstrates that scalable oversight can help humans more comprehensively assess these Solutions particularly in writing in code now the core idea over here is simple they trained an auto regressive policy to take a question and answer pair and then output a text critique pointing out errors using rlf on challenging real world data so they developed a GPD 4 based critic model which is called critic GPD which outperforms humans at detecting bugs here you can see in this figure they've shown that llms catch significantly more inserted bugs than humans and that model critiques are preferred over human critiques more than 80% of the time and human machine teams basically combining humans with critic GPD the model that they train those combinations this combination of humans and critic GPD they write more comprehensive critiques and avoid the nitpicks and hallucination better than the models alone the contribution of This research basically includes demonstrating a scalable oversight method for real world rlf data which showcases critic gpd's Superior bug detection and critique preference and then highlights the effectiveness of human machine teams and further you know introduces uh a technique called Force sampling beam search fsbs to balance real and spous issues in uh critiques which we'll see in a bit now let's talk about the methods that have been Incorporated in order to train this model the llm critics are basically Auto regressive Transformer models similar to your chart gbd they take question answer pair as input and generate a text critique which highlights the potential problems now the critiques include comments attached to specific quotes from the answer and to talk about the evaluation methods that they have the steps that they have taken first of all if we talk about the critique attributes now the critiques include comments contractors now are hired to evaluate these critique based on comprehensiveness inclusion of specific bugs uh which is CBI presence of hallucinated bugs or nitpicks and overall subjective helpfulness and they rate them on a 1 to S scale where one being least helpful and seven highlights the most helpful comment or critique now how are critiques compared and rated contractors compare four critiques of the same problem without knowing their sources and score them on various attributes which we talked about this comparison gives a clearer estimate of the model performance the scores are then summarized using ELO scores which estimate the win rate of one model's critique over anothers and confidence intervals uh for these scores are provided using a non-parametric bootstrap method let's talk about the data distribution how did they prepare the data for this model almost all training and evaluation data comes from the openi rhf pipeline consisting of question and answer Pairs and they focused on cases where the model's response was at least 50% python by line count using the largest code block as the answer and discarding the rest now they worked with two sets of data containing known bugs first the human inserted bugs where the contractors were paid to insert bugs into the answers and second the human detected bugs where the programming tasks flagged by original rers in the CH GPD labeling pipeline now human detected bugs are more natural and usually easier for humans to spot since they were already caught once on the other hand inserted bugs are tougher and more severe all right now the meaty part of this paper how are llm critics trained using rlf now the process follows a pipeline similar to charity first several critiques are sampled for each question and answer pair contractors then rate these critiques on various attributes including overall quality now a reward model is trained to predict these quality rankings and a policy is optimized against this model using pop now finally an inference time sampling strategy which is called your fsbs or Force sampling beam search is applied now additionally there's an optional First Step called tampering now a large fraction of the data actually comes from this particular step here the contractors introduce subtle bugs into the model written answers and explain the problems they introduced now this adversarial approach makes the bugs harder to detect and contractors also verify that an LM critic misses these bugs at least once in three samples and to validate discriminators are trained to predict whether the code has been tampered with or not now the tampering procedure provides the majority of training data making it more similar to human inserted bugs than human detected bugs now what is this Force sampling beam search fsbs approach that they have introduced and how have they used this in addition to rlf fsbs combines a reward model with search to generate longer and more comprehensive critiques reducing hallucinations and nitpicks because whenever you whenever these models are critiquing a long data long code snippet they tend to hallucinate and make up answers so we can break down fsbs in three parts first is the sampling during critique generation fsbs forces the model to produce highlighted sections of the input text that are quoted in the critique now this is done through constraint sampling where the model is encouraged to select text spans or code Snippets that are likely to contain bugs or issues then we have beam search now this model uses beam search to explore multiple candidate uh critiques in parallel and this allows it to generate longer more comprehensive critiques by considering a wider range of possible highlighted sections and the comments that are written alongside then finally we have the scoring model now this is basically just a scoring function that balances two objectives one is your RM score the RM score is the reward model which indicates how well the critique uh points out bugs or issues and then the second parameter is length modifier uh multiplied with number of highlights so this is a length penalty that discourages the model from generating overly long critiques with excessive hallucination or nitex so this fsbs model then basically provides a balance or you know manages the tradeoff with the hallucinations and nitpix and this was a new new method a new approach that they have introduced in this paper now let's look at the kind of results that this whole process has produced llm critiques especially those by critic GPD are often preferred over human critiques and they catch more inserted bugs that's what they're claiming and contractors significantly favor critic gpd's critiques over those from prompted chart GPT and human written critiques the tamper plus the rlf pipeline that uh they have produced this has improved the bug detection rate with llm critics catching more bugs than human annotators and When contractors didn't have llm assistants they made fewer comments possibly due to the need for specific domain knowledge and difficulty of some tasks now these factors explain why llms outperform human contractors and additionally critic gpd's reinforcement learning training this enhances model performance across different scales and they evaluated a family of gp4 mods and found that increasing the mod size improves bug detection to match critic gpd's performance without extra data or modifying the chat GPD pipeline uh about 30 times more pre-training compute would be needed in order to achieve that critic GPD helps humans create more comprehensive critiques especially in human plus critic GPD teams it outperforms the Char GPT on both human detected and inserted bugs while models have higher rates of nit pegs and hallucinations the human plus critic GPD combination these this teams when they work together they strike a balance now while chat GPD was trained with more data and compute critic GPD was was tested with a similar setup for a fair comparison and critic GPD showed higher precision and recall on human detected bugs proving more effective for code critique models without tamper data underperformed likely due to lower agreement rates and a less effective reward model as well llm critics generalize Beyond code and yes they can critic GPD was tested on General assistant Tas ask S from chat GPD training data marked as Flawless now in 24% of cases critic GPD identified problems that lowered the rating significantly compared to only 6% without critiques Now using critique reward models to prioritize tasks also improved problem detection and reduced the hallucinations so what's the key takeaway about large language models it's that they have become so Advanced that typical humans can't consistently evaluate their output without help and this highlights the growing need for scalable oversight methods whether rlf Remains the primary uh post trining method or not we must ensure that the model uh outputs are trustworthy so the approach here is straightforward training models to help humans evaluate other models and these llm critics are already successful in catching bugs in real world data and even accessible models like chat GPT uh can significantly assist these human annotators and contractors as llm intelligence continues to improve finding scalable methods to reward these uh right behaviors in AI systems is going to be crucial and llm critics uh show promise as a starting point all right uh so did you guys like it if you did give it a thumbs up let me know in the comments what other variations or you know details I should add uh how can we make it more interactive would love your feedback uh on the series and uh do tell me the kind of papers that we should read together uh if you want to give suggestions you can also join my Discord uh Community where we keep uh sharing these sort of uh resources but yeah that's it for this time I'll catch you guys in the next one until then keep learning keep building