RLHF vs SFT to break out of local maxima 📈

Key Takeaways

[Music] the intuition General is that like for instance for code because this is factual you can check if the code is correct or notf is not the way to go you prefer to do like supervis f tuning as a human to write the code but in fact because humans make mistakes because actually even in code there's some preferences that they might like that and maybe for some other reasons that we don't know LF is so much more scalable It Gos less it's easier than it leads in general to just better performance and maybe we can come with a compromise we suggested teacher critic where it reconciliate and unified super tuning and lhf such that when you do human preference and you have two outputs but both are very bad in the code for instance you will ask a human to edit the best answer to make it correct now so now you are doing sft when all the answer was really bad so that you can get out from the local minimum of your