LLMs itself CAN create BETTER LLMs

Key Takeaways

in the last couple of days there is one paper that has caught my attention and a lot of people's attention and that paper is self rewarding self rewarding language models it is a paper from New York University and also meta AI why is this paper interesting is what I was trying to Del dive deeper into the main thing here is uh as the title suggests you might have already guessed it that this is a paper that talks about models rewarding itself so the concept reward comes from reinforcement learning but why is it interesting why is everybody talking about it the first reason is if you think about large language models people are building a lot of large language models and one problem is still there and that problem is alignment so how do you make these models as good as human beings or how do you make these models value human values this is the question that everybody has been trying to solve that is one of the biggest bottleneck in fact a lot of times open AA has been been blamed for lobotomizing the model by doing rlf reinforcement learning from Human feedback and in the last few weeks or I should say few months there is another very popular technique that has risen up to the level where rlf may not be very important it so that new technique is called DPW direct preference optimization so it has got a preference pair or a preference data set and based on which the model has been trained or aligned or fine tune so whatever that is the final stage is DPO now what this paper is trying to do is this paper is saying that you do not need humans at all even to generate the preference data set you can skip humans I mean a little bit of human is required for the SE data set but you can skip humans a rest of the process you can have model zero build model one based on it build model two based on it build model three based on it and this model three like let's say llama 2 70 billion parameter model would be as good as mral medium so this is a very huge claim and I want to dive deeper into how they go went about it and what this paper is suggesting first of all this is still experimental I don't see like a data or model for us to play with it so take it with a pinch of Sal so the first thing is what is this paper this is self-rewarding language models in this study or in this work they're talking about a self- rewarding language model where the language model itself is used as an llm as a judge and this is not the first time we are seeing llm as a judge llm as a judge is a very popular Concept in fact a lot of popular benchmarks use gp4 as a judge for example I think mty bench maybe if I'm right empty bench uses gp4 to ultimately score the llm responses so there are a lot of benchmarks and a lot of existing techniques that use one of the best models like gp4 to score the response so llm as judge is a very popular concept so they're using llm as as a judge prompting to provide its own rewards during the training so we show that iterative DPO direct preference optimization training not only does instruction following ability improve but also the ability to provide high quality rewards to itself this is the main impact the most important thing of this paper is that they figured out that the it's not only the instruction following ability improves but also to provide high quality rewards to it like to the model in itself so fine tuning Lama 2 70 billion on three iterations like I said m0 is the base model M1 M2 M3 of our approach yields a model that outperforms many existing systems on alpaka evil 2.0 leaderboard including claw 2 Gemini Pro and GPD 40613 the latest version I guess while only a preliminary study this work opens the door to the possibility of models that can continually improve in both axes now one thing to note is uh very later somewhere in this they also acknowledge that this system may not hold entirely true or may not scale up in the real world scenario because sometimes it might saturate see at the end of the day you are using a base model to reward or create instruction data set and then reward the model so how much it would scale in the real environment I think that is something that we need to see whether it is going to saturate whether you know after a point it just goes garbage in garbage out that is not something that we know yet but yeah we have a model that can do certain things how does it work there are two main steps one is the self instruction creation the second one is the instruction following training so you have got Model T and you have got model t+1 so always remember the model t+1 uses the data and the rewards generated by Model T So Model T is like model zero and model one is M1 is Created from the data that is let's say created from model zero and what kind of data that it creates is the main thing so it tries to create a couple of things or at least the steps the first one is an instruction following step the second one is a self instruction creation step and uh and that is where the concept of AI feedback comes into picture and AI feedback data set AI F also will come into picture and if you are familiar with this channel uh you might have already seen that we have covered a different technique called RL a so rlf reinforcement learning from Human feedback is a technique where human beings sit read the question OT the response or score the response and then give it back and that goes back to the reward training for the model and using reinforcement learning the model is improved RL aif is taking the human out of the loop and then putting an AI there and to reward so this the paper also acknowledges like later in this paper you would also see that they would have acknowledge that there are techniques like RL a if that is already available where the model has been used as a as a reward uh you know the scoring mechanism but if you see here what is the model is creating so the model is first trying to create a couple of things one they're creating uh they're starting so the first thing is you have a base model you have a base base model and as you can see in the setup you have a base model and uh the model then what it tries to do is it tries to use a seed data for instruction fine tuning and this is the only place where human beings data is used like uh or I mean technically the base model also has got human data but this is this is where the human author examples are used and from that that is what they're calling as sft Baseline and from that is now you have a new set of data called EFT C data now what is EFT this is evaluation data so anytime uh if is create if is there uh the model m0 is used uh to create something new and that is how the EFT comes into picture that is the evaluation so I think it comes from if in itself so you can see here that this results in 1775 train and 531 evaluation and uh they're also looking at the distributions to make sure that these do not overlap so if you see here at the end you can see here so they have created a TSN it's a very popular technique to Cluster things so you have got the if data you have got the a data and then you have got the EFT data so the distribution the it shows that when the distribution of if and EFT are separate that means the the overlap is less so the concept of overfitting or leakage uh Benchmark breaking on all these things reduce so so this dist distribution good on them that they have managed to do it now how is it working so like we discussed like there is a seed model there is a set of new prompts that generates the responses and from that there is a new set of rewards that are created and then from that the preference pays are created and then there is an iterative DPO that creates the new model in itself and then it continues so m0 M1 then M2 and M3 and you can see how the responses are created so this is a sample prompt and it basically says okay review the user question add the response the main thing is okay conclude the score in this particular format and you can also give the score and what is like how do you score it so this is basically given to the model for it to give you the response and this is the llm as a judge prompt so this is how the llm as a judg is and if you see the model sequence the iterative training you have got the m0 the base model the llm base model the Lama 2 L 70 billion parameter model without any fine tuning at all initialized with llama sorry initialized with m0 or in this case Lama 2 70 billion then fine tuned on if and EFT C data during sft during the supervis fine tuning process they used the if instruction fine tuning data set and the evaluation fine tuning data set to make sure that the model is instruction fine tune and that is your M1 now that M1 initialized and then trained with AI F M1 data using DPO how is this AI F created and that is what is is this is the AI F comes from there and that is what is used to train the model and then you have got one more step of you creating an M3 so what they have figured out is that when you have this kind of models like from the base model the EFT plus if supervis fine tuning then the iteration 2 and the iteration 3 you could see the models win rate increases so you can see let's say this is the with the when you compare it with the sft Baseline this is the M1 model this is the M2 model and this is the M3 I feel like almost I'm dealing with Apple MacBook so you have got M1 M2 M3 and you can see that in every iteration the self reward win rate actually increases and you can also see here that uh if you have got different models this is on alpaka evil 2.0 data set so if you've got different models like GPD 4 0314 22.0 7 mistal medium 21.86 claw 2 and everything is below this so now honestly like this is something to be happy about but um I'm a little skeptical at this point because this is particularly looking at only one data set I'm still not very sure about how much of this is maybe an overfitting that we have not figured out I know that they have done distribution separately I know that they're showing that the over distribution doesn't overlap between if a and EFT so if and a actually overlaps which is like what you would expect because if is human generated data and aif is non-human generated data so the fact that these two are in the same distribution actually tells you that e is almost like if so which is a good thing e EFT stays outside of this distribution which is also a good thing because that's how you evaluate it but what is my concern is that whatever we are doing we at least this paper is primarily on only one data set I mean it's it's good to be happy I'm happy that we are going to take humans out of the equation but I'm not sure how much of this iterative development is either because there is something better happening or because alpaka evil 2.0 data set has something that you know maybe we couldn't have um figured out see I come from a classical machine learning background and a lot of times we have had like lot of you know interns and the beginners joining the company and then they would say that okay every time I I'm trying to add a new column and then train the model the model accuracy is increasing and that would always give me the skepticism to say that okay maybe you know what there is a noise that the model is trying to learn and that is what it is understanding a signal and it is ultimately overfitting so I'm not exactly sure whether it is overfitting here I'm happy that it is a self-rewarding model and the Improvement is happening but because it is improving at every single iteration that gives me a little bit of skepticism and that is exactly why I told you also to take the results with a pinch of salt because it's one evaluation Benchmark alpaka EOL 2.0 and it is the same model and which generates different um the a data set and then it kinds of uh you know dpos iteratively on it and the results are promising um I think the paper is promising everybody's been talking about it I think whenever you take humans out of equation uh there's a bunch of AI en who are very happy but um but how is it going to translate in real world is something that I'm definitely interested in learning about and if you have any questions please let me know in the comment section otherwise this is a very fun paper to read see you in another video Happy prom day