Leveraging large-scale models for efficient learning

Key Takeaways

[Music] so today my talk will be about leverage life skilles models for efficient learning or in a much more simple word distillation so what are we talking about today well we are facing the really Dreadful and existential question how can you improve a large language model well there are not that many ways actually first one you can increase the number of parameters indeed the bigger the better you have more and more computations then you can make much better language model that's for this reason for example that created a 400 billion parameters model which is great isn't it then oh sorry you can improve the training data set by that you can say get more data well we have an issue here because we have basically SED the entire internet to create data set and it's not only true for Google gini GMA it's also true for open ey and most of our competitors so we can not easily get more data okay no problem maybe we can get higher quality data like taking out out the repetition the garbage or generated content it's a lot of work but at some point you also hit a wall here final possibility you improve your architecture change a few layers here and there add a few calculations improve your Optimizer and so on and so on but there like you can improve your architecture for years and at some point you will also hit a wall like you cannot get that much improvement from architecture of course you will but you won't have a revolution for architecture alone right now so what can you do once you have got the biggest model you could get like for GMA you want Rel you 100 billion parameters model because we want you to be able to run the model without needing a data center we have scrapped the entire internet and we have on the architecture until we get the best one possible well there is one final option get higher quality data and this is where distillation comes so to understand distillation first you need to understand how you train a language model all language model whether it's Gemini Gemma open or anyone are trained to do a really simple task predict the next World so it's it's a bit of a simplification but for the sake of this presentation let's go simple so they all do the same task they predict the next World why do I mean by that well you have a sequence of world for example in here the dog is and the model we try to predict the next one which will appear in the text to do so it will predict an probability distribution on its entire vocabulary so if you remember Al presentation earlier the vocabulary is given by the tokenizer and we are talking about 200 of thousand of tokens of possible wordss so in this case the the dog can be small it can be blue kind of very likely it can be brown white can be tree because Tre is a token vocabulary too it could also be a Chinese character a number it could be many and on this the model will give a probability distribution so the dog has 10% of chance to be blue 1% 1% of chance to be blue 10% to be small 2% to be brown and so on but but on the task we know that the dog is actually Brown so what is the model going to do well it's going to adjust its probability distribution so that BR probabilities will go we closer to one and all the other one are getting closer to zero and that's as simple as that okay but how can you do better well truth is not the best solution here you can do much better than the actual truth let me give you an example if I tell you my favorite animal is well 50% of you might tell me a cat 30% of weirdos might tell me a dog and the 20% left yes I love cats the 20% left will tell me a bunch of random things I don't know snake koala oron anything but if in my text I have cat what my model we see oh for this Step cat is for 100% sure the best answer and all the other one are are zero this is not the truth the truth is 50% for cat 30% for dog and a bunch of stuff for everything else so how do you do that well you take what we call a teacher Model A teacher model is a bigger Model A Better model a stronger model which R trend on large data set which gives great performance and we can predict an accurate probability distribution and insist on the fact that your teacher model must be good if it's not it's not going to work so you give your teacher model your sequence of word and your teacher model output you a probability distribution for the next word and this is this probability distribution you're trying to match so oh sorry in this case we don't want Bron to be at one and all the other one to be at zero BR should actually be at 0.5 uh small should be at 0.1 so it's already 0.1 so I are not ingesting blue and maybe there are some blue D 1% is fine and so on and so on so if I I some a little bit standard training you have your input text actually you put it through a tokenizer why because language model actually works with numbers you have a sequence of token and you try to predict the next token with a Target probability one for the answer and zero for everything else this is standard training distillation is a little bit different you take your teacher model and you really through the inut text and you run it through your entire training data set you get a probability distribution for each Target and this is what you're going to match but does it work this is a real question and actually I can tell you it does 100% training is much faster you need less time and less data to get the same results which is pretty good because as you may know training a language model is insanely expensive so if you can use less data less training is is better you have better performances like your evaluation gos better but you have some downfall first of all you need a teacher model and a big a good one if you don't have a good teacher model you're screwed and most importantly your student model cannot be better than your teacher you're limited to the capacity of your teacher so it's really interesting it's interesting for example you have a really good 100b model you are limited to 27b or to 10B like okay I can do my best with that but if your best model overand is a 10B you're not going to train a 27b on that it doesn't make any sense another issue is the teacher can leak in some way to the student suppose that you are super good 100b model but you were like I want it to be that really good on mats and accidentally whoops I overfitted on the test data set of mat well your teacher will also over your student will also overit so you need to be really careful ful about that and so some example so here are some impacts so we took a two billion parameters model and train it with and without distillation on several evaluation tasks that you can see here on reasoning code or I think un common sense and each cases you get an improve of performance thank you very much for listening to me [Music]