Fine Tune Llama 3 using ORPO

Key Takeaways

hello everyone welcome to AI anytime channel in this video we going to find tune Lama 3 Lama 3 is the newest llm by meta AI in the coming weeks guys they're again going to release a multimodal llm as well so stay tuned for that now meta AI has released llama 3 train on 15 trilon tokens Lama 2 was train on two trilon So it's 15 more 7x even more than 7x right so imagine the I millions of dollars have been spent on llama 3 why open source Community guys Facebook has tons of data like meta AI has tons of data from their Facebook Instagram whatever and they can leverage all these data you know to train these kind of llms of course there are public data available redit Wikipedia and blah blah blah now meta has released Lama 3 everybody is using it performing really well on the leaderboard have you know fantastic reasoning capabilities as people have said it but how can you fine tune it that's the question now there are different ways of fine tuning an llm even on my channel I have 20 plus videos on fine tuning llms in this video we're going to focus on Oro odds ratio preference optimization it's a very fairly new technique you know to find tune an llm for Downstream tasks or basically Alig an llm that might be the right word I already have a video on orpo let's jump into orpo and see how we can I'm already fine-tuning it I'm not going to write a lot of code guys because the codes are going to be pretty standard we're going to take some code from previous videos and whatever we have and the code is available even Maxim has created a notebook the creator of merge kit and when I can share his notebook and go through the link but I'll just show you that I'm right now while creating this video I'm fine-tuning it for few steps just to trying it out let me show you what I am doing in this video now first go into it but if you want to understand orpo you should watch this video of mine which says orpo explained Superior llm Alignment technique versus DPO and rlf okay so basically orpo eliminates a need of a reference model that's a separate separate model for example if you go with a DPO direct performance optimization then you might need a reference model in that case first you find tune using sft the base model you will take it find tun using sft and then again you will use a reference model that becomes a bit of you know I mean that not at all costly when you compare this with r but at the same time it's complex because you need a separate model so there are different alignments techniques Chad GPT were based on uh rlf open AI has hir a lot of human annotators in Africa somewhere in Kenya Ghana I forgot the exact country but they hired some annotators they annotated the data you know reject reward whatever so rlf is extremely costly don't go into the bandwagon of saying that even you can do rlf you can't I mean you need a lot of money to do that not of you know good quality data to do that so rlf is costly so let's eliminate that from the equation then we have DPO PPO you know and different techniques that you can align an llm but those again requires a reference model what Aro does you know in the training F the fine tuning phase itself you know it basically eliminates a reference you can just directly do it in when you are using TR lsft whatever you are doing you can just do it okay so we have an orput trainer now that's a wrapper and that you can just use it now if you look at here you should watch this video okay where I have aligned mistal 7B but here we're going to train Lama 3 let me show you what I am doing guys okay so first thing is you need Transformers data sets accelerate Pi TRL bits and bytes and when B weights and biases because I'm also logging the logs in weights and biases just to see the how the performance has been the overfitting and things like that now once you install that I'm on a00 you need a00 GPU you can also do it on L4 if you need but then you need a smaller amount of data samples to do that okay now scroll down here are all the libraries that I'm have imported you can see I imported torch wend be uh low data sets because I'm going to load the data set from hugging phase I have user data just to take the secrets from the this secret the credentials for w b and then I have Laura config Pi model prepare model for KB training from pip parameter efficient fine-tuning and then from Transformers I'm loading all these libr modules like Auto model for Cal LM because we going to do a text generation task it's not a uh mask language modeling or MLM kind of a stuff it's a text generation thing then we have autot toonize bits and bytes config because we're going to do this in a quantized way the model that we going to Lo probably then Pipeline and then if you look at when I say Quant you're going to use Kora guys probably in this now have look at here we have from TRL it's a Transformer Library UT Oro config now orpo config is part of TRL itself so you can just use that wrapper and I have authenticated with my WB account wb. login you can see this is where my wend be is I will anyway uh this is where I have created my project you can see it's a new project called Lama 3 so Lama 3 is the project that is happening here once all of your logs will be recorded now scroll down a bit now what I'm doing in this case I'm saying okay if I am using the newest architecture of Nvidia gpus or the oldest one on based on M architecture so if you have a00 h00 and things like that then you can use Flash attention tool so in my case it is true because I using a00 so flash attention will be ported flash attention is really critical because of the memory and the GPU compute that you need it basically accelerates your training uh phage now that's what I'm doing then I'm saying notebook loging there are two reasons for it because if you want to find tune the base Lama 38b model it's a gated repository so you have to log in with your credentials so I have already logged in you can see it over here and then what I'm doing next I'm saying base model take this base model which is meta Lama 38b and I'm naming my model as orpo Lama 38b F you can see it over here and I have a bits and bytes config bits and bytes config load in 4bit I'm saying true because we're not going to do a full fine tuning so we are saying load in 4bit and I'm saying nf4 which is a data structure based on quantile quantization so I already have created a fine tuning crash course guys in fine-tuning playlist you should def you must watch this video before any any channel you go on the YouTube or anywhere else you should have basic understanding of this terminologies the data structure the algorithms how all of these things Works what are gradient accumulation how do you decide a learning rate what is nf4 what do you mean by rank what do you mean by alpha everything has been covered in that fine tuning video please watch that I I'll give the link in description now I'm saying use torch D tab type which is BNB 4bit compute and I'm saying use double Quant as true next is I'm using Lowa config where I have a rank of 16 you can start with eight and you can go with 64 if you go higher you need more compute to do that so I have kept it a pretty standard on 16 I using Laura Alpha Dropout bias n Tas type Cal LM as I said it's a text generation thing and Target modules up projection downward projection geted projection everything you should also learn this I have that covered in the video what do you mean by Target mode as we are not not adjusting the weights for each and every layer we are targeting a very specific modules which are compatiable with llama 3 Model you have to go through the research paper to understand this in detail what kind of Target modules you can Target for these kind of models now we have to specifically Target these modules now I'm loading the tokenizer you can see tokenizer and loading the model let me bring my monitor yeah so loading the model in model I'm using a model for Cal lm. from pre-train base model quantization config BNB config I passing the BNB config that I defined over here if you look at this BNB config I'm just defining it over here and then I have a device map Auto because whatever device you can upload something on CPU if you have a very limited compute and then I have an attention implementation which is Flash attention to and then I'm just loading the model in kbit you can see prepare model for kbit that's very important guys you know you should learn all of these uh watch my video I'll recommend that now you can see that this is the model has been loaded completely over here and then I'm using this data set Ultra feedback binarized by hugging F H4 so hugging face H4 has created a DPO specific data uh excuse me uh Oro specific or even for DPO you can use this for DPO as well Ultra feedback binarized there are different types of models that you can data sets excuse me that you can use it uh you can go on hugging face you find it out but I'm using this data set it has let me show you that data set probably quickly and I'll just show you this data set control come here and paste it now this is the data set I using now this data set has a prompt how can I develop a habit of drawing daily there is a choosen answer and there is a rejected you can look at the choosen and rejected column and you can look at the messages score choosen score rejected so each has a choosen and reject Ed how can I develop a habit of drawing daily so the role of user the content developing a daily habit blah blah blah it goes like this so basically aligning the llm guys you know but you have to be really careful your data quality should be of top note otherwise might it might be bias a bit because humans have this inherited bias we in all of us right we have some kind of biases so if we create this kind of data we have to be really careful now this is the data I'm using and I'm having a data set you can see it has around 6 61,000 rows for training and 2,000 for testing because I have splitted that I don't want to use that much so what I did I did a bit of changes here you can see I have a train samples only I want only 5,000 to train I need 5,000 I don't want to go with all the 61,000 because it will take a lot of time this is original train sample then I created a test sample took a ratio of it you can see here and then what I'm doing I have a trend subset test subset and then I have the train sub subset and test subset over here and then you can see what I'm doing here I have a function called and you can now see the number of rows have reduced to 5,000 and number of rows for TR testing has been reduced to 163 now what I'm doing next is I need to apply a map here on the data set choosen and dejected you can see the rows only rows we are applying it chat template the chat template is a function uh it's basically a module in tokenizer class that we have and then what we are doing we are creating two data set zero and data set one basically the nothing but the train and test set and then you can see the printed one 5,000 163 and the thing is there now this is our orpo augs which are important you should you know you should uh understand this a bit okay because these are see I'll I'll tell you a few insights now orpo according to their research paper orpo recommends to have a have a low learning rate when you compare that to traditional sft or you know DPO kind of technique so that's why we are using 0 minus 6 which corresponds to 1 E minus 5 in a safety thing so it's basically the similar stuff now we have max length Max prompt length you know if you have less compute you probably have to deal with this okay you have to reduce these numbers okay I have explained this guys you know training batch size this just in my pre let me show you which video you should watch for that okay if you want to watch that come to AI anytime chat are you ready to learn this this is the video you should watch which is here hyper parameter and training arguments I'll recommend you you watch when I found hostinger and I found out about the website builder I was really Blown Away it made this is the video you should watch guys mastering llm fine tuning hyper parameters and training arguments if you want to understand each and everything about these that you have what are the optim that we are using why are we using paged Adam and not only Adam you know we have a Max STS what the difference between Max STS and number of epo or things like that so these are very important important okay so this is what I'm passing we're reporting to WB output directory result where all of these result will get stored blah blah blah and then I have a trainer class you can see orpo trainer model argument passing the orpo asks train data set evalid assets P config and tokenizer okay now I'm training it you can see trainer. train train. save model new model it will take a it might take a lot of time guys you know to do that but I anyway I've been running it I'll see how much time it might take a might take a few hours guys to you know to get this completed but we will do it we'll try to or probably I will reduce the number of uh uh steps probably we'll do it for 100 or 200 steps now after the training is done how can you add the let me just add text how to how do add uh how to merge rather how to merge Lowa adapters now you will be you will get a new model here and if you want to push that new model again with after merging the adapters how can you do that so first thing that you have to do you have to flush the memory so you have to do Dill trainer and model let me just write the code probably I'll give you this notebook so you can just reproduce The Notebook you just run it and just change the data set if you want to change the data set in that case gc. SCT now we're going to delete everything which is in the cach because the the cash has the volume in we have persisted the volume in cash because we are restoring all the weights in the cache now I'm going to do torch. CA and Dot empty cach this is this is the thing that will basically delete everything empty everything in the catch and now we're going to reload so let me just write reload tokenizer and model excuse me me and then here I'm going to do tokenizer equals Auto tokenizer you can see from pre-trained and then goes the base model and then I'm going to pass my model which is auto model for Cal LM do from pre-trained base model and I'm passing let me pass a few things over here I'm going to use low CPU memory use which is true return dict I will also make this true it's a Boolean value you can see return dick equals true and then I'm going to have torch D type so torch D type torch float 16 and device map Auto we do not need this because we not going to train anything here so now this becomes your model the model is fine now after that we're going to uh uh model model tokenizer set of chat format this is right so we we do this and what else now we're going to merge at with basee model so we have the base model now so we're going to merge it so let me just write merge and then model equals and then you're going to use Pi model so I'm going to use Pi model from pre-rain this is right model and not Pi config this would be new model so we have the new model if you remember which is uh Oro 8B whatever that see that on top now model is done now you're going to use this model equals model do merge and unload so if you do this it will be it will be adapter will be merged and unloaded now you can also push it to hugging phase because most of you ask me this question how to push this now for to push this on hugging face guys you need to have an exess token which the right privilege where you can write the uh model into a repository so I already have done that push underscore 2core H and then you're going to pass your new model thingy so just write new model and and then you're going to use each temp directory as false okay use tempdir this becomes false model do and you can also push the tokenizer otherwise it will give you a config error if you don't do that push to have new model use temperature this is how you can push it guys now if you run this sale after the training step when the training is completed you know I'll of course I'll give you this uh notebook you can try it out now once that is done you can just uh push it to hugging face and you can try it out as well you can also use uh llm eval to eval it guys you know if you want to eval that okay uh let me know if you want to so we have llm Auto eval we have a lot of evaluation guys okay you can see by Maxim Labon okay and you can see automatically evaluate your llms in Google collab this is how you can evaluate using this this notebook Library whatever but there are other as well you can use open Hub you can use any number of others uh to evaluate as as well you have NOS Benchmark suit that you can use it to evaluate so feel free to evaluate that as well if you want to publish it on leaderboards wherever now this will take a lot of time guys probably I'll not wait for it but I just wanted to show you how you can do it let me just download this notebook I'll give it in the GitHub through a GitHub link or something you can see it has been downloaded now this this is how you should I think you should watch orpo video for Mel first then you should watch this one and go through uh implementing the notebook and you will have a fine-tuned llama 3 Model now this is how you can push it on hugging face as well after merging the adapter okay that's all for and here you can before that here you can once that is done here it will be visible guys you can also explore the logs and if you want to explore that but this is what it is guys if you have any question thoughts or feedbacks please let me know in the comment box you can also reach out to me through my social media channels find those information on channel banner and channel about us if you like the content I'm creating please hit the like icon and if you haven't subscribed the channel yet please do subscribe the channel guys that helps me you know to create more such videos in your future thank you so much for watching see you in the next one