LLM Fine Tuning Crash Course | LLM Fine Tuning Tutorial

Key Takeaways

This video is a complete crash course on LLM fine-tuning where we will start with some theory on what is fine-tuning. We'll cover some popular methods such as Laura, QLoRA, etc. And then we will write code in Unsloth to perform fine-tuning on a sample data set. Let's understand LLM fine-tuning in a very simple language. When I moved to US from India, >> [music] >> one of my friends took me to play baseball. And I was actually comfortable playing baseball. Why? Because in India I have already played cricket. Now in both of these games, you have bat, ball, you have to track the ball, and move your bat, run. So whatever raw skills I had from cricket, [music] I deployed that while playing baseball. Essentially, I'm transferring the skills from one game to another. So in AI there is a term called transfer learning where you take the model and then you retrain it or you fine-tune it on new data set, task, and format. So that is what LLM fine-tuning is. When you look at LLM such as GPT-5, Llama, etc., they are trained on a vast amount of internet data. But let's say you are an AI engineer working in a private company. Let's say you are working for Johnson & Johnson or Reliance Jio and >> [music] >> you are given a task to build a chatbot for your customers that has knowledge of the internal data of that organization. Also, it should respond in a specific format, tone, etc. At this time, you will use LLM fine-tuning. So LLM fine-tuning is a process of retraining the pre-trained model such as Llama on a specific task, data set, tone, and format. Let's take an example of an imaginary company called Loki Phones uh to understand this thing better. You're building a chatbot for this company using models such as GPT and Llama. And when your customer asks a question, "My Loki Phone 12 screen is cracked. What are my options?" The base LLM will give some general answer, right? But the right answer in this case is you should use this Loki Care Plus plan if you already have it. Now let's say this data is not available on internet. In that case, plain LLM will not be able to answer this. There might be other questions, okay, how long battery last, whatever. And assume this is the kind of data which is private to organization and LLM has not seen it. So using plain LLM, it won't be able to answer. So what you do is you use a simple technique called retrieval augmented generation. In this technique, you point your LLM to external source of knowledge, right? It can be a database, PDF files, whatever. This is private data. It's not available on internet. But when you're building a chatbot internally, you can point it and it will be able to answer it, okay? Now the benefit of RAG is that you are not retraining the model, right? So LLM is nothing but there is a neural network here, right? There is a neural network uh which you're training and the network looks something like this. It's a huge network. Let's say it has 70 billion parameter. Now what do I mean by parameter? Each of these edges have weight. Let's say this has a weight of 0.8, 0.7. And how many edges are there? Maybe 40, 50? In actual LLM, there are 70 billion parameters that you have to update. So the benefit of RAG is that you're not updating everything and it is very cost-effective, okay? But there are disadvantages. RAG may not be able to produce the best possible answer in your company's brand tone and the format that you expect. For example, for this question, RAG may produce this answer, but ideally you want this answer. Similarly, this is a second question. RAG may produce this and I but fine-tuned model will produce this. And you will notice that here there is more empathy. See? And let's say empathy is one of the values of your company that you want your chatbot to follow as well. So overall fine-tuning gives this benefit that it can handle emotion slang. It can answer with brand tone and overall it can produce more precise answers, whereas RAG has its own set of benefit, which is it is low cost, okay? So if I have to fully summarize pros and cons, then these are the pros and cons, okay? You can go over it. Overall, RAG is cheap, fine-tuning is expensive, but fine-tuning produces better answer compared to RAG. So what people do in the industry is they combine both RAG and fine-tuning, okay? I'm [snorts] giving you an example of a base LLM called Llama 3.2. And by the way, you can go to Hugging Face and find both of these models. So Llama 3.2 1B is a base LLM, whereas 3.2 1B instruct is a fine-tuned model. So this model is fine-tuned on this particular base LLM. And the base the foundation model is just an autocomplete. You all know, right? Like how how large language model works. The way it works is you're giving a sentence and it is just doing autocomplete of that sentence. So when I say, "The capital of India is", it will autocomplete Delhi. Okay, and then it will autocomplete next sentence, next next word, next word. And that's how it produces a big paragraph as an answer. But when you look at the instruct model, that is more similar to the response you get in ChatGPT. You have question-answer pair, okay? So summarize this article. Then it will summarize. Translate this sentence to Hindi. It will summarize, right? So this is So this fine-tuned model is more like ChatGPT, more consumable. There are two types of fine-tuning. One is full fine-tuning where you will retrain the entire network and it's going to be costly, right? You're updating 70 billion, 100 billion, whatever is the parameters of your LLM, you're updating all of them. Second one is parameter efficient fine-tuning. And there are two popular methods, LoRA and QLoRA. Here you are not updating all the layers actually. You keep these layers frozen and you add some new layers on top of it. If you know about transfer learning, in deep learning there is a technique called transfer learning where you keep certain layers frozen. You don't update them. And you update only some other layers. So PEFT is similar to that and these are the two popular methods. We are going to discuss LoRA, which is low-rank adaptation technique. It is one of the popular fine-tuning techniques for LLM. We have to first think about the transformer architecture. In transformer architecture, we had all these V, K, Q parameters. If you remember, query parameter, key value parameter, and so on. And when you're training this model, you are essentially training the trainable parameters, which is represented by these matrices. So if you remember from our deep learning module, we already discussed these metrics, WQ, WV. And these are again trainable parameters. These are the matrices which gets trained when you are training this particular transformer model. And I'm representing one slide here. We looked at this slide previously where we had this WQ, which was, you know, matrix that will tell you how to encode query of a token for attention computation. And we had couple of such matrices in attention layer. We had WQ, WKV, WO will be the output one. And then in the feed-forward layer, we had two matrices. If we represent all these matrices as W, let's assume we call it W in a generic way, then what we do in LoRA is we keep this W frozen. So let's say if you are using Llama model or GPT model, okay? And you are fine-tuning it. Now that model will have all these matrices. You keep them frozen. You don't change anything into them. But you take your training data set on which you want to fine-tune your model. And when you're fine-tuning, while you keep W frozen, where you don't change any parameter here, you add a new uh set of parameters called delta W. So you add a new matrix, delta W. And during the training, you change parameter only here. So this one is frozen, okay? It's like a box. It's frozen. It doesn't change. But this delta W will change. This is called parameter efficient fine-tuning. Here the whole model is frozen, but a small set of trainable parameter is added to the model, okay? So this is PEFT. Now you might ask, "How can this save computation time?" Because we are saying, "Oh, parameter efficient fine-tuning, right?" Um we we could have fine-tuned the entire model where we just change W. But we don't want to change it because we want this process to be efficient. But then if you're adding the delta W, it's the same thing. It's in fact more computation. So the question is, "How does this solve computation problem?" Well, in LoRA technique, we do something really clever. Let's say you have this delta W, right? Which is a D by D matrix. Okay? And Delta W is this. The new uh trainable parameter that you have added, okay? So, let's say it's D by D matrix. You decompose that into two matrices, okay? A and B. And here we are using this R parameter, which is called rank, okay? So, in LoRA, low rank adaptation, okay? So, this R is this hyperparameter. So, if you multiply these two matrix, like 4 by 2 and 2 by 4, you will get this particular matrix, okay? So, let's assume that D parameter is 512. So, let's say if you're training with Delta W, you will be training these many parameters. The total number of parameter will be 262,000. But, when you decompose it into this A and B, so this is a dot product, by the way. So, A.B, you get Delta W, okay? So, when you do that, uh and let's say your rank is eight, okay? So, this R, once again, is a rank parameter. When you say low rank adaptation, R is a rank. And this is a hyperparameter. It can be 2 8 6. There are different values, okay? That you can take. I'm just using R eight as an example value here. So, what happens is when R is eight is the parameters in this one is 4096. Parameters in this one is 4096. So, total number of trainable parameter is 8192. So, when you're training a model, you know, when you do back propagation and when you update the weights, you're updating only 8192 parameters. Whereas, in this case, you're updating so many parameters. So, this is how you achieve efficiency. And this is the crux of low rank adaptation. The rank parameter can take any values. Uh these are just general guidelines. But, as you can see, R can be 4 8 16 32. You know, it can be any of these values. I have seen eight to be the common value, eight or 16 to be the common value. But, once again, it's a hyperparameter, okay? If you look at uh PEFT library from Hugging Face, which is used for this PEFT optimization, R is taken as a hyperparameter in a LoRA config, okay? So, here eight is a common value. But, based on your data set, based on your problem, you can play with it and just see which method works the best. If you're interested in reading the full paper, you can just Google LoRA paper and you will find a PDF. So, just in case you're curious, you can go through it. But, I hope you got the crux of it, which is adding Delta W parameter and then decomposing that into A and B individual matrix matrix such that A.B is equal to Delta W. And then, total number of parameters that you have to train through A and B will be much lesser compared to Delta W. And in this technique, once again, you keep your W, which is your original set of parameters, frozen. You don't change them, okay? So, computationally, this method is very efficient. In order to understand the QLoRA technique, you need to first understand quantization. You might know about this concept from the deep learning module. But, let's uh refresh our memories. Let's say you have LLaMA 7B model. 7B means it has 7 billion parameter. And these parameters are nothing but the weights in neural network. Let's say each parameter takes 4 bytes or float 32 or 32 bits. These are all are same things. In that case, this model needs 28 GB of RAM. Like, when you're loading that model in your computer, it needs 28 GB of RAM. If you're talking about 70 billion model, it needs 280 GB of RAM, which is too much memory, okay? You can't run it on your local computer. But, what if we reduce the size for storing each parameter. Let's say from 4 byte you go to 8 bits or just 1 byte, okay? So, 8 bits is 1 byte. 1 byte is equal to int eight, okay? Int eight is a data type. In that case, it will take only 7 GB of memory. Here, I'm showing you a table where we have this different sizes for bits, okay? So, if Q is 32, which means 4 byte, it will take 28 GB, this scenario. But, if it is eight, it takes 7 GB. If it is NF4, int four, which is half bit, you know, 4 bits, it takes only 3.5 GB and you can run that model locally on your computer. If you have 70 billion model, you might need multiple GPUs with float 32. But, if you have int eight, you might need only one GPU. You know, you can run it with one GPU or two GPUs very easily. And this process of reducing the bytes, you know, that it takes to store one parameter is called quantization. It's a process of converting high precision numbers into low precision formats to reduce memory and computation requirements by machine learning models. Now, you'll be like, when you go from high precision to low precision, of course, you're losing accuracy. Well, that is true. But, it has been shown through the experiments that, practically speaking, you still get a pretty good accuracy, okay? For your given use case. And quantization helps especially with edge devices. Let's say you have a drone, which is flying over your field and you're using deep learning to uh do some kind of uh computer vision type of use case, then quantization really helps because you can have quantized model loaded into your drone. You know, drone will not have that much memory. So, it will work the best uh during the inference time. So, quantization is popular when it comes to edge devices. It is popular in other use cases as well, where you need efficiency when it comes to your computation and memory requirements. So, let's see how quantization works. Let's say you have a couple of weights which are float 32. Okay? And usually, these weights uh are minus one to one. So, the values of these weights will be .91 .78, okay? Negative. And then, .87 .28 and so on. So, I'm just showing you some sample weights from a neural network. And the scale here is minus one to one. When you quantize this to int two, int two will take how many What will be the range for int two? Just just tell me. You need to have this computer science fundamentals clear. Int two means two bits. And in two bits, we can store four number. Minus two to positive one. So, the way you can quantize this is you can divide your float 32 range into four equal bins, okay? Four equal bins and just map them to these four bits. So, these values n- .91 .78 will become negative two. .39 will become negative one. Any values that comes into this bin will become negative one. Okay? So, it is sort of like scaling, but scaling with surrounding and some other difference, okay? So, the values you had for weights, minus .91 -.78 and so on, after quantization will become this, okay? So, it's a pretty simple concept. What will happen if you are quantizing to int eight? Well, once again, what is the range for int eight? Int eight means eight bits. And eight bits can store values from minus 128 to positive 127. So, you divide, once again, this range into total 255 bins and you just map them. And let's say you call your original values X and you call your quantized values Q, okay? So, there is a mathematical formula to perform quantization. The first thing you need to do is find out the scale. Again, this is like you have learned scaling in machine learning, right? This is similar to that. You're scaling values from one range to another range. So, you first find scale by saying Xmax minus Xmin Qmax minus Qmin. So, Xmax is one. Xmin is minus one. Qmax is 127. Qmin is minus 128. So, you put all those values here and you get a scale of .0078. Now, to get quantized values, you use this formula, okay? This is a standard formula. So, you have your X values, minus zero point, which means what is the value where you want to center your points. In our case, let's say this is zero, okay? So, this zero point will be zero. X will be all these values and scale will be scale, okay? So, for one of the values, let's say for this negative .91 value, uh zero point is zero, then scale is this. When you round it, you will get negative 116. So, you map this value to negative 116. Once again, it's a concept similar to scaling with some differences, where you have a bunch of values, you're mapping it to limited range of integers. Okay? See, when we were doing scaling in machine learning module, the scaled value will still be a float. So, you are not saving any time anything in terms of memory. But here, the scale value is integer. So, you are mapping this float to integer through this formula, and that will give you this memory saving. So, I have this Jupiter notebook where I'm creating some hundred random values in range minus one to one, okay? So, let's say these values look something like this. Now, I'm using the same scale formula that we saw previously. Max X minus min X Q max minus Q min, and scale is going to be 0.0078, okay? Now, my zero point is zero underscore point is zero. And for quantization, I'm using that same formula. X minus zero point, okay? X minus zero point here, divide by scale, and you round it to an integer. And then you clip the values to Q min and Q max. You don't want to your values to go beyond minus 128 and plus 127. And when you apply this formula, you get Q values for your X values. So, for your X values which are these first 10 values, the first 10 Q values will be this. So, see? We map minus one to minus 128. Then um minus point 9797, whatever, to minus 125, and so on. And to do dequantization, which is from these values, let's say I want to get these values back, right? Because you have to do that. To do that, you will just use this formula. It's a reverse of quantization, okay? So, this formula should be pretty straightforward, folks. Scale into Q minus zero point, and you get these values back. So, this minus one value you got it back as minus 1.0039. So, there will be some loss of precision. You understand that, right? It's like when you zip your data, unzip it, there is some loss. Similarly, here there is some loss. So, minus 0.979, you got it back as minus 0.98. But it's not too bad, right? See, minus 0.818, you got minus 01.815. So, it's not too bad. You save a lot of memory, and you get similar accuracy. There'll be some drop in the accuracy, but for your practical use case, it will mostly works, okay? And this is how you plot it on the chart. Let's now look at normal float NF4 quantization. We previously saw that when you have in the two, possible values are four, okay? Minus two, minus one, zero, one. Total possible four values. So, you create four bins, and you map it, okay? What happens if instead of in two, you have int four? In int four, how many possible values are there? Folks, use your computer science fundamentals. Pause this video and tell me how many possible values are there. Well, total possible values will be 16, minus eight to positive seven. See, minus eight will be this, okay? This is a bit representation where this bit is a sign. So, sign one one bit means negative, and zero zero zero. Uh then one and one one one is minus seven, and so on. So, you create 16 bins and map them, okay? This is a linear mapping. You create 16 equal bins and map it. But this will create problems when we use quantization for our neural network, especially for LLM, because our weights, you know, all these weights, let's say you take that neural network, and you take all the weights, and if you plot a distribution, it will be a normal distribution. So, if it's a normal distribution, majority of the values are centered around zero. And when you are using regular in four, you are creating equal bins, correct? Equal equally spaced bins, and due to this, what will happen is majority of these values will be mapped to one value, right? Like like these values, let's say this is this value is, let's say, 30% of total values. Okay, I I don't know the exact number, but let's assume. Let's say this is 30%. And let's say this is 20% of the total values, and this is 20%. So, then all these values are being mapped to single value. Let's say this is minus two, this is minus one, zero. I think zero, one, two, three, four, five, six, seven, okay? So, majority of the values are being mapped to minus one and one. And when you do dequantization, you will lose a lot of precision. So, this is not a good approach, right? For a normal distribution, creating equal bins will not work. So, folks, please pause this video and tell me an idea which will work the best for this scenario. I'm telling you, it's a common sense. Okay, you have to just apply common sense. Pause this video and tell me how do you change this so that there is an equal mapping. All right, I hope you found the correct answer. The answer is you will not create equal bins. You will create bins based on the percentage of data points in your histogram. So, this total number of data points you have is 100%, right? And if you divide that by 16, you get 6.25%. So, you want 6.25% in each bin, and you will design that bin accordingly, okay? So, here, since in this histogram there are more number of data points, the width of the bin is lower, okay? So, if this is zero value, this might be zero to let's say 0.005. Let's assume that this is zero, this is one, and this is minus one. So, then this bin will be zero to 0.00 let's say one. Then 0.001 to 0.005, and so on. So, you create these bins in such a way that the data points, let's say these data points, right? Like so So, if you look at this particular area, this should be 6.25%. If you look at this particular area, this should be 6.25%. Okay? Let's look at the coding so that you get an idea. And this is called normal float four, right? Like normal normal for normal distribution. So, this is NF4 quantization. Let's look at the code now. So, in the code, what we're doing here is X is we are just creating 100 data points, okay? 100 random data points between minus one to one range. Let's say these are your weights of your neural network, and these NF4 values are 16 values. Okay? So, these values are minus 0.01 to minus 0.05. So, the distance between these two is 0.04. And then from here to here, the distance is 0.05. See, distance will increase. You see in this chart, this distance, okay? This distance right here is how much? 0.04. Okay? So, 0.04. But the distance between the dis- the width for the second bin, okay? Second bin, is what is this? If you add 0.05 into this, you get 0.1. So, this distance is 0.05. So, it will increase. Then the distance or the width for the third bin will be you even higher. So, 0.1 to 0.2 is 0.1, see? 0.1. So, you realize like when you increase that width, you can accommodate more data points. Because of this normal distribution, we need to do that. And these values are kind of fixed, okay? If you use some ready-made Python library like bits and bytes, you will get a better values, but this is a rough approximation. So, now let's quantize it. So, what you're doing essentially is uh once again, very simple. You are mapping all these data points to nearby value. So, what is a nearby value for minus one? This. So, you'll map it to this. What is more nearby to minus 0.97? Is it minus one or point 75? Well, it's minus one. So, you will map this value to here, okay? And what is most nearby to this? Is it minus 0.75 or minus one? Just not this. 0.87. Again, minus one. But if you look at minus 0.85, it is more near to this. So, these three values will be mapped to minus 0.75. These four values will be mapped to minus 1.0. You are just mapping it to nearby value. Very simple logic, and we have this Python function. And when you quantize it, see? You'll get this. See, all these values are mapped to minus one. Last three values are mapped to point minus 0.75 because it's it's more closer to it, okay? And this is how the distribution looks like. Q LoRA is essentially the quantized version of LoRA technique. Let's understand how this works. Say you are fine-tuning an LLM with with 65 billion parameter. Now, if this is using float 32, it will take 260 GB of memory to load the model. You will first quantize this using let's say NF4 quantization. And the quantized model will have a size of 32.5 GB because for each parameter is it it is taking half a byte. So, for 65 it will be 32.5. And then you apply LoRA fine-tuning using your domain-specific data set. This is what QLoRA is. It has few other elements as well, but just to go over definition of QLoRA, it's a memory-efficient fine-tuning method that combines 4-bit quantization with low-rank adaptation. Now, other than NF4 quantization, it uses a concept of double quantization and also page optimizers. If you look at their research paper, in the abstract itself, they talk about all these three points, which is 4-bit normal float quantization, double quantization, and page optimizer. So, we already looked at NF4 quantization. Let's look at what is double quantization. This is like zipping a file once again. So, you zip a file, you get a smaller version, then you again zip it, you get even a smaller version. So, same thing we do here. So, let's say you have 7 billion parameter model. To quantize it usually, you don't take all the parameters and quantize it in one shot. You will create a block of 64 parameters. So, you have all these 7 billion parameters, okay? You create a group of 64 billion parameters. And you will get n number of blocks, I think 109 million blocks in this case. And then you apply quantization, okay? NF4 quantization. So, each of these blocks will have less size and they will have their own individual scale values. If you remember quantization formula, there was scale and there was zero point, okay? So, for doing block-wise quantization, each block will have different scale. So, the scale one will be different than scale two and so on. It will also have a different zero point. Now, the people who wrote QLoRA paper thought that, okay, we got some optimization here, but the scales are still float 32, right? The scale is floating-point number. In our case, it was 0.0078 something. It is still a floating-point number. And you have so many such numbers. In this case, 7 billion parameters, you will have 109 million blocks, I think. And for each block you have one scale. So, 109 million float 32 numbers. So, what if you take all these scales and you quantize them again, okay? So, that way you also get some reduction in terms of size for the scale values and zero point values, okay? So, it's a simple concept just like zipping a file which is already zipped, okay? So, quantization you're doing and then you're doing a quantization again on the scale, etc. The third element of QLoRA is page optimizers. So, what happens is when you are fine-tuning this model, usually you'll use GPU. So, on GPU, let's say fine-tuning is going on. Sometimes it may run out of memory. So, you will use this paging concept similar to you know how paging works in terms of operating system. There is a page of memory which will be swapped in, swapped out of CPU memory back to disk and disk to CPU memory or RAM. We use similar concept here. So, when GPU is going out of memory, some of the pages that it has will be swapped out to CPU memory or RAM and then they will be swapped back in whenever needed. So, these are the three key elements of QLoRA technique. And I have the research paper open here. As you can see, you can fine-tune 65 billion parameter model on a single 48 GB GPU. So, you realize it says if 65 billion parameter with float 32, that multiplied by four, so 280 GB, whatever, it will be that huge number. So, I have a QLoRA paper open here. And here it says that a 65 billion parameter model can be trained on a single 48 GB GPU. This is humongous because 65 billion parameter model is around 100 260 GB. And if you can fine-tune that heavy model on a single GPU with 48 GB RAM, then that's really amazing, okay? I'm going to link this research paper. You can read it if you want to go into the details. We will use this Unsloth library, which is very popular when it comes to LLM fine-tuning. And they have this nice documentation which you can use to get help on API, etc. To install it, you will use pip install Unsloth on your computer. Now, to fine-tune, you need GPU on your computer. So, if your computer has GPU, you can do it locally, but I prefer doing this in cloud because in cloud you can get access to GPU as well as some of the dependency errors that I was getting while I was running locally, I did not get on cloud. So, I will start with Colab, okay? So, colab.research.google.com. Create a new notebook and then connect it with a GPU instance. So, by default this is connected to CPU. You can just say change runtime type. T4. So, this is giving you an access of Tesla T4 GPU, which is not very powerful, but good enough for our practical demonstration. If you have a paid account and if you can get access to higher-end GPU, then that's even better. I will demonstrate this entire thing through a pre-built notebook because running this code takes a lot of time, okay? So, I have already executed this notebook and I'll walk over it. So, in a Colab notebook, you first need to install Unsloth library, okay? Uh because when you connect with the Colab environment, it starts fresh. So, you can't just install it and save it. You have to install it, okay? So, in your cell, just run this after you have connected with that GPU instance. And then you will import fast language model class from Unsloth, okay? You'll also import PyTorch, which is torch. And here you are defining a bunch of parameters. And at this point, you are getting a pre-trained model. So, here we are fine-tuning Llama model, Llama 3.2 3B instruct. This is the model we are fine-tuning. So, you specify that. Then max sequence length is your context window length, okay? How much longer sequence you want to support in your fine-tuned model. Dtype none means let GPU figure out what kind of dtype will be the best. And by the way, I have provided help on all these parameters. See, it specifies the data type of model weights and computation. If you have something like Nvidia A100 GPU, then it will use this torch.bfloat16, okay? So, when you say dtype none, it will figure out the correct uh type for the storage. And load in four bit is used for 4-bit quantization, okay? So, we are going to use 4-bit quantization here. And when you run it, it will download a bunch of stuff because it is downloading this model. Uh and it is loading that model into this model parameter. You also get this tokenizer variable. If you look at this index, this looks similar to the Hugging Face Transformer library syntax where you say library class.from_pretrained, you load the model. Okay, so now I have Llama model here in this model variable. And I will fine-tune that, okay? You also need this tokenizer, okay? So, as a next step, what you do is you get the parameter efficient fine-tuned model, get PEFT model. So, here you are defining your parameters for LoRA. So, what do you need? Rank, okay? Rank I'm specifying as 16. You can also try eight, 32, etc. It's a hyperparameter, folks. So, I'm starting with 16, but you can start with eight, 32, play with it. Then for target modules, Q_proj, K_proj, V_proj, and O_proj refers to those uh remember those matrices? You remember these in a attention layer, you will have these four matrices. In feed-forward layer, you will have two matrices, W1, W2. And and and you will have few more in the output layer as well. So, that's what it is. Q_proj is for that uh WQ uh matrix, then WK, VV K, etc. Now, let's say if you don't specify this parameter, then it will not add that extra adapter, that extra delta W after that a particular matrix, okay? Now, since I have all these matrices uh defined, what it will do is see, you will have WQ and plus it will add delta W, right? Like delta W to this. Then since I have W K approach defined, it will have WK plus delta WK, something like that. Okay, so wherever you want to modify those matrices, you will add this parameter. So let's say if I don't add K approach, okay? If I don't add that, it will use WK original as a frozen layer. It will not add that adapter. I hope you're getting this point. This is coming from our LoRA lecture that we already discussed before, okay? And these are the matrices from the fully connected layer. Then we have LoRA alpha. So this LoRA alpha is this particular alpha. See, your equation is W0, which is original weight matrix, plus delta W. So A.B is your delta W. But you don't just do delta W, you add some scaling parameter. So that is alpha divided by rank, okay? And that alpha is this LoRA alpha parameter that you're specifying. This will tell you how much that adapter contributes to your weight update. Because this WW W0 is same as W that we have discussed before. This W is not changing, it's a frozen layer. What is changing is this A.B, which is delta W, correct? But how much delta W contributes to the final weights, you can configure that through this scaling parameter, okay? So that's what this LoRA alpha is. Then you have dropout. Dropout is for regularization. So while you're doing training, if you remember from our neural network lecture, if you want to drop certain neurons to kind of address overfitting, then you can specify that. In In our case, we are not going to have any dropout, we'll just say zero. Then this bias, okay? So I've given the information on all these parameters here, okay? Random state, all of that. So here, you are defining your LoRA model, your parameter efficient fine-tune model. So you loaded Llama 3 model, then you defined all these parameters for your LoRA, okay? Like R and LoRA alpha and so on. And now you got this new model, which is having all those LoRA parameters. As a next step, you will download the dataset on which you want to fine-tune your model. So here we are using ServiceNow R1 Distill SFT dataset. And this dataset library, by the way, it is a Hugging Face library. It allows you to access so many different datasets. I'm on Hugging Face website right now and I have this ServiceNow R1 Distill SFT dataset, as you can see. It has 172,000 rows and each row is sort of like you have a problem and you have solution. So see, this is a problem. It's like a puzzle. For Halloween, Sarah receives 66 pieces, whatever. This is like a puzzle and this is the solution. So you want to fine-tune your model on all these puzzles and you want to fine-tune it in a way that it will do reasoning similar to your DeepSeek model, okay? And for that, see, these are the two main columns in your dataset, right? Problem and this is a solution. And in solution, Sarah ate nine pieces of candy, see? The solution is this descriptive answer, but the actual number that you're looking at is nine, okay? And other than these two columns, you have third column, which is re-annotated assistant content. And this is a reasoning. This is like a think, you know, that think prompt that you get in DeepSeek. So if you want to solve this problem, this is how you will do reasoning step by step. You will say, first I need to determine total number of pieces candy, whatever. Next I will do this. This is sort of like how humans think, right? Like when you're given a puzzle, first you will do this first step, second step, third step and so on. And that's what this column is and at the end, there is this final answer. Now, in case you're wondering what is this {slash} boxed nine {slash} {slash} etc. Well, this is the LaTeX LaTeX markup language, okay? So let me just show you the It called LaTeX, but LaTeX is how you pronounce it. And this markup language, okay? What it does is people use it for in academia for scientific publications, etc. So let me just show you. Let's say you are having a research paper and you want to have all these mathematical equation, etc. For that, you will use this LaTeX package, okay? So it's like {slash} document {slash} use it. This is a markup language. Okay, folks, I hope you're getting what I mean by markup language. And see, LSD listing. So it has its own syntax. If you want to learn about it, you can go to YouTube. You'll find some LaTeX videos, but I don't think it's needed. It's just in in your brain, like you need to think that, okay, this is some kind of markup language used in this particular dataset, okay? So this is a dataset. Now, you load that dataset here through transformer dataset library, okay? So after the dataset is loaded, we'll look at the few records and see. It has a problem, it has a solution, it has an answer. See, problem. There were 27 boys, 35 girls. Remember in in high school days and college days, you used to get all these problems. There were fill in the blank children on the playground. So there's a problem. There's a solution. Solution is usually in that boxed parameter. Okay, so solution will be here somewhere and there is this reasoning pattern. Now, you take all those records. So there are 172,000 records in our dataset that we will use for fine-tuning. In this particular code, what we are doing is we take each of those records and you put it in this prompt. So you're saying you are a reflective assistant, blah blah blah, engaging iterative reasoning, mimicking human stream of your your your approach and present exploration start out, whatever. So you give this uh prompt, then here is a problem. So there are three brackets, right? So this is a problem, okay? So you you go through, see, you have a dataset, right? So you go through each record. When you say dot map, this dot map function will go through each record one by one. For each record, this function is called, okay? And and in every record, what do you have? Problem, solution and DeepSeek-like step-by-step thinking block, okay? For them, what you do is you take R1 prompt and R1 has these three parameters, right? So when you do format, it will in first block, it will first put problem, then in the second one, it will put thought, third, it will put solution. Okay? And it will return you the text. So for each of the records, just imagine you're getting this big prompt for each of the puzzle you have. After that, what happens is you're creating a trainer object. Now, trainer object is coming from transformer reinforcement learning library, which is once again Hugging Face, okay? So this is the library, folks. It's a Hugging Face library, transformer reinforcement learning library. And when you did pip install on sloth, it kind of install all these because these are the dependencies. So in that library, there is this SFT trainer, okay? Like supervised fine-tune training trainer, which you are importing and you are giving the base model, that LoRA model, okay? Then see, the the model is not trained yet, okay? You have just defined the configuration. So this model has that LoRA configuration that we gave here. See, this this particular configuration. So it knows what is the value of R, what is R LoRA alpha and so on. Then you will use the dataset and tokenizer. So when you run the training, see, later on I will say trainer.train. So when I do that, it will use this tokenizer for tokenizing. It will use this dataset to load the record and to train the model. Okay, so these are all the parameters, these are training arguments and these are some of these are familiar to you when you looked into the deep learning module. We had all this learning parameter, learning rate, weight decay, all of those, okay? Like optimizer. So Adam is your optimizer here and I have given information on all those parameters here, folks. Read through it. You don't need to remember the syntax here. You can get the syntax through ChatGPT or through the on sloth documentation. So don't stress too much, okay? Like right now you might be thinking, oh, this guy is just showing the code and he's not typing. Well, typing is not needed nowadays in the world of AI. It's like how you make use of those AI tools and make yourself productive while still learning the fundamentals. That is the entire uh goal that we need to have. Okay? So, you created this trainer object and then you will do trainer.train. It will take long time, folks. For me, I think it took like 20 minutes. And it is showing you the epochs and the training loss and you as you can see the loss is reducing, which is a good sign, which means my model is training in the right direction. And after 60 epochs, my loss is 0.48. So, I read somewhere in the documentation if it is less than 0.5, it's good enough. Uh I mean, it actually depends on your use case, but that's a general guideline. And then you will try some prompt. So, my model is trained, okay? After that that line, when you the trainer.train, model is trained. Which means model is fine-tuned. So, we took Llama 3 model. We trained it on the ServiceNow R1 dataset, okay? Which dataset? This dataset. 172,000 rows. And that Llama 3 model was not originally trained to think like this, to think like DeepSeek, you know, like step-by-step. But now that we fine-tuned it, the model will also start thinking like DeepSeek, okay? So, here I'm giving one sample uh question. How many Rs are present in strawberry? And folks, this might look like a simple problem. Actually, LLM gets confused with these problems, okay? So, it's kind of like a hard problem for LLM. So, I give this uh problem uh statement to my model, okay? And here you are doing tokenizer, whatever the basic setup, essentially. And then here is where you are doing inference. So, you'll say model.generate. Okay? Um whatever all these parameters, temperature, and so on. And whatever output you get, you will decode it into a response. So, here, if you look at the response, let me just show you the response. See? Now it is thinking step-by-step. Let me recap. See, let me recap. So, it is actually thinking step-by-step. It is saying that, "Okay, um you know, it goes through the entire sentence letter by letter and it's saying, "Okay, R is one. Here R is one." and so on. So, it it is is also doing double-check because in our uh prompt, we asked it to do human-like thinking. And then in the end, it will tell you total R R 2. Okay? Isn't this amazing, folks? We trained the model. We fine-tuned it on this ServiceNow R1 dataset. And Llama 3, which was not originally designed to produce output in this format, now it is able to do reasoning, okay? You can run this in Google Colab. It's going to take long time. Um the code is attached. So, folks, please read through the code. Once again, let me remind you. I know I'm saying this again and again. You don't need to remember syntax, folks. If you waste your mental energy in remembering syntax, you are not doing yourself any good. You can get all this help through ChatGPT AI and API documentation. Your goal is to remember fundamentals, okay? And connect those fundamentals on how do you solve a business problem. All right. So, that's about it. Uh the code is provided. Please run it in your Colab notebook and uh see how it goes.