LoRA Hyperparameters Explained: Choosing Rank, Alpha, and Target Modules

Key Takeaways

Hi there, welcome back. In this video, I want to talk about the three key hyperparameters for Lura and how you go about setting them. There are three the first one is rank R. You have the scaling factor alpha and then the target modules in the neural network that you are trying to fine-tune. So let's start with rank R. As you can imagine, R decides the dimensionality of those A and B matrices, the adapter matrices that we are using to fine-tune any particular matrix within the neural network. Now, the more the higher the value of R, the more expressive your finetuning capacity is going to be. If you have higher R, the model will be able to pick up more signal from the training data set. Having said that, the whole purpose of LoRa is to control how much compute and memory you need. So for the most part when you're setting R, one of the most important governing factors is how much memory you have. So it's really more like a filter on what you can do. Still, it's important to understand what effect it will have as you change the values of art. It's helpful to look at the original paper and see what they found in their experiments. So we'll take a look at that first. There's a section where they talk about the optimal rank R for Laura. Interestingly, what they found is even very low values of R can give you pretty competitive results when you're comparing with the full fine-tuned model. So here they actually have that lines. Surprisingly, Laura already performs competitively with very small R. So you have to imagine if values as low as 1 2 4 and 8 can actually produce as good a performance as if you had the original matrix matrix fine-tuned then what does that tell you about the necessity for having really large language models right low rank representations of these matrices are good enough to represent a signal that's a topic for another discussion but it's still helpful for us to know that we can have really small values of R and they still perform really well right the thumb rule The convention is to start with a value of D. That's the most common value. I see it expressed in various articles. If you were really limited by memory, then you could go lower four and you can try 16 and 32 if you can afford to do that in terms of the compute power. Of course, as you increase more, you expect certain improvement, but a lot of the research that exists out there shows that you were not really going to gain much improvement in performance at higher values of R. So the common range is going to be somewhere between 4, 8, and 16. That's the most common values you're going to see when people are fine-tuning large language models. So let's come back now. I'll talk about the next one which is alpha. Alpha was the one that originally confused me as to the purpose. Why do we even need it? Because as you might have understood, it's a global constant, right? If it's a global constant, why do we even need it? Because you can imagine like even during the training mechanism, the learning rate is a constant. So if you have a constant times another constant, we just need one constant why you need a second one. That was a little confusing but there's an explanation and the authors described that and the reason is it's more like during the experimentation phase what is happening is you might pick a value of r like 8 and run your experiment and then you try a value of 16 or four and you run another experiment and what happens is you have matrix multiplication of a and b you have if you're choosing a value of rank equals to 8 you're multiplying doing a sum product of two two vectors of different of size eight And if you go to 16 now you're adding up 16 numbers, right? And so what is happening is you have a scale issue there. If you don't reset the value of the learning rate, you could have very unstable training. You might know from your prior experience that a very high value of learning rate, you might even not learn, right? LOS starts increasing and you have to bring it down. On the other hand, if the learning rate is too small, you're learning too slow. and alpha as you're doing the experiments with the rank what is happening is in a way you also have to go and back go back and change learning weight and the author said if you choose this alpha which is a function of R then that will help scale the magnitude accordingly no matter what R you choose that some product that matrix multiplication is going to result in some values that are in a reasonable range you won't have to go and change the learning rate because of that I hope that made sense it's a scaling issue as you're flexing the value of are you don't want to have to go back and change the learning rate every time. So that's the explanation I understood from reading the paper and other articles. What the authors recommend is to choose the alpha equals to 2 * R and really the range you can use is 1* R to maybe three or four times R and stay within that range and your training is more likely to be stable. So that is scaling factor to help with stability in your training mechanism. All right. Now let's talk about the last one here. Target modules. This is the one that I'm going to give you the recommended answer and I'm going to tell you how I felt about it. It's the most unsatisfying answer I have when I was reading the paper that how you go about choosing target modules. The recommendation we usually see is that if you look at the attention block there's basically four matrices, four projection tensors there. So you have Q, K, V, and O. Now if you don't know what they are, you should read the attention paper as I mentioned before. Q is for query, K is for key, V is for value, and O is for output. The authors recommended that if you want to start with the default setting, you should try to target the query and value matrices or modules in the network. You can read through the numbers here. If you target all four in the attention block, it's really not much different. And so, might as well just go with the lower number of modules, you're tuning lower number of parameters. So the reason this is unsatisfying to me is it's basically saying no matter what your use case is start with this that to me doesn't quite sit well because different use cases should have different uh modules you want to target. This also goes back to one of the videos where I mentioned that the transformers one of the deficiencies is that they're like a black box. You can't really know what is being learned in what module. This is really getting back to that if you don't know what is being learned in what module how can you pick the right target module for lura you just don't know what to do there. So this is the default values that they're giving us. Still one of the intuitions I was having when I was thinking about this is imagine you are trying to tune the style and tone of the LLM. You're not trying to teach it any new knowledge. You just want it to change its t this its tone. If you think about those four attention block modules, I was imagining that maybe it's the output module because what it does it once it's retrieved the information, it's bundling it together and passing it on that is the module whose responsibility it is. At least that's the intuition around the attention block. So if you're changing the style and tone, not actually the information that is being retrieved, then maybe it's the O module that you should be targeting. But I didn't really see that anywhere like when you should be targeting Q and K and V and O there is no distinction being made. So that is the issue here which is you just go with convention what seems to have worked and for the most part the researchers have indicated it's Q and V matrices. Coming back here let's just summarize. I've mentioned R. Usually the value of 8 4 or 16 by three are given by your memory constraints. Alpha is just a function of R. Usually you would just set it to do 2 * R and that's that. And then target modules, you can read through here. If you're trying to do some smaller fine-tuning job, hit the Q and V values. And if you're trying to do bigger fine tuning, maybe you're trying to teach it new domain knowledge. Then you can go for all the attention blockers. Those are the recommendations. But as I've said before, machine learning, deep learning is more about experimentation. So start with this. But you always want to try whatever works for your own use case. You go with that. Okay. I hope you found that helpful and look forward to your projects as you work through this and what you find when you're tuning on your own use cases. All right, I'll see you in the next video. Thank you.