LoRA Config Parameters Explained: Master Your Fine-Tuning

Every LoRA tutorial gives you the same four lines. R = 16, LoRA alpha = 32, target modules = Q_proj and V_proj, bias = none. Nobody tells you why those numbers, what happens if you change them, or which of the other dozen parameters in LoRA config you're silently leaving at their defaults, and whether that's actually fine for your case. Welcome back to the fine-tuning series. This one pairs with LoRA. Today we go parameter by parameter through LoRA config. R, LoRA alpha, target modules, dropout, bias, rank stabilized scaling, modules to save, initialization strategy, and per layer overrides. What each one controls, the tradeoff it sits on, and what to actually set it to. Before the parameter list, one formula, because every parameter in LoRA config maps onto one piece of it. LoRA replaces a full weight update with a low rank approximation. The output is the frozen base weight times the input plus alpha over R times B times A times the input. B starts as all zeros, A starts as random values, so at step zero that whole second term is exactly zero. The adapted model starts out numerically identical to the base model, and training grows the delta from nothing. Every other parameter in LoRA config either changes what gets this treatment, target modules, how it's regularized, dropout and bias, how big B and A are, R, how strongly the result is scaled, alpha, or how B and A are initialized. Keep this formula in mind. Every slide from here maps onto one piece of it. R is the rank, the inner dimension shared by B and A, and it's the single most consequential number in this config. R directly sets how many trainable parameters the adapter has, and it scales linearly. Double R, roughly double the adapter's parameter count. It's also the adapter's representational ceiling. A low R genuinely cannot represent as complex an update as a high R can, no matter how long you train. PEFT's default is R = 8. In practice, 8 to 16 is usually enough for stylistic or format adaptation, teaching a model your output format, tone, or a narrow task. 32 to 64 is where you land if you're teaching genuinely new capability, a new language, a new domain, more complex reasoning patterns, because that needs more representational room. The rule of thumb, start around 16, and only raise it if you see a specific signal. Training loss keeps dropping while validation loss plateaus, suggesting the adapter has run out of capacity to keep improving. Don't raise R by default just because higher numbers feel safer. You're mostly just adding parameters and overfitting risk. LoRA alpha doesn't add capacity. It controls how strongly the adapter's output gets weighted before it's added back to the frozen weight, through that alpha over R scaling factor from the formula. Increasing alpha without touching R is functionally similar to increasing the adapter's effective learning rate. Same capacity, stronger update per step. The common starting convention is LoRA alpha equals 2 * R. R 16, alpha 32. R 32, alpha 64, which keeps that alpha over R ratio at two. It's an empirically reasonable default, not a law of nature, so treat it as a starting point. Here's the part people miss. Alpha and learning rate interact. If you double alpha, you're roughly doubling the magnitude of every update the adapter makes. So, if you don't also lower your learning rate to compensate, training can become unstable, loss spikes, gradient blow-ups, especially as R climbs higher. And at high rank specifically, this alpha over R scaling has a known weakness, which is exactly what rank stabilized LoRA fixes. More on that in a few slides. Target modules is the biggest lever you have, full stop. It decides which layers actually get a LoRA adapter at all. Everything not named here stays completely frozen. For a LLaMA-style model, your option scale up roughly like this. Attention only, Q proj and V proj, is the original LoRA paper's setup and the smallest footprint. Add K proj and O proj for all four attention projections, and you get a bit more capacity. Target all linear layers, attention plus the MLPs gate, up and down projections, and you get the most capacity. And in several empirical comparisons, including the Q LoRA paper, the best results from most fine-tuning objectives at the cost of more trainable parameters. One practical note, these module names are architecture specific. Q proj and V proj are correct for LLaMA family models, but a different architecture will use different names. Don't copy a target modules list from a tutorial built for a different model. Run a quick scan over the model's named modules and confirm the actual linear layer names before you write your config. Two more knobs, both usually fine left near their defaults. LoRA dropout applies dropout only inside the LoRA path. The frozen base weights are untouched, so it regularizes specifically the part you're training. Typical range is 0.0 to 0.1. Push it towards 0.05 to 0.1 if you're training on a small dataset prone to overfitting. Leave it at zero if your dataset is large and diverse, which is also why frameworks like Unsloth default to zero. Bias controls whether bias terms get trained. None trains no biases, all trains all bias terms in the model, and LoRA only trains only the bias terms inside adapted layers. None is correct for the overwhelming majority of setups. Here's the gotcha if you do change it. Setting bias to all or LoRA only means that even when you disable the adapter, the model will no longer produce the same outputs as the original base model because you've actually modified shared non-adapter weights. That's a real behavior change, not just a training detail. Use RS LoRA addresses a specific weakness in the standard alpha over R scaling. As rank grows large, that scaling shrinks the effective gradient magnitude in a way that empirically destabilizes training. Rank stabilized LoRA changes the denominator from R to the square root of R, alpha over square root R instead of alpha over R, which keeps gradient magnitudes more consistent as you push rank higher. It's a one-line change. Set use RS LoRA equals true in your config. Use it whenever you're working at rank 32 or above and especially at rank 64 or higher. Down in the typical eight to 16 range most fine-tunes use, it usually makes little practical difference. So, it's not something you need to flip on by default, just something to remember when you're pushing rank up for a capability-heavy fine-tune. Modules to save is for the parts of the model that shouldn't get a low-rank adapter at all. They should just be trained directly in full. The two common cases, a classification head you've added for a new task where the head itself needs real capacity, not a rank-limited approximation, and embedding layers or the output head when you've added new tokens to the vocabulary. Since a brand new tokens embedding starts essentially random, and the low-rank update can't realistically learn a good representation for it from scratch. The catch, anything listed here gets saved in full inside your adapter checkpoint, not as a small low-rank delta. A few new token embeddings are cheap. Accidentally including the full embedding matrix or the full LM head can balloon what should be a 20 megabyte adapter into hundreds of megabytes. So, use this list deliberately, not by habit. In it, LoRA weights controls how matrices A and B start out. And the default is doing something specific on purpose. The default, true, initializes B to all zeros and A to random Gaussian values, which is exactly the formula from slide two. It guarantees the adapted model starts out numerically identical to the base model, and the delta grows from nothing as training proceeds. There are alternatives. Gaussian randomizes both matrices, which is rarely worth it. PiSA, principal singular values and singular vectors adaptation, initializes the matrices from the base weights own singular value decomposition instead of randomly starting the adapter already aligned with the most important directions in the existing weight. That can converge faster on some tasks, but it costs a one-time SVD computation per layer at startup and adds setup complexity. Stick with the default unless you have a specific reason to change it. Pisa's faster convergence is real in some benchmarks, but it's worth trying once your basic config is already working, not as your first attempt. Rank pattern and alpha pattern let you override R and alpha for specific layers instead of applying one global value everywhere. Each takes a dictionary mapping a layer name pattern to a different rank or alpha for just that layer. For example, giving the last few transformer layers a higher rank than the rest, since deeper layers often carry more task-specific representation in fine-tuning studies. This is an advanced optional knob. Most fine-tunes never need it. Reach for it only after you've already validated a simple uniform rank config and have a specific hypothesis worth testing, not as something you can figure on day one. Let's put all of it together into one starting config you can actually copy. R of 16, alpha of 32, keeping that two-to-one ratio. Target modules covering attention and MLP from maximum capacity. Dropout at 0.05 as a light regularizer. Bias left at none. Rank stabilized scaling off since R 16 doesn't need it. Default zero initialization. This is a sensible starting point for instruction fine-tuning a model in the 7 billion parameter class, not a universal answer. Iterate in this order. Adjust R first based on the under or over-fitting signal from part nine's evaluation stack. Only revisit target modules if accuracy plateaus even after R looks right. Everything else on this list stays at its default until you have a specific reason to touch it. Seven takeaways. One, R sets capacity and parameter count. Start around 16 and raise it only on an underfitting capacity bound signal, not by default. Two, LoRA alpha scales the adapter's contribution through alpha over R. Start at alpha equals two times R and remember it interacts with your learning rate. Three, target modules is the biggest lever you have. Targeting all linear layers usually beats attention only for capability heavy fine-tunes. Four, LoRA dropout and bias are best left near their defaults. Small dropout, bias equals none for most setups. Five, use RS LoRA swaps alpha over R to alpha over root R. Turn it on once you're using rank 32 or higher. Six, modules to save fully fine-tune specific modules alongside the adapter like new heads or resized embeddings, but it inflates checkpoint size. So, use it deliberately, not by habit. Seven, init LoRA weights and rank or alpha pattern are advanced knobs. Validate a simple uniform config first and only reach for these with a specific hypothesis worth testing. That's every parameter in LoRA config, what it controls, the tradeoff it sits on, and what to actually set it to instead of copy-pasting four lines and hoping. Resources, the PEFT documentation for the full parameter reference, the original LoRA paper for the math, and the rank stabilized LoRA and PISA papers for the advanced initialization and scaling strategies covered today. Like the video if it filled in gaps your other LoRA tutorials left out. Subscribe for the rest of the series and drop a comment with which parameter you'd never actually looked up before this video. Configure deliberately, not by habit. See you in the next one.