PEFT, LoRA & QLoRA Explained

You don't need a massive supercomput to fine-tune a 70 billion parameter language model. Parameters are just the numbers that store everything the AI has learned. Today we are breaking down the complete anatomy of modern finetuning from the base model down to the actual math and physics of Laura and Qura. To understand fine-tuning, we start at layer 1 pre-training. A giant cluster of GPUs ingests massive amounts of raw internet data to build a base model. Think of it like reading the dictionary. The model understands grammar and facts perfectly, but it doesn't actually know how to have a conversation or follow instructions yet. To make it useful, we have to fine-tune it. You could do continued pre-training for new languages or human feedback for safety. But the most crucial path is instruction tuning, teaching the base model to answer questions exactly like a chatbot. The problem, a standard 70 billion parameter model stores its weights in 16bit precision. That base memory alone takes up roughly 140 GB of VRAM. VRAMm is just the dedicated memory on your graphics card. It's where the entire model has to physically fit while it's learning. But during full fine-tuning, you also have to store the optimizer states and gradients. Gradients are the feedback signals that tell each parameter which direction to adjust. To store all of this, you suddenly need over 500 GB of VRAM. If you try this on a standard GPU, it simply crashes with an out of memory error. This is the memory wall and it's exactly why we use parameter efficient fine-tuning, specifically quantized Laura. Layer two, the targeting. Before the math, we prep the data. In instruction tuning, we format our data clearly as a prompt and a response telling the model exactly what we want. But we don't want the model wasting compute power trying to learn the prompt we just gave it. So we apply the minus 100 trick. We assign a special label of minus 100 to all the words in the prompt. The loss function is what scores the model's accuracy. In PyTorch, this function is hard-coded to completely ignore the label minus 100. This means gradients, those feedback signals are only calculated for the response. It is highly targeted and highly efficient. Layer three, the bypass. Inside a neural network, knowledge is stored in giant grids of numbers called matrices. Instead of updating the massive 16 million parameterbased matrix, Laura puts a padlock on it. We freeze those weights completely. Laura bets on one key insight. You don't need to change every single weight in the factory to teach the model a new task. The required changes are actually very small. So we bypass the frozen factory by injecting two tiny matrices A and B. Instead of updating 16 million parameters in the original grid, we only train these two small matrices. We've just reduced our trainable parameters from 16 million down to 65,000. During inference, which is when the model is actually generating a response for you, we multiply matrix B by matrix A. This creates a low rank update matrix of the exact same dimensions as the base grid, which is then cleanly added to the final output. The physical size of this bypass pipe is controlled by a hyperparameter called rank. A higher rank means a wider pipe, giving the model more capacity to learn complex tasks, but it does use slightly more memory. We also set an alpha. Think of alpha as a literal volume knob. It's a scaling factor that controls how loudly your new adapter weights speak over the original base model. Layer 4, the squeeze. Laura fixes the gradients, but the base weights still take up 140 GB. Chlora solves this using quantization. Imagine rounding every price in a store to the nearest dollar. You lose some precision, but save massive space. We squeeze 16 bit floats down to 4bit buckets. Standard 4bit quantization creates 16 evenly spaced buckets. But neural network weights aren't evenly distributed. So standard uniform buckets waste massive precision on rare outliers at the edges while the center buckets completely overflow losing vital information. Because weight values follow a natural mathematical distribution they actually form a perfect bell curve. Most values cluster tightly together right around zero with only very rare outliers existing at the extreme ends. Qura introduces normal float for. It dynamically shrinks the buckets near zero and widens them at the tails. By hugging the exact curve of the data, every bucket holds the exact same number of weights. We get maximum precision exactly where the majority of the intelligence lives. To reconstruct the 4-bit data later, we need 32bit mapping constants. Chlora takes it a step further with double quantization. literally shrink wrapping those mapping constants down to eight bits, saving even more VRAMm. Even with Qura, training isn't completely flat. VRAMm usage spikes rhythmically as optimizer states are updated during training, threatening to cause a catastrophic memory crash at the very peak. To prevent this, we use paged optimizers. It acts as an automatic safety valve. When RAM spikes dangerously close to 100%, it temporarily drains those heavy optimizer states across the PCIe bus into your system slower but much larger CPU RAM, targeted data masking, low rank bypass matrices, normal float for quantization, and paged memory valving. That is the complete anatomy of modern finetuning. Now you know exactly what's happening under the hood.