How Much GPU Memory Is Needed for LLM Fine-Tuning?

when fine-tuning a model how much GPU memory is required this is a question that many people are curious about let's take an example suppose we want to fine-tune a 1B parameter model here 1B means a model with 1 billion parameters if you want to fine-tune larger models you can scale the calculations proportionally additionally let's assume the model parameters are represented using 16bit Precision since 16bits equal 2 bytes we use this for the calculation some models may use 32bit Precision but the logic Remains the Same to calculate GPU memory requirements we need to understand which components consume GPU memory during fine-tuning the main components are model weights the model itself needs to be loaded into GPU memory the size of the model determines the memory required gradients during training gradients are computed to update model parameters these gradients require memory Optimizer States Optimizer states such as those used for momentum or adaptive learning rates also occupy memory the memory requirement varies depending on the optimizer used such as sged or atom activation Maps intermediate activations during forward passes also require memory but but their contribution is smaller compared to the above components for Simplicity the majority of memory usage comes from model weights gradients and Optimizer States for model weights a 1B parameter model with each parameter requiring two bytes results in the following memory usage 1B parameters time 2 bytes equals 2 GB for gradients the memory requirement is typically the same as for model weights 2 GB for Optimizer States the memory requirement is usually about four times that are the model weights 2 GB * 4 is equal to 8 GB adding these components together the total memory required is 2 GB weights plus 2 GB gradients Plus 8 GB Optimizer States equals 12 GB this means the total memory requirement is roughly six times the size of the model weights for larger models you can scale this calculation proportionally for example for a 7B parameter Model 12 GB * 7 = 84 GB this would typically require two a 100 gpus each with 40 GB of memory this approach is called Full fine tuning in addition to full fine-tuning another commonly used method is parameter efficient fine tuning PFT a widely recognized PFT method is Laura low rank adaptation Laura reduces memory usage by fine-tuning only a small subset of additional parameters while keeping the original model unchanged for example if the original model has 1 billion parameters lurine Tunes only a fraction such as 2% or even less these new parameters called adapters are then added to the original model to create the fine-tune model visually you can think of the original model remaining unchanged while the new adapter parameters are added on top of it for instance if the adapter represents 2% of the original model's parameters the adapter is much smaller for lur of fine-tuning the original model still needs to be loaded into GPU memory but the memory required for gradients and Optimizer States is much smaller because it only applies to the adapter parameters for example if the adapter weights are 2.5% of the original model size adapter weights 2 GB * 0.025 = 0.05 GB gradients and Optimizer States for the adapter 10 GB * 0.025 = 0 25 GB adding these components model weights 2 GB adapter weights 0.05 GB gradients and Optimizer States 0.25 GB the total memory required for lurine tuning is approximately 2 GB plus 0.05 GB + 0.25 GB = 2.3 GB in practice the adapter's parameter size is often even smaller than 2.5% to further reduce memory requirements another approach is Q Laura quantized Laura Q Laura compresses the model weights using quantization techniques reducing their memory footprint for instance 16bit weights can be quantized to 8 bit or 4bit reducing memory usage from 2 GB to 1 gab or 0.5 GB with this method memory consumption is significantly reduced finally it's worth noting that the calculations presented here are basic estimates in real world scenarios additional factors need to be considered whether to use single GPU or multi-gpu setups whether to train on a single node or across multiple nodes the use of Frameworks like deep speed with techniques such as 02 or 03 for more precise calculations I recommend Consulting a detailed technical article linked in the comments