Understanding Model Quantization and Distillation in LLMs

Key Takeaways

in this video I will explain what model quantization and distillation are these two techniques fall under the category of large language model compression when deploying large language models and providing services to users the costs are often very high therefore we often aim to compress large language models to reduce these costs compression can be understood as reducing a very large model into a smaller one thus directly lowering costs and often speeding up model inference first let's talk about quantization to understand quantization we need to recognize that large language models consist of many parameters for example GPT 3 has over 100 billion parameters each parameter can be thought of as a value such as 1. 12768 to to store this value memory space must be allocated the required space depends on the Precision of the parameter lower Precision generally requires less space for instance if we round this number to a lower Precision value such as 1 or 1.28 the storage space required for these numbers is smaller than for the original value this is the core idea of quantization using more precise computer terminology a parameter in a large language model is typically represented as a float 32 type requiring 32 bits of space or four bytes these are units of memory with larger units like MB GB or TB used for larger scales if we convert float 32 parameters into float 16 or inate types the space required is reduced for example float 16 uses 16 bits half the space of float 32 and8 uses 8 Bits one4 the space of float 32 quantization then is the process of converting each or some of the models parameters into lower Precision data types this reduces storage requirements and speeds up inference however this process inevitably results in some loss of information will quantization affect model accuracy if the quantization process is well managed the resulting model can maintain reliable accuracy quantization is currently one of the most commonly used methods for compressing large language models next let's talk about distillation distillation is fundamentally different from quantization its Essence lies in imitation for example suppose we've trained a very large model with hundreds of billions of parameters this model is too large so we want to compress it using distillation in this method we construct a smaller model and train it to mimic the behavior of the larger model the idea is similar to a child imitating an adult whatever the large language model does we want a small model to do as well here's the process given an input for example a prompt we feed it to the pre-trained large language model as teacher model which generates an output the same input is then fed to the small model as student model which also generates an output the goal is for the small model's output to match the large model's output as closely as possible the better the match the more successful the distillation through this process the small model learns to replicate the behavior of the teacher model the smaller model requires Less storage space and has faster inference speeds distillation is frequently used during large language model training for example if GPT for is the most advanced model on the market we can use GPT 4's inputs and responses to train our own model this allows our model to mimic GPT 4's Behavior closely many open- Source models on the market are trained using distillation techniques Beyond quantization and distillation other techniques like pruning can also be used for model compression however pruning is less practical for large language models the two most mainstream methods remain quantization and distillation