Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

1. 1Apply width pruning to GLU-MLP layers in Llama-3.2 using the Maximum Absolute Weight criterion
2. 2Evaluate the impact of width pruning on instruction-following capabilities using IFEval metrics
3. 3Compare the performance of pruned models on tasks relying on parametric knowledge, such as MMLU and GSM8K
4. 4Analyze the trade-offs between instruction-following and parametric knowledge capabilities in pruned models
5. 5Test the robustness of instruction-following capabilities in pruned models using various evaluation metrics

Researchers and developers working with large language models like Llama-3.2 can benefit from understanding the effects of width pruning on model capabilities, particularly in instruction-following tasks

💡 Width pruning reveals a dichotomy in Llama-3.2 model capabilities, highlighting the need for careful evaluation of trade-offs between instruction-following and parametric knowledge

💡 Width pruning in Llama-3.2 improves instruction-following by up to 75%, but degrades parametric knowledge performance