Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts

1. 1Study the problem of width expansion as a candidate-selection problem over full training states
2. 2Compare different warm start strategies, including exact-copy, perturbative, asymmetric-reset, and structured non-clone methods
3. 3Evaluate the effectiveness of each strategy in a small-scale proxy, such as TinyStories
4. 4Select the optimal warm start strategy based on the evaluation results and apply it to larger-scale language model training

ML researchers and engineers working on language model development and fine-tuning can benefit from this research, as it provides insights into improving width growth and selecting optimal warm starts

💡 Regime-sensitive selection of dense LM warm starts is crucial for effective width growth

💡 Width expansion for LMs requires more than just preservation #LLMs #WidthGrowth