Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
📰 ArXiv cs.AI
Using the same optimizer for pretraining and finetuning can reduce forgetting in large language models, leading to better performance on new tasks
Action Steps
- Choose a suitable optimizer for pretraining a large language model
- Use the same optimizer for full finetuning on a new task
- Compare the performance of the model with and without optimizer consistency
- Evaluate the tradeoff between learning and forgetting in the model
- Apply the optimizer consistency technique to achieve better results on downstream tasks
Who Needs to Know This
Researchers and engineers working with large language models can benefit from this insight to improve their model's performance and reduce forgetting, especially when finetuning on new tasks
Key Insight
💡 Optimizer-model consistency can improve the learning-forgetting tradeoff in large language models
Share This
🚀 Using the same optimizer for pretraining & finetuning can reduce forgetting in LLMs! 🤖
Full Article
Title: Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
Abstract:
arXiv:2605.06654v1 Announce Type: cross Abstract: Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretraining achieves a better learning-forgetting tradeoff, i.e., forgetting less while achieving the same or better performance on the new task, than other optimizers and, possibly surprisingly, LoRA, during the supervised finetuning (SF
Abstract:
arXiv:2605.06654v1 Announce Type: cross Abstract: Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretraining achieves a better learning-forgetting tradeoff, i.e., forgetting less while achieving the same or better performance on the new task, than other optimizers and, possibly surprisingly, LoRA, during the supervised finetuning (SF
DeepCamp AI