An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models

1. 1Compare SFT-only, DPO-only, and staged SFT-to-DPO training methods
2. 2Evaluate the performance of each method on tasks like paraphrase detection and Shakespearean sonnet continuation
3. 3Consider using full fine-tuning (FFT) or LoRA as alternatives to SFT and DPO
4. 4Analyze the results to determine the most effective approach for small language models

ML researchers and engineers working on language models can benefit from this study to optimize their models' performance, especially when using small backbones and modest data

💡 DPO can provide small, task-dependent improvements in language model performance, but its effectiveness depends on the specific task and model architecture

🤖 DPO yields small task-dependent gains in small language models 📊