Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs

1. 1Evaluate the performance of uncertainty-based Active Preference Learning (APL) against Random sampling in online DPO
2. 2Assess the impact of modern LLMs' strong priors on post-training data-selection strategies
3. 3Consider the trade-offs between query efficiency and the richness of on-policy candidate pools
4. 4Investigate the conditions under which Random sampling outperforms APL in online DPO

Machine learning researchers and engineers working on LLMs and online DPO can benefit from understanding the limitations of active selection strategies, and how Random sampling can be a surprisingly effective approach

💡 Modern LLMs' strong priors can limit the effectiveness of active selection strategies, making Random sampling a surprisingly effective approach

🤖 Random sampling can be a strong baseline in online DPO with modern LLMs! 📊