Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

1. 1Identify the failure modes of on-policy distillation in long-horizon settings
2. 2Analyze the effects of reducing distribution matching to a one-token signal
3. 3Explore simple fixes to improve the reliability of on-policy distillation, such as modifying the teacher feedback mechanism

ML researchers and engineers working on large language models can benefit from understanding the limitations of on-policy distillation and how to address them, as it can impact the performance of their models

💡 On-policy distillation can be unreliable in long-horizon settings due to the reduction of distribution matching to a one-token signal

💡 On-policy distillation for LLMs can be fragile, but simple fixes can help #LLMs #OPD