Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives.

arXiv:2609.38666v1 Announce Type: cross Abstract: On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). However, its benefits and fragilit