Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

arXiv:2609.33791v1 Announce Type: cross Abstract: Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a dis