Training-Free On-Policy Distillation.

arXiv:2610.22236v2 Announce Type: replace-cross Abstract: On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need trainin