On-Policy Transition Distillation.

arXiv:2610.28048v1 Announce Type: cross Abstract: On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts. We investigate whether depthwise pred