2026-10-08 · America/Los_Angeles · 论文 · #20

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model…

热度 48.5 / 100；排名与评分保留该期记录。