On-policy distillation gives a student dense token-level supervision along its own rollouts, but every teacher query in standard OPD is local: the teacher scores only the next token at a student-generated context, and that judgement is unreliable when the prefix it conditions on is incomplete or low-quality. The paper's own baseline table shows how badly this can bite — distilling Qwen3-4B into Qwen3-1.7B-Base, plain SFT on teacher-generated trajectories beats vanilla OPD on maths, 36.33 against 33.65 Math Avg., with OPD's AIME24 pass@16 collapsing to 13.3 next to SFT's 23.0.

The diagnosis is a measurement the authors name Teacher Uncertainty Contraction (TUC): let the teacher keep generating from a student prefix and its own predictive uncertainty falls with depth. Entropy and Gini impurity both decrease along teacher-generated continuations, fitting exponential saturation at R² = 0.982 and 0.990, while no comparable contraction appears along student continuations — so the effect comes from the teacher conditioning on itself, not from depth alone. A variance–bias decomposition of the teacher-branch gradient then makes the trade-off explicit: contraction shrinks supervision variance while teacher–student path divergence grows bias, which places the optimum at a finite continuation rather than at one token or at infinity. AC-OPD acts on that. It selects K = 8 anchor positions per rollout by a squared log-ratio score — the positions where teacher and student log-probabilities disagree most — has the teacher continue up to mmax = 16 tokens from each, and picks every continuation's effective horizon by minimising the theory-guided criterion, using teacher Gini impurity as the variance proxy and cumulative teacher–student KL as the bias proxy. The branch loss is an overlay on the ordinary OPD trunk loss at λ = 0.5.

On 16 NVIDIA H800s, averaged over five independent runs and measured against SFT, OPD, ExOPD (reward extrapolation) and EOPD (entropy-adjusted token divergence), AC-OPD improves on vanilla OPD by 4.37 points of Math Avg. and 3.50 of Code Avg. — 38.03 and 61.20 for Qwen3-4B → Qwen3-1.7B-Base, 50.38 and 76.65 for Qwen3-32B → Qwen3-4B-Base, on AMC23/AIME24/AIME25/Minerva (pass@16) and HumanEval+/MBPP+ (pass@1). The margins over the stronger OPD variants are far thinner than the headline: in the 32B setting the Code Avg. gap to ExOPD is 76.65 against 75.95, and on AIME25 ExOPD actually wins, 26.8 to AC-OPD's 26.7; in the 4B setting MBPP+ is effectively a tie with SFT (57.6 against 57.5). The controls are the most informative part. A shuffled-adaptive policy holding the same mean continuation length of 5.74 tokens scores 60.10 against adaptive's 61.20, so roughly a point comes from where the budget goes rather than how big it is — but a plain Fixed-8 horizon already reaches 60.85 while spending more tokens, leaving adaptivity a 0.35-point edge over the simplest fixed alternative. Anchor choice matters more: squared log-ratio's 61.20 beats evenly spaced (58.97) and random (58.57) positions. The authors concede that evaluation is confined to the Qwen3 family and that the method requires additional teacher-side inference. A CAS/UCAS–Ant Group collaboration (lead author Jingang Zhou wrote it as an Ant Group intern; corresponding author Xu-Yao Zhang); no code is released yet — the paper promises it "upon publication".

Paper

Authors: Jingang Zhou · Yuyi Zhou · Haiyang Guo · Xukai Wang · Shuai Feng · Sirui Gao · Jian Xu · Qingpei Guo · Xu-Yao Zhang
distillationpost-trainingreasoningcodingtheoryresearch

Related