On-policy distillation gets its density from the teacher's full next-token distribution, which is exactly what you cannot afford to move across a 30K-token reasoning trace. Production systems therefore expose a sparse teacher interface — the sampled token's score, or a small Top-k head — and that forces a choice this paper frames as a dilemma. Optimise a Top-k objective and you are solving a biased surrogate (the paper shows the truncated, renormalised and stopped-coefficient variants are all biased). Keep the full-vocabulary reverse-KL objective and estimate its gradient from the sparse payload and you stay unbiased but noisy: sampled-token OPD reads one token's score and pays single-sample Monte Carlo variance, while HETS (Head-Exact Tail-Sampled, the hybrid from Zhang and Ba's EMA-PG) integrates the head exactly but treats the tail correction as entirely unknown despite knowing its aggregate mass. Every tail hit must then compensate for the whole omitted mass. Even at 98.4% head coverage a 30K-token trace contains 460–470 such spikes, thousands per optimizer step — which is why both sampled-token OPD and HETS peak early and degrade later in training.

ResOPD attacks the estimator rather than the objective. The authors first show that any estimator that evaluates the head exactly, adds a prefix-dependent control vector weighted by the student's tail mass, and corrects on tail hits is unbiased for the full-vocabulary reverse-KL gradient — HETS being the special case of a zero control. A covariance-trace decomposition then splits the variance into a control-invariant within-tail sampling term and a between-event term proportional to the control's distance from the true tail mean, which names HETS's spikes precisely: a zero control pays for the full tail mean on every hit. The fix is that the residual masses outside the head already define a coarse pseudo-token whose analytic gradient is a computable predictor of that mean. So ResOPD integrates the coarse (k+1)-way distribution exactly on every draw and samples only the unresolved fine-tail residual on tail hits — same payload, no extra teacher queries or forward passes. The release is a core implementation on top of verl at a pinned revision.

Two kinds of evidence. In exact frozen-prefix audits that enumerate every next-token action at student-visited prefixes, ResOPD lowers the gradient covariance trace by 52.5–75.0% relative to sampled-token OPD under Top-4 feedback, below both HETS and a payload-matched vOPD baseline. End to end, distilling Qwen3.5-27B into Qwen3.5-4B on DeepMath gives 50.12% average against 45.72% for HETS (+4.40) over DeepMath held-out, AIME26 (30 problems), HMMT26 (33) and the English split of OlymMATH (200), with avg@16 on the two small competition sets and avg@2 elsewhere; a science-and-coding transfer from Qwen3-30B-A3B-Instruct to Qwen3-4B reaches 44.15% against 43.63%. Robustness is the more convincing result: pushed to a 5×10-6 learning rate, HETS collapses from 45.94% to 42.54% while ResOPD holds 46.35%, widening its lead to +3.81 points.

The paper is unusually careful about what this does not buy. Variance reduction "does not imply monotonic downstream accuracy gains" — ResOPD can suppress gradient noise without beating HETS at every checkpoint. The advantage over HETS narrows as k grows, since a larger head absorbs more mass and tail corrections get rarer. The apparent super-teacher code scores are retracted in the text: paired bootstrap 95% CIs cross zero (ResOPD−Teacher spans [−2.48, +5.33] on LiveCodeBench v6), so on-par rather than significant — and plain sampled-token OPD is the strongest method on the science tasks. Most pointedly, the concurrent SC-TA estimator has strictly lower covariance trace than ResOPD across all three audited prompt distributions (DeepMath, MegaScience, SynLogic), with only 18.0% of SynLogic prefixes favouring ResOPD, yet ResOPD takes the higher terminal score — the authors read this decoupling as informative rather than contradictory. Unbiasedness also assumes ancestral full-support on-policy sampling: truncated sampling, stale rollouts, repeated optimisation passes and sampling-altering logits processors need separate treatment. The released port excludes Megatron students, SGLang teachers, fused loss kernels and chunked Top-k, and its validation covers 22 standalone and 13 integration checks with no full GPU training run.

It sits next to MBZUAI's IER-OPD on the complementary axis of the same problem: IER-OPD asks which tokens carry a gradient worth estimating at all, ResOPD asks how much of the gradient you can recover exactly from whatever payload the teacher already sent. Shanghai AI Laboratory with SJTU, USTC and Fudan; authors include Kai Chen and Yuhang Zang, who also appear on the lab's E²-OPSD cluster of October distillation work. ~23 GitHub stars at filing; Apache-2.0.

Paper

Authors: Penghui Yang · Long Xing · Xuanlang Dai · Ziyu Liu · Kai Chen · Yuhang Zang
opddistillationpost-trainingrlefficiencyresearch

Related