On-policy distillation is capped by the quality of its supervision source, so the obvious move is to hand the teacher — or the student itself — privileged information it will not have at deployment: verified reasoning hints for an LLM, bounding boxes for a VLM. This paper's contribution is to name why that backfires. It calls the failure mode privilege illusion: once a policy is conditioned on privileged input, the apparent teacher–student gap conflates two fundamentally different things — the transferable capability gap that distillation is meant to close, and an information asymmetry gap that the student can only mimic but never replicate. Distilling such a teacher indiscriminately makes the student fit privileged outcomes rather than acquire transferable ability. The measurements back it up: granting privileged input to the teacher only, or to the student only, buys modest early gains and then late-stage performance degradation coupled with entropy collapse; granting it to both policies makes the superficial advantage vanish, and uniform distillation over that pair leaves the student passively adapting to the hint, ending up with smaller gains than Vanilla OPD. Token non-uniformity amplifies it, since only a small subset of tokens carries capability-bearing signal while the rest may be privilege-dependent. The paper's proxy for separating the two is the privilege advantage gap — the absolute log-probability difference between privileged teacher and privileged student on the same token. Pruning the top-20% highest-gap tokens from distillation leaves only about half of Vanilla OPD's gain at step 100 (about a fifth in the VLM setting), while pruning random or lowest-gap tokens barely registers — though the high-gap-pruned variant still gains 3.4 and 1.5 points, so the remaining tokens are not dead weight.

DOPD therefore routes each token to a different supervision source, objective and granularity. Both policies see the privileged input (the privileged student shares parameters with the deployed student and sees the hint only in training; the privileged teacher is frozen), outliers in the top 5% are discarded and the statistics normalized within the batch, and four regimes fall out. Low gap with both probabilities high — consensus knowledge inside the hint — gets light Top-K reverse KL toward the teacher. Low gap with both low — connectives and unreliable segments outside either model's competence — gets weak Top-K reverse KL toward a stop-gradient copy of the privileged student, used as a consistency anchor rather than a knowledge source. High gap with the teacher more confident is the capability-bearing case and gets full-vocabulary JS divergence at unit weight. High gap with the student more confident is read as self-consistent exploration and gets light distillation toward the privileged student, so the teacher does not suppress it. Experiments run on non-thinking Qwen3 and Qwen3-VL, mainly Qwen3-8B → Qwen3-1.7B and Qwen3-VL-8B → Qwen3-VL-2B, on 8 NVIDIA H200s, with βw=0.3, βl=0.6 and K=128. The privileged inputs are generated and then re-filtered by GPT-5.4, yielding 32K LLM and 25K VLM training samples.

Averaged over eight LLM benchmarks DOPD reaches 51.4 against the 1.7B student's 39.1 and the 8B teacher's 52.8 — +12.3 points, 89.8% of the teacher–student gap — ranking first on all eight, beating Vanilla OPD by 7.5 and the strongest standard and adaptive baselines ExOPD/Uni-OPD/EOPD by 4.4/4.8/5.3, and passing the teacher outright on four (AIME25 23.3 vs 20.2, ZebraLogic 26.9 vs 25.0, BFCLv3 60.2 vs 60.0, LCBv5 27.1 vs 23.6). On eight VLM benchmarks it reaches 58.4 against 48.3 and 62.9, recovering 69.2% of the gap, but the win is narrower: +2.1 over VA-OPD, and it is second on MMMU-Pro (53.9 vs 54.2); every method here, DOPD included, gains more on visual understanding than on visual reasoning. Across five teacher–student pairs the gain over the student runs +11.1 to +14.1, two to three times Vanilla OPD's, and it holds up where naive imitation does not: at the worst mismatch, Qwen3-8B → Qwen3-0.6B, Vanilla OPD manages +3.5 while DOPD manages +14.1. It also beats the second-best baseline by 3.1 and 4.3 points on the two out-of-distribution transfers, and passes the baselines' step-200 scores by step 80 while the self-distillation baseline hits entropy collapse around step 95. The concessions are the useful part.

The whole effect rests on privileged input existing: ablate it and the model drops to 63.6 C-Eval / 38.3 LiveBench, below Vanilla OPD's 65.2/40.9. Handing over the ground-truth final answer as the privileged input is worse than no privileged input at all (59.5/36.7 vs 63.0/39.4) — shortcut learning — and step-wise hints that include the execution trace are no better than none either; only step-wise hints without the execution trace pay off (71.3/49.8). Under equal weighting, adding the other three token types to the high-teacher-advantage ones actively hurts (41.3 vs 45.3 on LiveBench at step 160); the adaptive routing is what converts them into a gain (49.8). The paper also reports JS divergence beating forward and reverse KL under its settings, against some prior work. Listed limitations: dependence on the availability, cost and quality of privileged information, one extra student forward pass per step over Vanilla OPD, and a routing rule that is still heuristic. The privileged parameter-shared student is the same device NVIDIA's PivotOPD uses as a “privileged self-teacher”, here generalized from a single gold action to every token in the trajectory. Affiliation 1 is NUS, with MMLab at CUHK, Peking University and JD's Explore Academy; senior author Shuicheng Yan. No code released.

Paper

Authors: Xinlei Yu · Gen Li · Qingyi Si · Guibin Zhang · Yuqi Xu · Congcong Wang · Shuai Dong · Kaiwen Tuo · Xiangyu Zeng · Kaituo Feng · Qunzhong Wang · Yang Shi · Xiaobin Hu · Xiangyu Yue · Jiaqi Wang · Shuicheng Yan
distillationpost-trainingmultimodalresearch

Related