The sequel to OPD (Rethinking On-Policy Distillation), running head "One-Shot On-Policy Distillation," and it attacks the half the first paper left alone. OPD now sits in the post-training stacks of Qwen3, MiMo, GLM-5, DeepSeek-V4 and Kimi K3, and every attempt to explain why it works — objective geometry, update subspaces, when teachers are too strong — has been algorithmic, taking the training set as given. So the authors borrow one-shot RLVR's controlled extreme and run OPD on a single query. It keeps improving the student for hundreds of steps. Averaged over MATH-500, AMC 2023 and AIME 2025, one query reaches 68.5 against 69.8 for full-data OPD at step 300 — 87% of full-data OPD's gain and 69% of the teacher–student gap; through step 1000 the recovered fraction stays in a 62–89% band. It is not a maths artefact: the effect holds across three student–teacher families (R1-Distill-1.5B 77.1→85.5, Llama-3.2-3B-Instruct 28.2→40.2, OLMo-3-7B-Instruct-DPO 70.8→82.4) and recovers 73%, 66% and 64% of the gap on code generation (LiveCodeBench v6), instruction following (Multi-IF) and agentic tool use (BFCL v3). It also survives a query the student never solves, and tightening the response cap or lowering the rollout temperature.

The explanation is the contribution, and it has two halves that the paper compresses into "OPD is data-overfed but algorithm-starved." On the data side, what a query is worth is the part of the state space its rollouts reach, a state being a prefix at which the teacher supplies a token-level target; with 64 rollouts per update even one query yields tens of thousands of supervised states. The authors make this measurable as state coverage: represent each state by the teacher's final-layer hidden vector, pool the states full-data OPD visits over DAPO-Math-17K (DAPO), partition them into 200 k-means clusters, and report the fraction a run reaches. One query reaches 71.5% by step 300, 65.9% of it inside the first 100 steps; 16 semantically distinct queries, one per BGE-M3 cluster, reach 98.9% and match full-data OPD, and a control holding the count at 16 while drawing them from a single cluster drops both coverage and accuracy. On the algorithm side what decays is the absorption rate, the share of the remaining teacher–student distance one update closes: it falls throughout the run, and falls the same way on 1, 4, 16 and all 17k queries, each removing 78–84% of its step-30 distance by step 300. The pace is a property of the optimiser, not of the dataset — and an always-off-policy run on 64 trajectories sampled once, with its states frozen, still takes about 200 steps to finish improving, so the length of a run is not explained by fresh states arriving.

The result carries to the multi-teacher setting that frontier pipelines actually use (MOPD): training one R1-Distill-1.5B student over maths, code and instruction following with a teacher per domain, full-data MOPD lifts average validation accuracy from 43.5 to 52.8 against 53.8 for three separate full-data OPD runs, and 16 diverse queries per domain reach 52.9 — 101% of full-data MOPD's gain (93% maths, 136% code, 109% instruction following) on the same rollout and optimisation budget. Pushed further, the query barely needs content: an empty user turn ending in <think>, the same template with a one-line domain system prompt, and off-domain WildChat chat logs (0.17% maths-related by the paper's own heuristic) all track the real-query baseline, with explicit domain content worth about a point — though a scaffold that closes the thinking block immediately collapses into untrainable meta-replies, so the requirement is not empty. A head-to-head against one-shot RLVR on the same query explains the asymmetry: GRPO's advantage vanishes once nearly every rollout solves the query, while OPD's per-state targets persist, and OPD's validation gain over 1000 steps is more than twice RLVR's whether matched by steps or by rollout tokens.

The paper's own limits: state coverage is a semantic proxy measured against a reference space built from full-data rollouts, weighting every cluster equally however often it is visited or how much teacher signal it still carries; what sets the absorption rate is unresolved; MOPD was run with three domains and one teacher each; every student here is 1.5B–7B, with larger students, long contexts and more teachers listed as future work; and OPD's ceiling is the teacher's distribution where RLVR's is not. The practical reading the authors want is that data curation for OPD stops being a question of how many problems to collect and becomes one of which states an input induces — and that the step efficiency of the algorithm, not the corpus, is where the headroom is. Code is Apache-2.0 under Thinking-Space/One-Shot-OPD; corresponding authors Ning Ding and Chaojun Xiao at Tsinghua, with Zhiyuan Liu, and collaborators at UCAS, Northeastern, UIUC and Johns Hopkins.

Paper

Authors: Zixuan Fu · Bingxiang He · Yuxin Zuo · Haohuan Huang · Jinqian Zhang · Ruhang Xiao · Cheng Qian · Qinyu Luo · Huan-ang Gao · Yudong Wang · Zhiyuan Liu · Ning Ding · Chaojun Xiao
distillationpost-trainingtraining-datareasoningresearch

Related