Rethinking Self-Distillation for Multi-Teacher Capability Merging
paperYour notes
The controlled study the on-policy distillation wave has been missing, and it comes out against the wave. Merging several experts trained from a shared base is now routine in frontier post-training, and multi-teacher on-policy distillation has become the assumed-best way to do it. Apple and Duke argue that most of the reported advantage is an artifact of training design choices and hyperparameter tuning rather than the algorithm. Across two multi-teacher settings — MIA (maths, instruction-following and agentic experts, deliberately disjoint) and Agent×3 (ALFWorld, WebShop and InterCode-SQL, deliberately overlapping) — four models (Qwen3-8B, Qwen3-1.7B, Qwen3.5-4B-Base, Ministral-3-3B-Instruct) and eleven benchmarks, they tune SFT, soft-label KD, offline MOPD and MOPD independently, up to 160 configurations before seed replication. All four land at nearly identical accuracy — on Qwen3-8B/MIA the in-domain recovery rates are SFT 98.2, Soft-KD 97.3, Off-MOPD 96.9 and MOPD 97.9, inside each other's error bars — while MOPD costs 14.8–23.1× SFT's training GPU-hours.
The mechanism is the part worth keeping, because it generalizes beyond this debate: learning rate alone moves SFT's validation score by 8.5 points on MIA and 11.3 on Agent×3, and the tuned off-policy rates come out about five times higher than the on-policy ones, so a shared learning rate silently handicaps the off-policy baseline. Data filtering cuts the same way — rejection-sampling teacher trajectories is worth 6.5–7.0 points to SFT against 0.5–1.6 for the MOPD variants, so the cheaper method is the one that benefits most from the care nobody spent on it. Then they reproduce four published SFT-versus-OPD comparisons — Open-MOPD, SKD, TCOD and ReOPD — and report that OPD's margin over a properly tuned, rejection-sampled SFT baseline is smaller than published in every one. On ReOPD the published gap is 45.0 to 51.0; their rejection-sampled SFT reaches 54.6 against OPD's 54.7. On TCOD their SFT reproduction alone scores 65.7 where the paper reported 28.8.
They also check the training-free option: simple weight merging recovers expert capability in minutes of CPU time, scoring 98.9 in-domain recovery on Qwen3-8B/MIA — better than any of the four trained methods there — though it degrades as models shrink and as task interference rises. The honest asymmetry the entry should not bury is that SFT is not uniformly equal: on out-of-domain retention it falls 3.0 points from base where MOPD falls 0.8, so the cheap baseline matches on what it was trained for and forgets more elsewhere. The paper's conclusion is a methods demand rather than a new technique — tune the efficient baseline before attributing a gap to the objective. Ten authors across Apple and Duke University, every run on eight B200 GPUs; code is announced at apple-aiml-research/ml-rethink-mopd but the repository is not yet public, the paper noting it is under legal review.
A Salesforce AI Research study, "Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO" (arXiv 2611.08453, 5 November 2026), asks the same kind of question in training FLOPs rather than tuning budget. It mostly agrees on cost, with one exception. With an external post-trained teacher, vanilla reverse-KL OPD generally loses to GRPO in accuracy per training FLOP, even when the teacher has had GRPO itself, and responses often grow longer. With the student's own GRPO checkpoint as teacher, OPD recovers that checkpoint's maths accuracy with 50× fewer distillation FLOPs. That only matters when several domains are being merged. In that setting, multi-teacher OPD over per-domain GRPO experts costs 5.2–8.0% of the specialist training FLOPs, and the whole pipeline matches joint GRPO on a two-domain mix with 42.0% fewer total FLOPs. Read with Apple's result, the case for OPD shifts from per-token supervision being better to OPD being a cheap way to merge experts that RL has already trained.