"Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads," from Microsoft (11 authors, including Chris Quirk and Xia Song). Multi-token prediction speeds up generation by letting a model draft several tokens per forward pass, with a verification step that keeps the backbone's output distribution intact. Every open MTP release so far — MiMo-7B, DeepSeek-V3, Qwen3, EAGLE-3 — trains those heads jointly with the backbone across the whole pretraining run, which fixes drafter quality at pretraining time and leaves no way to add MTP to a model after the fact. The paper asks whether a short post-training pass on the target model's own chain-of-thought can match that, and finds it can.

On a frozen Qwen3-8B with three chained heads, plain cross-entropy on about 2.5B tokens of on-policy target-generated text — three to four orders of magnitude less MTP-training data than MiMo-7B's joint pretraining — matches or beats the jointly trained baseline on mean accepted length, the expected tokens committed per verification step. Against MiMo-7B-RL at matched per-head capacity, running exact rejection-sampling verification at batch 4 on one A100-80GB in vLLM: GSM8K 3.05 against 2.80 (1.09×), MT-Bench 2.67 against 2.40 (1.11×), MBPP 2.84 against 2.70, MATH500 3.00 against 2.87, MMLU-Redux 2.64 against 2.54, and parity on AIME24 and AIME25 (2.81 against 2.85 and 2.82). The gap comes from the deepest head: on GSM8K the third head is accepted 51.1% of the time against the baseline's 33.6%, even though the baseline's first head is slightly better.

Two serving-time additions build on that checkpoint. A chain-aware relaxation of the verification rule admits a bounded drift from the backbone's distribution, governed by a threshold that trades accepted length against fidelity; it lifts expected speedup 12–16% per benchmark while holding task accuracy, and the sweep shows math accuracy behaving as a step function of accepted length while ROUGE varies smoothly. An adaptive controller then picks how many heads to engage at inference: an online lower-confidence-bound bandit over six arms (draft depth 1 to 6) credits the active arm with observed batched throughput after a short round-robin warm-up, recovering 11–14% of the speedup a fixed maximum draft length gives up when batch size and prediction difficulty vary. The paper is CC BY-NC-ND and ships no code.

Paper

Authors: Prachi Badarayani · Aidan Jay · Chenghui Zhou · Dayquan Julienne · Yuan Gao · Tianwei Chen · George Zerveas · Ishmam Zabir
inferenceefficiencypost-trainingresearch

Related