On-policy distillation gives a student dense teacher supervision on its own trajectories, but in multi-turn agents a wrong action changes every state that follows, so errors compound. This paper from NVIDIA's Learning and Perception Research group starts from a measurement: using ALFWorld's symbolic oracle, 59% of failed rollouts across three Qwen3 models (8B–235B) contain a pivotal mistake — an action that lengthens the remaining optimal trajectory or makes the task unsolvable. The first one lands at a median of turn 8 to 12 out of 30, and the agent wastes the rest of the episode. It is also recoverable: correcting the pivotal turn lifts replayed success from 8% to 59%, and guiding only the two turns after it still reaches 58%. Standard OPD does not capture this. It cuts the overall failure rate from 79% to 56%, but failures after a pivotal turn barely move (51% to 49%), because the recovery action stays below 1% probability — a group of eight rollouts never samples it, so there is no learning signal.

PivotOPD therefore trains for both halves of the problem. A teacher reads each rollout in hindsight and names a gold action at candidate turns; a turn is pivotal when the student's committed action disagrees. Preventive distillation re-scores the student's recorded response against the gold action under reverse KL, steering it away from the mistake; recovery distillation uses forward KL, whose mass-covering behaviour puts probability on recovery actions the student almost never produces. Both run through a privileged self-teacher — the frozen student conditioned on a hint naming an action — so the teacher only ever names actions while the token-level targets stay in the student's own reasoning style, and both terms enter a single PPO update as per-token distillation advantages.

Against 13 baselines spanning RL, self-distillation, turn-level distillation and skill or pivotal-turn guidance, it ranks first on all eight per-benchmark averages over three seeds for both Qwen3-1.7B and Qwen3-8B students: +5.5% on ALFWorld and +5.9% on Search-based QA over the strongest baseline with the 1.7B student, with the largest gains where successful rollouts are scarcest and outcome-only RL gets no signal. On WebShop it beats RLSD by only +1.2% in score but +14.1% in success rate — partial progress converted into finished tasks. Teacher capacity is not the explanation: with Qwen3-8B as its own teacher it still wins all three benchmarks, by at least +1.5% each and +3.9% on average. Replayed from the same 72 oracle-labeled pivotal mistakes it recovers in 72.7%, against 8.3% for the base model, 20.3% for standard OPD and 45.8% for the preventive-only variant, in the fewest turns, improving 60 of the 72 and worsening none. The gains cross model families: on a curated bug-fix curriculum with Nemotron-3-Super teaching a Nemotron-3.5-SFT student, SWE-Bench Verified resolve rate goes 62.8% to 66.0% — about a third of the gap to the teacher — where standard OPD from the same teacher adds 0.2 points. Code is announced as coming soon.

Paper

Authors: Yinghui He · Yapei Chang · Khushi Bhardwaj · Daniele Molinari · Tugrul Konuk · Jan Kautz · Ali Hatamizadeh
agentsreinforcement-learningpost-trainingdistillationresearch

Related