ReOPD: Multi-Turn On-Policy Distillation with Prefix Replay

Makes on-policy distillation practical for multi-turn agentic tasks, where fully online OPD needs fresh student rollouts through the environment plus teacher queries at every visited history. ReOPD (Replayed-Prefix OPD) instead reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps while the teacher gives dense per-step supervision — zero environment/tool calls during student training.

Identifies the "prefix trap": pushing histories more student-on-policy improves relevance but queries the teacher where its targets are unreliable — a two-sided distribution shift between student occupancy and teacher reliability. ReOPD treats prefix selection as reliability-aware distribution design (a step-decaying schedule favoring early, lower-shift prefixes). On math-with-Python and search environments it matches or beats full OPD at ≥4× faster rollouts, turning expensive agent–environment interaction into a reusable offline resource. By Liao, Dong, Monz, Xu, Li Dong, and Furu Wei (MSR GenAI with U. Amsterdam).