VPD: Learning from Language Feedback via Variational Policy Distillation
paperYour notes
RLVR's learning signal is one bit per trajectory, so — in the paper's framing — "a minor arithmetic mistake in a complex derivation receives the same zero-reward as a completely nonsensical hallucination," and on problems the model almost never solves there is no positive signal at all no matter how many rollouts are drawn. On-policy self-distillation fixes the density by conditioning the model on language feedback — a compiler trace, a successful sibling rollout, a critique — and distilling that feedback-informed next-token distribution back into the unconditioned policy. The flaw this paper identifies is that the self-teacher in methods like SDPO is a fixed, passive function: it is never trained to read the feedback, so the whole signal rests on zero-shot interpretation, and as the student internalizes the easy corrections the teacher's ability to discriminate subtler errors plateaus and the gradients dry up. They measure it rather than assert it — the reward margin between correct and incorrect responses shrinks over LiveCodeBench training under SDPO while it keeps widening under VPD.
Variational Policy Distillation (VPD) recasts the feedback-conditioned model not as a heuristic but as an approximate posterior over the reward-tilted optimal policy, which turns training into variational EM. The E-step actively trains the teacher to interpret feedback. It has to be done off-policy on the student's own rollouts (making the teacher search for its own successes would reintroduce the sparsity being escaped), and because each critique is written for one specific trajectory no two rollouts share a context, so DPO-style preference pairs cannot be formed at all; VPD uses Binary Classifier Optimization, an unpaired objective, instead. The reference prior is re-anchored to the current student every round, a sliding trust region that keeps the teacher's target reachable. The M-step is ordinary token-level KL distillation on the student's own rollouts. Teacher and student are one shared-weight network distinguished only by whether the feedback is appended to the prompt, so there is no second model in VRAM and no extra sampling or verification — the E-step's gradient work costs a 30–55% runtime increase over SDPO, absorbed by running one E-step per five M-steps.
On LiveCodeBench v6 with Qwen3-8B (thinking off) VPD reaches 49.62% against 47.33% for SDPO, 45.61% for GRPO and 28.05% for the base model; the single-phase hybrids that fuse RL and distillation into one update are the interesting comparison, because two of the three are worse than pure SDPO (advantage reshaping 46.95, reweighting 44.85, joint loss 47.52). On SciKnowEval with contrastive sibling rollouts as feedback, the four-domain average is 74.34 for Qwen3-1.7B (GRPO 69.81), 77.15 for Qwen3-8B (SDPO 74.44) and 70.80 for OLMo3-7B-Instruct; autonomous self-critique works about as well (78.14 vs SDPO's 74.87 on Qwen3-8B). Both design choices earn their place in ablation: one E-step per five beats one per one (70.21) and one per ten (69.27), and a fixed rather than student-anchored prior falls to 67.84. The paper's own stress tests are where it concedes: applied to a base model, self-distillation collapses, and VPD only delays the collapse before still losing to GRPO; on DAPO-Math training, SDPO collapses outright and pure GRPO remains dominant (83.8% on Math500 with Qwen3-8B), which the authors attribute to mathematics being unforgiving enough that distilling noisy intermediate guidance constrains the exploration that sparse RL rewards. Shared weights also cap the teacher's representational capacity at the student's. A companion to the group's PMD, token-credit audit and RISE, and to the stage-ordering results in Sequential Beats Joint. No code was linked at filing.