Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
paperYour notes
A negative, diagnostic paper about the quantity the whole on-policy self-distillation literature rests on. OPSD builds a dense token signal by conditioning a teacher on privileged information available only in training — a reference solution, environment feedback, a judge's critique — and scoring each token by how much that context raises its log-probability (one-sided), or by contrasting a helpful against an unhelpful context (two-sided). Those scores are then used as if they estimated the token-level action value. The paper's claim is that they answer a different question: how the model's prediction shifts when its input is enriched, not how the expected outcome shifts when a token is changed. The log-ratio form is what makes the substitution look safe, because in one standard case — hindsight credit assignment with the true success and failure conditionals — Bayes' rule makes exactly such a ratio rank binary action value. That guarantee needs the two contexts to be coherent conditionals of one joint distribution; two free-form texts labelled "helpful" and "unhelpful" need not be, and can also differ in wording, length and which values they reveal. The paper further shows (Proposition 1) that ranking is not enough — to price every local redistribution of next-token probability on the same scale the score must be a positive affine transform of the action value, so a rank-preserving but magnitude-distorting score over-concentrates mass on its top token.
Hence three checks, which the paper insists are separable — a score can fail the first while the second is moot, or pass the first and have the loss destroy it. Q1: does the score track task success? The accessible test is at the sequence level: average the token scores over a trajectory and ask whether that scalar separates correct from incorrect rollouts (AUC). It is necessary, not sufficient — but a score that cannot order trajectories cannot be ordering tokens. Q2: was the feedback written from the rollout it scores? A critique generated after reading a rollout makes the score depend on the rollout twice over, so it can be high simply because text and description are about the same thing. Cross-fitting (score each rollout with feedback written from a donor rollout of the same problem) removes that loop. Q3: what does the training loss do with the score? Baseline subtraction, clipping and entropy weighting determine which scores actually become parameter updates — policy-weighted centring can flip a positive two-sided score into a negative token advantage, and Proposition 4 shows that preserving an advantage's sign and total across tokens does not preserve the expected policy gradient, since the per-token score-function vectors point in different directions in parameter space.
On the main system — gpt-oss-20b at low effort, trained on DeepScaleR and evaluated on AIME 2025, with an external judge writing a helpful and a plausible-but-wrong critique — all three checks fail. Q1: the one-sided teacher–policy residual is negative for both correct and incorrect rollouts (−0.0334 and −0.0502), so the teacher down-scores the student's own sampled tokens regardless of correctness, though the +0.017 gap shows some relative information survives; the two-sided score actually used in training reaches a pooled AUC of 0.505, i.e. chance, and 0.474 after length adjustment, which slightly favours incorrect trajectories. The information was available — an external judge given the same inputs picked the better solution at about 94% accuracy — the token-likelihood score just does not extract it. Q2: same-rollout feedback scores 0.588 pooled, but each rollout helped write its own critique; cross-fitted scores give 0.512 and 0.416, so removing self-dependence is mechanically sound (Proposition 2) yet changes what the score measures rather than debiasing it, and neither held-out construction is stably related to correctness. Q3: all five token-score training configurations — three additive variants with entropy gating and source clipping, two KL-regularized — reach only 24.2–33.9% Avg@4 against 64.2% for outcome-only GRPO, drops of 30.4 to 40.0 points with every 95% paired confidence interval entirely below zero; and in an ungated analysis 57–71% of the absolute token-advantage mass lands in the highest-entropy token decile, precisely the regime where the score is least informative. It is not a formatting artefact: the dense variants return boxed, complete answers more often than the baseline they lose to.
The paper is careful not to generalize into "dense distillation does not work," and carries a positive control to prove it. Qwen3-8B trained with one-sided SDPO on SciKnowEval Biology, where the only privileged information is "The correct answer is [gold answer]", improves held-out Avg@8 from 31.50 to 58.25 (thinking off) and 26.25 to 54.25 (thinking on), with raw trajectory AUCs of 0.924 and 0.813. The conclusion is a usage rule rather than a method: strong outcome separation accompanies the runs that work, so trajectory AUC is a cheap Q1 screen, and a privileged-likelihood score should be validated as a proxy for outcome-relevant credit before it is trusted as token credit. The related work is unusually even-handed about the surrounding evidence, citing concurrent failure reports (teachers that track adherence to a reference trajectory rather than correctness; a corrupted solution from a different problem retaining much of the OPSD gain) alongside successes, including the same group's own VPD and PMD, both of which it classifies among the constructions that do work. A companion caveat to the token-selection results in IER-OPD and extremely sparse supervision, and to RISE, which drops privileged conditioning altogether. First and corresponding author Xuan-Phi Nguyen; no code was linked at filing.