RLVR and its self-distillation variants are episode-local: a rollout is scored, the policy is updated, and the rollout is discarded. That is the right design when episodes are independent, and they are not — across epochs a model keeps meeting the same and related problems under a policy that is itself changing, which produces cross-episode signals that a one-step update cannot represent: which strategies consistently pass verification, which failure modes persist, which patterns recur. SDPO already showed the training-time signal can be richer than a scalar reward by conditioning a self-teacher on feedback or a successful sibling rollout, but its updates are still episode-local and nothing the model discovered in an earlier attempt is preserved.

Procedural Memory Distillation (PMD) accumulates that history online from the model's own rollouts and distils it into the weights during training. The memory has three levels of abstraction — experience (raw trajectories, rewards, feedback per problem), insight (per-problem strategies and lessons from self-reflection, with anti-shortcut filtering) and behavior (reasoning patterns distilled across semantically related problems) — and the paper is explicit that this is a fidelity/transfer trade-off, with experience faithful but local, behavior broadly transferable but coarse, and insight the middle ground. A memory-conditioned self-teacher then supervises the student on its own rollouts through the same SDPO-style reverse-KL objective. The memory is a training scaffold only: it is never available at test time, so the resulting model is memory-free at inference with no retrieval stack. The stated design principle is co-evolution — the policy writes the memory, the memory shapes the supervision that updates the policy.

With Qwen3-8B and OLMo3-Instruct-7B (thinking off, 8 rollouts per prompt), PMD lifts the SciKnowEval four-domain average (avg@16) over SDPO from 74.4 to 77.2 and 69.5 to 73.3, and LiveCodeBench v6 (score@4) from 47.9 to 51.7 and 45.0 to 51.1 — the 3.8–5.5% and 7.9–13.6% relative gains quoted in the abstract. The decomposition is more informative than the headline. Discarding memory after every step still beats SDPO, but only by 1.3pp on SciKnowEval and 0.2pp on LiveCodeBench; letting it persist adds 1.5pp and 3.6pp, and in code generation — where successful patterns are rare and must be consolidated over many attempts — persistence accounts for nearly all of the improvement.

The two controls that isolate co-evolution both fail badly: evolving the memory while freezing the policy barely moves at all (54.0 / 35.9), because nothing is distilled and the model only sees a richer prompt it never internalizes, and training against a memory bank built once at initialization reaches 65.0 / 47.5 but goes stale as the policy moves. A separate result is arguably the more useful one: PMD preserves answer-space coverage that SDPO collapses — the maj@k-to-best@k verifier headroom band is 2–4× wider, PMD's solved set covers 9–14% more problems with only 2–4% solved exclusively by SDPO, and on Materials SDPO's headroom goes to zero outright, a signature of mode collapse that would blunt any downstream best-of-N or verifier reranking. Cost is 1.45× step time with peak GPU memory essentially unchanged (+0.3 GB), since the bank lives off-GPU and extraction runs asynchronously. The authors limit their claim deliberately: the evaluation is repeated training over fixed task distributions in two verifiable domains, narrower than long-horizon agentic settings where memory, tools and skills must evolve across heterogeneous tasks, and they explicitly do not claim to have solved online agent self-improvement. Same research line as VPD, the token-credit audit and RISE. No code was linked at filing.

Paper

Authors: Ye Liu · Srijan Bansal · Bo Pang · Yang Li · Zeyu Leo Liu · Yifei Ming · Zixuan Ke · Shafiq Joty · Semih Yavuz
distillationpost-trainingmemoryself-improvementreasoningcodingresearch

Related