A task vector is just the delta a post-training run writes into the weights, τ = θt − θ0, and merging those deltas is the cheap way to combine capabilities. All the evidence for that trick, though, comes from vectors produced by supervised fine-tuning or RL. On-policy distillation writes a different kind of delta — the student learns from teacher feedback on its own trajectories, so its update need not resemble the RL teacher's — and nobody had checked whether OPD vectors compose at all. This is a study, not a method: no new training algorithm is proposed, and its result is a pair of claims with the failures left in.

Two backbones: SmolLM3-3B off a MixSFT anchor with three domains (maths, code, instruction following), and Qwen3-4B-Instruct-2507 with five, adding science and agentic tool use. RL experts are reused from the Open-MOPD and LLM-Fusion releases; within each family and domain, RL and OPD train on identical prompts, so the comparison isolates the method. Four merging rules — Task Arithmetic, TIES, TSV-M and Raw Sum — across AIME 2024/2025, LiveCodeBench v5/v6, IFEval + IFBench, GPQA-Diamond and a BFCL v3 subset. The headline findings are real but conditional. Merging a domain's OPD vector with its own RL teacher's can beat both constituents even though the OPD student is the weaker model: on Qwen3-4B code, TIES reaches 41.24% against RL's 37.51% and OPD's 37.18%. And across tasks, OPD compositions average higher than the matching RL compositions in seven of eight backbone×rule pairs. In parameter space the two updates are substantially non-collinear — orthogonal energy share 93.0% for maths and 95.2% for code on SmolLM3-3B — and OPD's high-energy regions are less shared between tasks, with mean Jaccard overlap of the top-10% MLP channels dropping from 0.15368 to 0.11064, a 28% reduction, across all ten Qwen task pairs.

The negative half is what makes the title's claim about interdependence rather than additivity. Within-task merging is not reliable: on Qwen3-4B it wins decisively on code (TIES +3.73, Raw Sum +3.50 against the RL teacher) but on maths three of four rules fall below RL, TSV-M by 3.63 points (56.51 against 60.14) — under both constituents — and Task Arithmetic loses on code (−0.03) and IF (−0.61). The paper's own summary: "the benefit of composition depends on both the task and the merging rule." The one cross-task loss is Raw Sum on SmolLM3-3B, RL 32.45% to OPD's 32.06%, and under Raw Sum the Qwen RL composition still wins on code and IF outright. More awkwardly, merging costs you against simply keeping the specialists: on Qwen3-4B the per-domain expert average is 57.23% for RL and 56.16% for OPD, and no merged model reaches the RL figure — the best is OPD under TIES at 56.19%, a 0.03-point edge over the OPD specialists. Science is where it breaks, GPQA-Diamond falling from 44.95% for the RL specialist to 35.86% under Raw Sum.

On the three-domain SmolLM3-3B setup merging does clear the specialist average (OPD under TIES at 32.71% against 30.34%), so the direction of the effect flips with the number of domains. Direction retention improves for code, science and agent but declines for maths and IF. Joint composition — OPD and RL vectors together — posts the best averages (33.28% on SmolLM3-3B, 56.67% on Qwen3-4B) and beats published multi-teacher distillation baselines on SmolLM3 (Naive M-OPD 28.05%, Open-MOPD 31.24%), but on Qwen its per-domain wins are confined to science and agent while maths, code and IF each sit slightly behind a single-method composition. The authors hedge throughout and should be read as meaning it: the orthogonal residual "do[es] not establish whether the residual improves task performance"; the norm-matched code control, where Raw Sum exceeds norm-matched RL and OPD by 1.77 and 2.07 points, holds only "at the tested global update norm"; the diagnostics "do not establish a general relationship between alignment and performance retention"; and the mean improvements are "descriptive point estimates" that "do not, by themselves, establish statistical significance." The usable takeaway is narrow and genuinely new — a weaker standalone model is not a weaker composer, so composability deserves measuring separately from benchmark score — and the stated limitation is that only general-purpose merging rules were tried, leaving OPD-aware rules to future work. A CAS/UCAS–Ant Group collaboration sharing its lead author and corresponding author with AC-OPD; no code released.

Paper

Authors: Jingang Zhou · Feiyu Han · Han Zhu · Yuyi Zhou · Ruiyang Zhang · Jian Xu · Sirui Gao · Qingpei Guo · Xu-Yao Zhang
distillationpost-trainingmodel-mergingreinforcement-learningreasoningresearch

Related