Finding the Right Fit: Model–Harness Interactions across Agent Tasks
paperYour notes
Choosing an agent system means choosing two things, a model and the harness it acts through, and almost everything published treats one of them as fixed. This paper varies both. It evaluates 66 configurations — four configurable harnesses (OpenHands, DeepSeek Harness, PI and openJiuwen) across five models on TUA-Bench, ALE-CLI and Terminal-Bench 4, plus the native Codex–GPT and Claude Code–Claude pairings — and asks whether a strong model, a strong harness, or a strong pairing stays strong when the setting changes.
It does not. Model rankings reverse across harnesses: on Terminal-Bench 4, Claude leads GPT by 7.94 points under OpenHands and trails it by 30.16 under PI — a 38-point swing produced by changing nothing but the scaffold. For four of the five models the best harness changes from one benchmark to another, so there is no harness to recommend unconditionally and no model ranking that survives a change of execution layer. Some pairings do hold across settings, which is the constructive half of the result: the unit that behaves consistently is the pair, not either part.
The practical consequence is a measurement one. Any leaderboard that reports a model's agentic score has silently fixed a harness, and a benchmark that compares models under one scaffold is measuring the pairing rather than the model. Bo An's group at Nanyang Technological University; no code released at filing, though the paper reports releasing 6,204 scored trajectories.
This result is contested, and the disagreement is the more useful thing to know. Three papers from the same fortnight reach weaker or opposite conclusions about how much the harness moves the score with the model held fixed. What Does a Harness Buy? Tokens, Mostly (arXiv:2610.04433, 3 Oct) runs five models through three production harnesses and reports that swapping the harness flips no more tasks than simply rerunning the same one — 13% either way — with the only effect clearing noise being cost, up to 3×. Beyond the Model (arXiv:2609.32459) finds component value depends jointly on model capability and task type, with context compression and general subagents sometimes actively hurting. And How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering? (arXiv:2609.40303, EPFL and Apple) reports that elaborate MLE harnesses buy nothing over a single session of a minimal read/write/bash agent at equal time budget. All four are in dialogue with Harness-Bench. Treat the 38-point swing as one lab's measurement on one benchmark set, not a settled property of harnesses.
Three October 2026 papers extend the argument. OpenBMB's JustGoal (arXiv:2610.36434, code) is the sharpest version of the minimal-harness result, in GPU kernels. A general-purpose agent given only the optimization goal and constraints, building its own evaluation loop, beats open-source libraries on 8 of 10 operators on B200s (1.14–2.34× geometric-mean speedups). In a same-model comparison on Claude Fable 5.1, a specialized kernel harness reaches only 0.95× of it, and ablations find no consistent benefit from domain knowledge or predefined workflows. The same group's ForgeKernel pulls the other way: a self-evolving kernel agent that promotes each accepted kernel to the next reference and matches or beats 18 of 22 expert-library kernels (FlashAttention-4, cuDNN, DeepGEMM) across 27 operators. RUC's Harness Scaling (arXiv:2610.29458, code) treats harness diversity as a training axis rather than a deployment choice: RL across a factorized space of white-box harness variations generalizes to unseen harness combinations and to external black-box harnesses better than training through one complex black-box harness.