A benchmark for building a whole program from scratch. ProgramBench gives an agent only a compiled program and its documentation. The agent has to design and write a codebase, plus a build script, whose executable behaves like the reference. The 200 tasks run from small CLI tools (nnn, fzf, gron) to FFmpeg, SQLite and the PHP interpreter. The median reference is 8,635 lines of code, and the largest is 2.7M. Hidden end-to-end behavioural tests (median 770 per task) come from agent-driven fuzzing of the reference binary. Because the tests check only behaviour, agents can use any language and any architecture. Runs flagged for cheating, such as looking up the original source, are disqualified. With internet access, a panel of nine LM judges flagged 36% of Claude Sonnet 4.6's runs and 21% of Claude Opus 4.6's. Code is MIT-licensed.

The headline metric is % Resolved: every hidden test passes. "Almost resolved" means at least 95% pass. In the paper's mini-SWE-agent baseline, with 1,000 steps and 6 hours per run, none of nine models resolved a single task. Claude Opus 4.7 led the almost-resolved column at 3.0%. Models wrote much less code than the originals (median 1,173 vs 3,068 lines among solutions passing 75% of tests), usually as monolithic single files: GPT-5.4 wrote a median 96% of its code in one turn. Five months later the public leaderboard (updated 2026-10-22) has moved a long way. Claude Opus 5.5 (xhigh) resolves 18.5% and almost-resolves 70.5%, ahead of Claude Fable 5.1 (9.0% / 55.5%) and Muse Wonder Full (8.5% / 60.0%). Launch posts have adopted it quickly: Muse Wonder, Step 5 Preview (80.5), Hy4 preview (39.5 vs Opus 5's 17.5) and DeepSeek V4.1 Flash (29.1) all report it. Vendors do not always say which metric or harness they used, and the leaderboard has three columns (resolved, almost resolved, average test pass rate), so vendor numbers cannot be compared directly.

The benchmark comes from the SWE-bench / SWE-agent group, now at Meta. Most authors are Meta FAIR (John Yang, Kilian Lieret, Gabriel Synnaeve, Ofir Press) or Meta TBD (Pengcheng Yin, Rui Hou). Diyi Yang (Stanford) and Jeffrey Ma (Meta FAIR and Harvard) are co-authors. It follows SWE-bench and uses the mini-SWE-agent harness, but replaces single-issue repair with whole-program construction.

Paper

Authors: John Yang · Kilian Lieret · Jeffrey Ma · Parth Thakkar · Dmitrii Pedchenko · Sten Sootla · Emily McMilin · Pengcheng Yin · Rui Hou · Gabriel Synnaeve · Diyi Yang · Ofir Press

Evaluation Details

Tasks 200
Domains 1
Scoring Hidden behavioural tests generated by agent-driven fuzzing of the reference executable. % Resolved = all tests pass and the run is not flagged for cheating; 'almost resolved' = at least 95% of tests pass; the leaderboard also reports average test pass rate. Baseline harness: mini-SWE-agent, 1,000 steps, 6 hours, 20 CPUs / 60 GB RAM per run.
Saturation Not saturated: best fully resolved rate 18.5% (Claude Opus 5.5 xhigh) on the 2026-10-22 leaderboard; 0% for every model in the May 2026 paper.
Domains: software engineering

Top Scores

Model Score Date
Claude Opus 5.5 (xhigh, mini-SWE-agent; resolved / almost resolved 70.5%) 18.5% 2026-10-22
Claude Fable 5.1 (xhigh, mini-SWE-agent; almost resolved 55.5%) 9.0% 2026-10-22
Muse Wonder Full (mini-SWE-agent; almost resolved 60.0%) 8.5% 2026-10-22
benchmarkevaluationcodingagentic

Related