ProgramBench
evalYour notes
A benchmark for building a whole program from scratch. ProgramBench gives an agent only a compiled program and its documentation. The agent has to design and write a codebase, plus a build script, whose executable behaves like the reference. The 200 tasks run from small CLI tools (nnn, fzf, gron) to FFmpeg, SQLite and the PHP interpreter. The median reference is 8,635 lines of code, and the largest is 2.7M. Hidden end-to-end behavioural tests (median 770 per task) come from agent-driven fuzzing of the reference binary. Because the tests check only behaviour, agents can use any language and any architecture. Runs flagged for cheating, such as looking up the original source, are disqualified. With internet access, a panel of nine LM judges flagged 36% of Claude Sonnet 4.6's runs and 21% of Claude Opus 4.6's. Code is MIT-licensed.
The headline metric is % Resolved: every hidden test passes. "Almost resolved" means at least 95% pass. In the paper's mini-SWE-agent baseline, with 1,000 steps and 6 hours per run, none of nine models resolved a single task. Claude Opus 4.7 led the almost-resolved column at 3.0%. Models wrote much less code than the originals (median 1,173 vs 3,068 lines among solutions passing 75% of tests), usually as monolithic single files: GPT-5.4 wrote a median 96% of its code in one turn. Five months later the public leaderboard (updated 2026-10-22) has moved a long way. Claude Opus 5.5 (xhigh) resolves 18.5% and almost-resolves 70.5%, ahead of Claude Fable 5.1 (9.0% / 55.5%) and Muse Wonder Full (8.5% / 60.0%). Launch posts have adopted it quickly: Muse Wonder, Step 5 Preview (80.5), Hy4 preview (39.5 vs Opus 5's 17.5) and DeepSeek V4.1 Flash (29.1) all report it. Vendors do not always say which metric or harness they used, and the leaderboard has three columns (resolved, almost resolved, average test pass rate), so vendor numbers cannot be compared directly.
The benchmark comes from the SWE-bench / SWE-agent group, now at Meta. Most authors are Meta FAIR (John Yang, Kilian Lieret, Gabriel Synnaeve, Ofir Press) or Meta TBD (Pengcheng Yin, Rui Hou). Diyi Yang (Stanford) and Jeffrey Ma (Meta FAIR and Harvard) are co-authors. It follows SWE-bench and uses the mini-SWE-agent harness, but replaces single-issue repair with whole-program construction.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| Claude Opus 5.5 (xhigh, mini-SWE-agent; resolved / almost resolved 70.5%) | 18.5% | 2026-10-22 |
| Claude Fable 5.1 (xhigh, mini-SWE-agent; almost resolved 55.5%) | 9.0% | 2026-10-22 |
| Muse Wonder Full (mini-SWE-agent; almost resolved 60.0%) | 8.5% | 2026-10-22 |