GSO
evalYour notes
A software-optimization benchmark for coding agents from UC Berkeley (Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen and Ion Stoica, all Berkeley). An automated pipeline mines repository commit histories for real performance improvements and generates performance tests for them, yielding 102 tasks across 10 codebases and five languages. The agent gets the codebase before the expert's commit plus a performance test as the specification, and must make the code faster while keeping it correct. The headline metric, Opt@1, is the fraction of tasks where a single attempt passes the correctness tests and reaches at least 95% of the human developer's speedup. At release, leading SWE-agents succeeded on under 5% of tasks, struggling with low-level languages, settling for lazy optimizations and mislocalizing bottlenecks. MIT license.
Epoch AI includes GSO in the Epoch Capabilities Index, sourcing scores from the GSO leaderboard, which runs every model in OpenHands. The maintainers have repeatedly hardened it: a hack detector since November 2025 penalizes deceptive optimizations such as memoization or harness hijacking (reported as a "hack-adjusted" score), and since July 2026 containers are cut off from GitHub/GitLab after newer models were caught fetching upstream commits. Scores jumped after a 27 September 2026 elicitation change asking models to keep measuring and improving after their first speedup: Claude Fable 5.1 now leads at 88.2% Opt@1 (87.3% hack-adjusted), ahead of GPT-6 Astra (79.4%) and Claude Fable 5 (78.4%), against 47.1% for Claude Opus 4.8 in July. A July 2026 independent audit (Chen et al.) found that the official reference patches satisfy GSO's own validity rules in every cross-machine replay on only 39 of 102 tasks, so small leaderboard differences carry real measurement noise.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| Claude Fable 5.1 (xhigh, OpenHands) | 88.2% | 2026-09-27 |
| GPT-6 Astra (xhigh, OpenHands) | 79.4% | 2026-09-27 |
| Claude Fable 5 (xhigh, OpenHands) | 78.4% | 2026-09-27 |