A software-optimization benchmark for coding agents from UC Berkeley (Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen and Ion Stoica, all Berkeley). An automated pipeline mines repository commit histories for real performance improvements and generates performance tests for them, yielding 102 tasks across 10 codebases and five languages. The agent gets the codebase before the expert's commit plus a performance test as the specification, and must make the code faster while keeping it correct. The headline metric, Opt@1, is the fraction of tasks where a single attempt passes the correctness tests and reaches at least 95% of the human developer's speedup. At release, leading SWE-agents succeeded on under 5% of tasks, struggling with low-level languages, settling for lazy optimizations and mislocalizing bottlenecks. MIT license.

Epoch AI includes GSO in the Epoch Capabilities Index, sourcing scores from the GSO leaderboard, which runs every model in OpenHands. The maintainers have repeatedly hardened it: a hack detector since November 2025 penalizes deceptive optimizations such as memoization or harness hijacking (reported as a "hack-adjusted" score), and since July 2026 containers are cut off from GitHub/GitLab after newer models were caught fetching upstream commits. Scores jumped after a 27 September 2026 elicitation change asking models to keep measuring and improving after their first speedup: Claude Fable 5.1 now leads at 88.2% Opt@1 (87.3% hack-adjusted), ahead of GPT-6 Astra (79.4%) and Claude Fable 5 (78.4%), against 47.1% for Claude Opus 4.8 in July. A July 2026 independent audit (Chen et al.) found that the official reference patches satisfy GSO's own validity rules in every cross-machine replay on only 39 of 102 tasks, so small leaderboard differences carry real measurement noise.

Paper

Authors: Manish Shetty · Naman Jain · Jinjian Liu · Vijay Kethanaboyina · Koushik Sen · Ion Stoica

Evaluation Details

Tasks 102
Domains 2
Scoring Opt@1: fraction of tasks where one attempt passes correctness tests and reaches at least 95% of the expert commit's speedup (harmonic mean over workloads); a hack detector produces a hack-adjusted score.
Saturation Approaching saturation: under 5% at release (May 2025), 88.2% Opt@1 for Claude Fable 5.1 by September 2026
Used in: Epoch Capabilities Index
Domains: software performance optimization, repository-level code editing

Top Scores

Model Score Date
Claude Fable 5.1 (xhigh, OpenHands) 88.2% 2026-09-27
GPT-6 Astra (xhigh, OpenHands) 79.4% 2026-09-27
Claude Fable 5 (xhigh, OpenHands) 78.4% 2026-09-27
benchmarkevaluationcodingagentsoptimization

Related