A capability-ladder benchmark for exploit construction from Seunghyun Lee and David Brumley (Carnegie Mellon; Brumley also Bugcrowd). Its premise is that a crash is not an exploit: rather than score success as a binary, ExploitBench decomposes exploitation into 16 deterministically graded flags in five tiers, from coverage of the vulnerable code (T5) and reproduction (T4), through V8-sandbox primitives such as addrof/fakeobj (T3) and arbitrary read/write beyond the sandbox (T2), to control-flow hijack with arbitrary code execution (T1). Each flag is checked by an oracle (per-run randomized challenge-response for primitives, differential execution against ground-truth binaries, a signal-handler proof for code execution). The first instance, v8-bench, uses 41 V8 N-day bugs reported in 2024 or later, attacked through the real JavaScript/WebAssembly surface of default release builds with the heap sandbox, ASLR and stack canaries on. MIT-licensed, about 440 GitHub stars.

The paper found a sharp split. Across eight publicly deployed models, reaching the bug and crashing it was routine, but only GPT-5.5 hijacked control flow, on one WebAssembly bug; the non-public Claude Mythos Preview reached arbitrary code execution on 18 of 41 bugs within a 300-turn budget. The public leaderboard, which reports each model's best run (seeds, turn budgets and harnesses vary), is topped by Mythos Preview with the AutoNudge keep-going harness at 78% mean capability coverage, then GPT-5.5 in Codex with AutoNudge at 72%.

Epoch AI includes ExploitBench v0.1 in the Epoch Capabilities Index, plotting the leaderboard's mean fraction of flags lit. It is also a standard row in frontier cyber reporting, and at the top it is already saturated on vendor numbers: OpenAI reports 100% for GPT-6 Astra, while Meta reports 58.9% for Muse Wonder Full and Z.ai 54.4% for GLM-5.3 (against 24.4% for GLM-5.2). These vendor scores are self-reported and do not appear on the public leaderboard.

Paper

Authors: Seunghyun Lee · David Brumley

Evaluation Details

Tasks 41
Domains 3
Scoring 16 capability flags in five tiers (T5 coverage to T1 control-flow hijack and arbitrary code execution), each verified by a deterministic oracle; leaderboard and Epoch report mean capability coverage (fraction of flags lit, averaged over seeds).
Saturation Saturated at the top on vendor reporting: OpenAI reports 100% for GPT-6 Astra; the public leaderboard (which does not list Astra) tops out at 78% for Claude Mythos Preview
Used in: Epoch Capabilities Index (ExploitBench v0.1)OpenAI GPT-6 Astra launchMeta Muse Wonder safety reportZ.ai GLM-5.3 launch
Domains: browser exploitation (V8 JavaScript/WebAssembly), exploit primitive construction, sandbox escape

Top Scores

Model Score Date
GPT-6 Astra (OpenAI-reported) 100% —
Claude Mythos Preview (AutoNudge, 5 seeds; public leaderboard) 78% —
GPT-5.5 (Codex, AutoNudge; public leaderboard) 72% —
Muse Wonder Full (Meta-reported) 58.9% —
benchmarkevaluationagentssecuritycybersecurity

Related