ExploitBench
evalYour notes
A capability-ladder benchmark for exploit construction from Seunghyun Lee and David Brumley (Carnegie Mellon; Brumley also Bugcrowd). Its premise is that a crash is not an exploit: rather than score success as a binary, ExploitBench decomposes exploitation into 16 deterministically graded flags in five tiers, from coverage of the vulnerable code (T5) and reproduction (T4), through V8-sandbox primitives such as addrof/fakeobj (T3) and arbitrary read/write beyond the sandbox (T2), to control-flow hijack with arbitrary code execution (T1). Each flag is checked by an oracle (per-run randomized challenge-response for primitives, differential execution against ground-truth binaries, a signal-handler proof for code execution). The first instance, v8-bench, uses 41 V8 N-day bugs reported in 2024 or later, attacked through the real JavaScript/WebAssembly surface of default release builds with the heap sandbox, ASLR and stack canaries on. MIT-licensed, about 440 GitHub stars.
The paper found a sharp split. Across eight publicly deployed models, reaching the bug and crashing it was routine, but only GPT-5.5 hijacked control flow, on one WebAssembly bug; the non-public Claude Mythos Preview reached arbitrary code execution on 18 of 41 bugs within a 300-turn budget. The public leaderboard, which reports each model's best run (seeds, turn budgets and harnesses vary), is topped by Mythos Preview with the AutoNudge keep-going harness at 78% mean capability coverage, then GPT-5.5 in Codex with AutoNudge at 72%.
Epoch AI includes ExploitBench v0.1 in the Epoch Capabilities Index, plotting the leaderboard's mean fraction of flags lit. It is also a standard row in frontier cyber reporting, and at the top it is already saturated on vendor numbers: OpenAI reports 100% for GPT-6 Astra, while Meta reports 58.9% for Muse Wonder Full and Z.ai 54.4% for GLM-5.3 (against 24.4% for GLM-5.2). These vendor scores are self-reported and do not appear on the public leaderboard.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| GPT-6 Astra (OpenAI-reported) | 100% | — |
| Claude Mythos Preview (AutoNudge, 5 seeds; public leaderboard) | 78% | — |
| GPT-5.5 (Codex, AutoNudge; public leaderboard) | 72% | — |
| Muse Wonder Full (Meta-reported) | 58.9% | — |