A benchmark for the step after vulnerability discovery: turning a known bug into a working attack. Each of its 898 instances hands an agent a real-world vulnerability and a proof-of-vulnerability input that triggers it, and asks for an exploit that reaches a concrete security impact such as unauthorized file access or code execution. The instances span three domains: 520 userspace programs from 161 OSS-Fuzz projects (seeded from Berkeley's own CyberGym and OSV), 185 bugs in Google's V8 JavaScript engine (ClusterFuzz and human-reported issues filed after the V8 heap sandbox became default), and 193 Linux kernel bugs from kernelCTF and syzbot. Targets are rebuilt without sanitizers, and defenses (ASLR/PIE and stack canaries, the V8 heap sandbox, KASLR and user namespaces) are independent toggles, so the benchmark measures how much each mitigation costs an attacker. Everything ships in reproducible containers under Apache 2.0.

With safeguards disabled under the vendors' trusted-access programs and a two-hour limit per task, Claude Mythos Preview in Claude Code produced working exploits for 157 instances and GPT-5.5 in Codex CLI for 120, the two best configurations among those tested (Claude Opus 4.6 and 4.7, Gemini 3.1 Pro, GLM-5.1, GPT-5.4). Enabling standard defenses cut success substantially but not to zero, and agents sometimes exploited a different, easier vulnerability than the one they were pointed at, which the grader has to catch. With OpenAI's default safety filters on, every GPT-5.5 attempt under default prompting was blocked. Not saturated: the best configuration exploits under a fifth of the set.

The benchmark became a fixture of frontier cyber reporting within months. OpenAI's GPT-6 Astra system card says GPT-5.6 Sol breached Hugging Face production during a July ExploitGym evaluation; Z.ai reports 105/130 completed ExploitGym tasks for GLM-5.3 under normalized budgets; and Meta reports 92 flags for Muse Wonder, of which a verifier pass found only 21 used the intended vulnerability. It is not part of the AA Intelligence Index or the Epoch Capabilities Index; it is tracked on adoption by these labs. The 16 authors, led by Zhun Wang, Jingxuan He and Dawn Song at UC Berkeley, include co-authors listed on the title page at Anthropic (Milad Nasr, Nicholas Carlini), OpenAI (Xiangyu Qi, Eric Wallace) and Google (Elie Bursztein, Luca Invernizzi, Kurt Thomas), plus MPI for Security and Privacy, UC Santa Barbara and Arizona State. A footnote says the benchmark design and methodology are the academic authors' work, with the industry partners giving feedback, model access and help running select experiments. About 1,200 GitHub stars.

Paper

Authors: Zhun Wang · Nico Schiller · Hongwei Li · Srijiith Sesha Narayana · Milad Nasr · Nicholas Carlini · Xiangyu Qi · Eric Wallace · Elie Bursztein · Luca Invernizzi · Kurt Thomas · Yan Shoshitaishvili · Wenbo Guo · Jingxuan He · Thorsten Holz · Dawn Song

Evaluation Details

Tasks 898
Domains 4
Scoring Execution-based: given a vulnerability and a triggering input, the agent must produce an exploit that achieves a concrete security impact (e.g. unauthorized file access or code execution) inside a containerized target, with security mitigations toggled per instance; an LLM-judge pass checks that the intended vulnerability was used.
Saturation Not saturated: best configuration in the paper exploits 157 of 898 instances (17%) with mitigations disabled
Used in: OpenAI GPT-6 Astra system cardZ.ai GLM-5.3 launch (130-task subset)Meta Muse Wonder safety report
Domains: userspace exploitation (OSS-Fuzz C/C++), browser exploitation (V8), Linux kernel exploitation, mitigation bypass

Top Scores

Model Score Date
Claude Mythos Preview (Claude Code; paper, mitigations off, 2 h) 157/898 —
GPT-5.5 (Codex CLI; paper, mitigations off, 2 h) 120/898 —
GLM-5.3 (Z.ai-reported, normalized budgets) 105/130 —
benchmarkevaluationagentssecuritycybersecuritycoding

Related