A framework and benchmark of 40 professional-level Capture-the-Flag tasks drawn from four recent CTF competitions (HackTheBox, SekaiCTF 2022 and 2023, Glacier, HKCert), chosen to span a wide difficulty range. Each task has a description, starter files and a Kali Linux environment with any task servers, and the agent works through bash and network calls until it submits a flag. Because many tasks were beyond 2024 agents, Cybench adds subtasks that break each task into graded intermediate steps, and it measures difficulty by first solve time, how long the fastest human team took in the original competition (24 hours 54 minutes for the hardest). In the original evaluation of eight models, unguided agents built on Claude 3.5 Sonnet, GPT-4o, o1-preview and Claude 3 Opus solved only tasks that took human teams up to 11 minutes. The 27 authors are at Stanford, led by Andy K. Zhang with senior authors Dan Boneh, Daniel E. Ho and Percy Liang. ICLR 2025 oral; Apache 2.0.

Cybench became the default open cyber benchmark of frontier safety reporting. US and UK AISI used it in their joint pre-deployment tests of Claude 3.5 Sonnet and o1, and UK AISI ships it in Inspect Evals; Anthropic has reported it in system cards from Claude 3.7 Sonnet through Claude Opus 4.7, alongside xAI (Grok 4 family), Amazon (Nova Premier) and Meta (Muse Spark). Epoch AI includes it in the Epoch Capabilities Index, using scores reported by model developers. It is now saturated: the authors' leaderboard lists Claude Mythos Preview at 100% and Claude Opus 4.7 at 96% (on 35-task subsets), Claude Opus 4.6 at 93%, and Mistral reports 93% for Mistral Large 4. The vendor numbers use subsets and pass@1 estimates that differ from card to card, so they are not strictly comparable. The same group's BountyBench is its successor for real-world bug bounties.

Paper

Venue ICLR 2025 (Oral)
Authors: Andy K. Zhang · Neil Perry · Riya Dulepet · Joey Ji · Celeste Menders · Justin W. Lin · Eliot Jones · Dan Boneh · Daniel E. Ho · Percy Liang

Evaluation Details

Tasks 40
Domains 6
Scoring Unguided: fraction of the 40 tasks where the agent submits the correct flag; subtask-guided and per-subtask scores give partial credit; first solve time of the hardest task solved measures difficulty reached.
Saturation Saturated: 100% for Claude Mythos Preview (35-task subset) and 93-96% for several frontier models, against a best of 17.5% unguided (Claude 3.5 Sonnet) at release
Used in: Epoch Capabilities IndexUS/UK AISI pre-deployment testsAnthropic, xAI, Amazon and Meta system cards
Domains: capture the flag, cryptography, web security, reverse engineering, binary exploitation, forensics

Top Scores

Model Score Date
Claude Mythos Preview (system card, 35 tasks) 100% —
Claude Opus 4.7 (system card, 35 tasks) 96% —
Claude Opus 4.6 (system card, 37 tasks) 93% —
Mistral Large 4 (Mistral-reported) 93% —
benchmarkevaluationagentssecuritycybersecurity

Related