Cybench
evalYour notes
A framework and benchmark of 40 professional-level Capture-the-Flag tasks drawn from four recent CTF competitions (HackTheBox, SekaiCTF 2022 and 2023, Glacier, HKCert), chosen to span a wide difficulty range. Each task has a description, starter files and a Kali Linux environment with any task servers, and the agent works through bash and network calls until it submits a flag. Because many tasks were beyond 2024 agents, Cybench adds subtasks that break each task into graded intermediate steps, and it measures difficulty by first solve time, how long the fastest human team took in the original competition (24 hours 54 minutes for the hardest). In the original evaluation of eight models, unguided agents built on Claude 3.5 Sonnet, GPT-4o, o1-preview and Claude 3 Opus solved only tasks that took human teams up to 11 minutes. The 27 authors are at Stanford, led by Andy K. Zhang with senior authors Dan Boneh, Daniel E. Ho and Percy Liang. ICLR 2025 oral; Apache 2.0.
Cybench became the default open cyber benchmark of frontier safety reporting. US and UK AISI used it in their joint pre-deployment tests of Claude 3.5 Sonnet and o1, and UK AISI ships it in Inspect Evals; Anthropic has reported it in system cards from Claude 3.7 Sonnet through Claude Opus 4.7, alongside xAI (Grok 4 family), Amazon (Nova Premier) and Meta (Muse Spark). Epoch AI includes it in the Epoch Capabilities Index, using scores reported by model developers. It is now saturated: the authors' leaderboard lists Claude Mythos Preview at 100% and Claude Opus 4.7 at 96% (on 35-task subsets), Claude Opus 4.6 at 93%, and Mistral reports 93% for Mistral Large 4. The vendor numbers use subsets and pass@1 estimates that differ from card to card, so they are not strictly comparable. The same group's BountyBench is its successor for real-world bug bounties.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| Claude Mythos Preview (system card, 35 tasks) | 100% | — |
| Claude Opus 4.7 (system card, 35 tasks) | 96% | — |
| Claude Opus 4.6 (system card, 37 tasks) | 93% | — |
| Mistral Large 4 (Mistral-reported) | 93% | — |