Toolathlon (The Tool Decathlon)

Execution-based benchmark for long-horizon tool use (ICLR 2026): language agents operate 32 real software applications and 604 tools — from Google Calendar and Notion to WooCommerce, Kubernetes, and BigQuery — mostly via curated Model Context Protocol servers, with realistic environment state and reliable execution-based checking over 108 tasks averaging ~20 turns. At release the best model (Claude-4.5-Sonnet) reached only 38.6%, the top open-weights model (DeepSeek-V3.2-Exp) 20.1% — and it saw fast adoption as the long-horizon tool-use eval.

HKUST-led (Junxian He's hkust-nlp) with All Hands AI, CMU, and Duke collaborators.