SWE-bench (Verified / Multimodal / Multilingual)

The de-facto standard coding-agent evaluation family from Karthik Narasimhan's group. SWE-bench Verified (Aug 2024), a human-validated 500-task subset released jointly with OpenAI, became the industry frontier-model coding metric (74K+ HuggingFace downloads); SWE-bench Multimodal (ICLR 2025) extends the task to visual/JS bug-fixing; and SWE-bench Multilingual (2025) adds 300 tasks across 9 languages and 42 repos (113K+ downloads, the most of any dataset in the org). The datasets live under Princeton's SWE-bench org; Verified is an OpenAI co-branded release. The original SWE-bench (2023) is the pre-window anchor of the line; external spin-offs SWE-bench-Live (Microsoft), SWE-bench Pro (Scale AI), and Multi-SWE-bench (ByteDance) build on it but are not Princeton output.