SciCode
evalYour notes
A research-coding benchmark written by working scientists. SciCode has 80 main problems decomposed into 338 subproblems, drawn from 16 subfields of mathematics, physics, chemistry, biology and materials science (condensed matter and optics are the largest groups). Each subproblem asks for a Python function that combines knowledge recall, reasoning and code synthesis, and is checked against scientist-annotated gold solutions and domain-specific test cases. A main problem counts as solved only if every subproblem in it passes. Problems come with optional scientist-written background text, so the benchmark can be run with or without that help. 15 main problems (50 subproblems) are a development split; the other 65 (288 subproblems) are the test set. Code and data are Apache 2.0.
At release the best model, Claude 3.5 Sonnet, solved 4.6% of main problems in the standard setting (no background) and 26.0% of subproblems. GPT-4o and Claude 3 Opus solved 1.5% of main problems, and the best open model, DeepSeek-Coder-V2, 3.1%. SciCode was a 10% component of the Artificial Analysis Intelligence Index from v4.2 through v4.3.2, in the Coding category next to Terminal-Bench 4.0, and was removed in v5.0 (27 October 2026), where FrontierCode 2.0 took over most of the Coding weight; AA still reports it as a legacy evaluation. AA runs the 288-subproblem test set three times with background prompting and scores at the subproblem level, so its numbers are not comparable with the paper's main-problem rate. In AA v4.2 it raised the grading timeout from 60s to 300s and isolated script execution, so slow but correct code stopped failing (regraded as v1.0.1). On AA the top scores are about two-thirds: Claude Opus 5.5 (max) 66.9%, Claude Fable 5.1 (max) 63.1%, Muse Wonder Full (max) 62.6%. As of 2026-10-24 AA marks SciCode "under review", saying an independent audit has flagged possible errors in the dataset.
The lead authors are Minyang Tian (UIUC and Argonne) and Luyu Gao (CMU), with equal contribution. The corresponding authors are Tian, Hao Peng (UIUC) and Eliu Huerta (UIUC, Argonne and UChicago). Ofir Press and Kilian Lieret (Princeton) and Jamie Callan (CMU) are co-authors, and most of the data curators are UIUC scientists. The same UIUC/Argonne group later built CritPt, the other UIUC-led benchmark in the index.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| Claude Opus 5.5 (max; Artificial Analysis, subproblem scoring) | 66.9% | 2026-10-24 |
| Claude Fable 5.1 (max; Artificial Analysis) | 63.1% | 2026-10-24 |
| Claude 3.5 Sonnet (paper, main problems, standard setting) | 4.6% | 2024-07-18 |