A research-coding benchmark written by working scientists. SciCode has 80 main problems decomposed into 338 subproblems, drawn from 16 subfields of mathematics, physics, chemistry, biology and materials science (condensed matter and optics are the largest groups). Each subproblem asks for a Python function that combines knowledge recall, reasoning and code synthesis, and is checked against scientist-annotated gold solutions and domain-specific test cases. A main problem counts as solved only if every subproblem in it passes. Problems come with optional scientist-written background text, so the benchmark can be run with or without that help. 15 main problems (50 subproblems) are a development split; the other 65 (288 subproblems) are the test set. Code and data are Apache 2.0.

At release the best model, Claude 3.5 Sonnet, solved 4.6% of main problems in the standard setting (no background) and 26.0% of subproblems. GPT-4o and Claude 3 Opus solved 1.5% of main problems, and the best open model, DeepSeek-Coder-V2, 3.1%. SciCode was a 10% component of the Artificial Analysis Intelligence Index from v4.2 through v4.3.2, in the Coding category next to Terminal-Bench 4.0, and was removed in v5.0 (27 October 2026), where FrontierCode 2.0 took over most of the Coding weight; AA still reports it as a legacy evaluation. AA runs the 288-subproblem test set three times with background prompting and scores at the subproblem level, so its numbers are not comparable with the paper's main-problem rate. In AA v4.2 it raised the grading timeout from 60s to 300s and isolated script execution, so slow but correct code stopped failing (regraded as v1.0.1). On AA the top scores are about two-thirds: Claude Opus 5.5 (max) 66.9%, Claude Fable 5.1 (max) 63.1%, Muse Wonder Full (max) 62.6%. As of 2026-10-24 AA marks SciCode "under review", saying an independent audit has flagged possible errors in the dataset.

The lead authors are Minyang Tian (UIUC and Argonne) and Luyu Gao (CMU), with equal contribution. The corresponding authors are Tian, Hao Peng (UIUC) and Eliu Huerta (UIUC, Argonne and UChicago). Ofir Press and Kilian Lieret (Princeton) and Jamie Callan (CMU) are co-authors, and most of the data curators are UIUC scientists. The same UIUC/Argonne group later built CritPt, the other UIUC-led benchmark in the index.

Paper

Authors: Minyang Tian · Luyu Gao · Shizhuo Dylan Zhang · Xinan Chen · Cunwei Fan · Xuefei Guo · Roland Haas · Pan Ji · Kittithat Krongchon · Yao Li · Shengyan Liu · Di Luo · Yutao Ma · Hao Tong · Kha Trinh · Chenyu Tian · Zihan Wang · Bohao Wu · Yanyu Xiong · Shengzhu Yin · Minhui Zhu · Kilian Lieret · Yanxin Lu · Genglin Liu · Yufeng Du · Tianhua Tao · Ofir Press · Jamie Callan · Eliu Huerta · Hao Peng

Evaluation Details

Questions 338
Tasks 80
Domains 5
Scoring Execution against scientist-written unit tests. A main problem is solved only if all its subproblems pass, and subproblem pass rate is reported as an easier metric. Artificial Analysis scores the 288-subproblem test set at subproblem level, pass@1 over 3 repeats, with scientist-annotated background prompting.
Saturation Not saturated: best AA subproblem score 66.9% (Claude Opus 5.5 max, October 2026). AA flags the dataset as under review after an independent audit reported possible errors.
Used in: AA Intelligence Index through v4.3.2 (10% from v4.2; Coding category; removed in v5.0)
Domains: physics, mathematics, chemistry, biology, materials science

Top Scores

Model Score Date
Claude Opus 5.5 (max; Artificial Analysis, subproblem scoring) 66.9% 2026-10-24
Claude Fable 5.1 (max; Artificial Analysis) 63.1% 2026-10-24
Claude 3.5 Sonnet (paper, main problems, standard setting) 4.6% 2024-07-18
benchmarkevaluationcodingscience

Related