A physics benchmark built from unpublished research problems. CritPt ("Complex Research using Integrated Thinking – Physics Test", read as "critical point") has 71 composite research challenges, each meant to simulate an entry-level research project, split into 190 checkpoint tasks. More than 50 active physicists wrote the problems from their own research, across condensed matter, quantum, AMO, astrophysics, high-energy, mathematical, statistical and nuclear physics, nonlinear and fluid dynamics, and biophysics. Each problem has a guess-resistant, machine-checkable answer (number arrays, symbolic expressions, Python functions), graded by a physics-specific pipeline. One example challenge is public. The answers to the other 70 test challenges are private and graded on an online server, which limits contamination.

In the paper, the best base model, GPT-5 (high), averaged 5.7% on full challenges, rising to about 10% with code tools. CritPt joined the Artificial Analysis Intelligence Index in v4.0 (January 2026) and was a 10% component through v4.3.2, in the Scientific Reasoning category next to HLE. In v5.0 (27 October 2026) AA replaced it, at the same 10% in the renamed Science category, with a private 40-question hard subset of CritPt v2: the 70 v2 challenges minus the 30 that seven frontier models solved in all five attempts at launch, run in AA's Stirrup harness with a Python sandbox. GPT-6 Astra (high) leads that subset at 72%, then GPT-6.1 Sol (max) at 69% and Claude Opus 5.5 (max) at 61%. On the v1 set AA ran pass@1 over five repeats through the official grading server, and its top scores were near a third: Muse Wonder Full (max) 32.9%, then GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5 tied at 31.7%. The authors now maintain CritPt v2, a revision of the 70 test challenges in which the authors and outside physicists fixed underspecified statements, errors and typos. AA also runs the full v2 set as a standalone evaluation, in its Stirrup agentic harness with a Python sandbox and no internet. The authors' own v2 leaderboard (average over 5 runs) is far higher: GPT-6 Astra 83.1, GPT-6.1 Sol 82.0, Claude Opus 5.5 77.7. The v1 index scores and the v2 numbers measure different task versions and harnesses, so they should not be compared directly.

Minhui Zhu (Argonne) and Minyang Tian (UIUC) are the equal-contribution lead and corresponding authors, and Nicolas Chia, Eliu Huerta and Hao Peng (Argonne / UIUC / UChicago) are the senior authors. The paper has 65 authors from more than 30 institutions. It is filed under UIUC because Argonne is not tracked. The same group built SciCode. arXiv v4 (May 2026) is the current revision.

Paper

Evaluation Details

Questions 190
Tasks 71
Domains 11
Scoring Full-challenge accuracy on the 70 private test challenges (190 checkpoints give finer-grained scores). Answers are numbers, symbolic expressions or Python functions, graded automatically on the authors' server. Artificial Analysis: pass@1 over 5 repeats.
Saturation Not saturated on the index version (AA best 32.9%, October 2026). The authors' v2 leaderboard tops out at 83.1% (GPT-6 Astra) under a different harness.
Used in: AA Intelligence Index v4.0 through v4.3.2 (CritPt v1; 10% from v4.2; Scientific Reasoning category)AA Intelligence Index v5.0 (CritPt v2 private 40-question hard subset, 10%; Science category)Artificial Analysis standalone evaluation (full CritPt v2)
Domains: condensed matter physics, quantum physics, atomic, molecular and optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics, biophysics

Top Scores

Model Score Date
Muse Wonder Full (max; Artificial Analysis, v1) 32.9% 2026-10-24
GPT-6 Astra (authors' CritPt v2 leaderboard) 83.1% 2026-10-24
GPT-5 high (paper, base model) 5.7% 2025-09-30
benchmarkevaluationsciencereasoning

Related