OpenAI's rubric-graded benchmarks for open-ended health conversations. HealthBench (May 2025) has 5,000 multi-turn conversations with lay users and health professionals, graded against 48,562 conversation-specific rubric criteria written by 262 physicians. The conversations cover seven themes, from emergency referrals to global health, and each criterion is tagged with one of five axes: accuracy, completeness, communication quality, context awareness and instruction following. HealthBench Professional (arXiv April 2026, released with ChatGPT for Clinicians) is a smaller, harder companion: 525 real clinician conversations in three use cases (care consult, writing and documentation, medical research). About a third are deliberate adversarial testing by physicians, and examples that were hard for OpenAI's own models were sampled at roughly 3.5x their rate in the 15,079-example pool. Both sets ship in OpenAI's open simple-evals repo.

The original paper showed fast progress on OpenAI's own models: GPT-3.5 Turbo 16%, GPT-4o 32%, o3 60%. It also released HealthBench Consensus (34 physician-validated behaviours) and HealthBench Hard (top score 32% at release). In the Professional paper, GPT-5.4 inside ChatGPT for Clinicians scored 59.0 against 43.7 for physicians (specialist-matched, unlimited time, web access) and 48.1 for base GPT-5.4. That result is vendor-run, and the benchmark was sampled for difficulty against OpenAI's models.

Artificial Analysis added both as standalone evaluations on 2026-10-19 (not part of the Intelligence Index). AA reports OpenAI's length-adjusted score, which penalises replies over 2,000 characters, and uses OpenAI's graders: GPT-4.1 for HealthBench and GPT-5.4 (low) for Professional. On Professional, GPT-6 Astra (max) leads at 66.0%, then GPT-6.1 Sol 65.1% and Claude Sonnet 5.5 63.0%. Claude Opus 5.5 has the highest raw score (71.4%) but falls to 59.9% after the length adjustment, because it averages about 5,900 characters per answer. On HealthBench, Claude Sonnet 5.5 leads at 62.1%, ahead of DeepSeek V4.1 Flash 60.9% and GPT-6.1 Sol 58.5%. Vendor launch tables now quote Professional often: Grok 4.7 56.7%, Muse Wonder 62.3 vs Opus 5.5 65.6, and GPT-6 Astra (claimed state of the art). Medical models such as Baichuan-M2 and Baichuan-M3 report the original set. The authors (Rahul Arora, Jason Wei, Karan Singhal and others) are OpenAI's health team, so the lab whose models top the Professional board also built the benchmark, chose its difficulty sampling and supplies its grader models.

Outputs 2

HealthBench

eval

5,000 multi-turn health conversations graded against 48,562 physician-written rubric criteria (262 physicians), with HealthBench Consensus and HealthBench Hard subsets.

Authors: Rahul K. Arora · Jason Wei · Rebecca Soskin Hicks · Preston Bowman · Joaquin Quiñonero-Candela · Foivos Tsimpourlas · Michael Sharman · Meghan Shah · Andrea Vallone · Alex Beutel · Johannes Heidecke · Karan Singhal
Questions 5,000
Domains 2
Scoring Model-graded rubric score: points for rubric items met over total positive points (negative items subtract), graded by GPT-4.1. AA reports OpenAI's length-adjusted variant (2.99 points per 500 characters away from 2,000).
Saturation Not saturated: AA best 62.1% length-adjusted (Claude Sonnet 5.5, October 2026); HealthBench Hard top score 32% at release.
Used in: Artificial Analysis standalone evaluation (from 2026-10-19)
Domains: medicine, health

Top Scores

Model Score Date
Claude Sonnet 5.5 (max; Artificial Analysis, length-adjusted) 62.1% 2026-10-19
o3 (paper) 60% 2025-05-13

HealthBench Professional

eval

525 real clinician conversations with ChatGPT for Clinicians (care consult, writing and documentation, research), about one-third adversarial, sampled for difficulty from 15,079 candidates, with rubrics adjudicated by three or more physicians and a physician-response baseline. Date is arXiv v1; the blog post's date was not independently verified.

Authors: Rebecca Soskin Hicks · Mikhail Trofimov · Dominick Lim · Rahul K. Arora · Foivos Tsimpourlas · Preston Bowman · Michael Sharman · Chi Tong · Kavin Karthik · Arnav Dugar · Akshay Jagadeesh · Khaled Saab · Johannes Heidecke · Ashley Alexander · Nate Gross · Karan Singhal
Questions 525
Domains 2
Scoring Model-graded rubric score, graded by GPT-5.4 (low); AA reports the length-adjusted score (1.47 points per 500 characters away from 2,000) over 4 repeats.
Human baseline 43.7 (specialist-matched physicians, unbounded time, web access)
Saturation Not saturated: AA best 66.0% length-adjusted (GPT-6 Astra max, October 2026).
Used in: Artificial Analysis standalone evaluation (from 2026-10-19)
Domains: medicine, clinical practice

Top Scores

Model Score Date
GPT-6 Astra (max; Artificial Analysis, length-adjusted) 66.0% 2026-10-19
Claude Opus 5.5 (max; Artificial Analysis, raw score, 59.9% length-adjusted) 71.4% 2026-10-19
GPT-5.4 in ChatGPT for Clinicians (paper, vendor-run) 59.0 2026-04-30
benchmarkevaluationmedical

Related