HealthBench
evalYour notes
OpenAI's rubric-graded benchmarks for open-ended health conversations. HealthBench (May 2025) has 5,000 multi-turn conversations with lay users and health professionals, graded against 48,562 conversation-specific rubric criteria written by 262 physicians. The conversations cover seven themes, from emergency referrals to global health, and each criterion is tagged with one of five axes: accuracy, completeness, communication quality, context awareness and instruction following. HealthBench Professional (arXiv April 2026, released with ChatGPT for Clinicians) is a smaller, harder companion: 525 real clinician conversations in three use cases (care consult, writing and documentation, medical research). About a third are deliberate adversarial testing by physicians, and examples that were hard for OpenAI's own models were sampled at roughly 3.5x their rate in the 15,079-example pool. Both sets ship in OpenAI's open simple-evals repo.
The original paper showed fast progress on OpenAI's own models: GPT-3.5 Turbo 16%, GPT-4o 32%, o3 60%. It also released HealthBench Consensus (34 physician-validated behaviours) and HealthBench Hard (top score 32% at release). In the Professional paper, GPT-5.4 inside ChatGPT for Clinicians scored 59.0 against 43.7 for physicians (specialist-matched, unlimited time, web access) and 48.1 for base GPT-5.4. That result is vendor-run, and the benchmark was sampled for difficulty against OpenAI's models.
Artificial Analysis added both as standalone evaluations on 2026-10-19 (not part of the Intelligence Index). AA reports OpenAI's length-adjusted score, which penalises replies over 2,000 characters, and uses OpenAI's graders: GPT-4.1 for HealthBench and GPT-5.4 (low) for Professional. On Professional, GPT-6 Astra (max) leads at 66.0%, then GPT-6.1 Sol 65.1% and Claude Sonnet 5.5 63.0%. Claude Opus 5.5 has the highest raw score (71.4%) but falls to 59.9% after the length adjustment, because it averages about 5,900 characters per answer. On HealthBench, Claude Sonnet 5.5 leads at 62.1%, ahead of DeepSeek V4.1 Flash 60.9% and GPT-6.1 Sol 58.5%. Vendor launch tables now quote Professional often: Grok 4.7 56.7%, Muse Wonder 62.3 vs Opus 5.5 65.6, and GPT-6 Astra (claimed state of the art). Medical models such as Baichuan-M2 and Baichuan-M3 report the original set. The authors (Rahul Arora, Jason Wei, Karan Singhal and others) are OpenAI's health team, so the lab whose models top the Professional board also built the benchmark, chose its difficulty sampling and supplies its grader models.
Outputs 2
HealthBench
eval5,000 multi-turn health conversations graded against 48,562 physician-written rubric criteria (262 physicians), with HealthBench Consensus and HealthBench Hard subsets.
Top Scores
| Model | Score | Date |
|---|---|---|
| Claude Sonnet 5.5 (max; Artificial Analysis, length-adjusted) | 62.1% | 2026-10-19 |
| o3 (paper) | 60% | 2025-05-13 |
HealthBench Professional
eval525 real clinician conversations with ChatGPT for Clinicians (care consult, writing and documentation, research), about one-third adversarial, sampled for difficulty from 15,079 candidates, with rubrics adjudicated by three or more physicians and a physician-response baseline. Date is arXiv v1; the blog post's date was not independently verified.
Top Scores
| Model | Score | Date |
|---|---|---|
| GPT-6 Astra (max; Artificial Analysis, length-adjusted) | 66.0% | 2026-10-19 |
| Claude Opus 5.5 (max; Artificial Analysis, raw score, 59.9% length-adjusted) | 71.4% | 2026-10-19 |
| GPT-5.4 in ChatGPT for Clinicians (paper, vendor-run) | 59.0 | 2026-04-30 |