MMMU-Pro
evalYour notes
A harder version of MMMU, the college-level multimodal benchmark (11.5K questions, 30 subjects in six disciplines). Users reported that text-only LLMs could answer some MMMU questions without looking at the image. MMMU-Pro fixes this in three steps. It removes questions that text-only LLMs can answer, it expands the options from 4 to up to 10 to cut guessing, and it adds a vision-only setting in which the question and options are inside a screenshot or photo, so the model has to read and see at once. The result is 1,730 questions, each in standard format and as an image (3,460 items). The headline score is the average of the 10-option standard setting and the vision setting.
Scores fell sharply from MMMU: models lost 16.8 to 26.9 points. GPT-4o (0513) dropped from 69.1 on MMMU validation to 54.0 (standard, 10 options) and 49.7 (vision). OCR prompts made little difference, and chain-of-thought generally helped. The paper was published at ACL 2025 (main) and is the standard multimodal row in launch tables (e.g. Gemma 4, Muse Glimmer, Command A+ and Intern-S2). Artificial Analysis runs it as an additional evaluation outside the Intelligence Index, on the 1,730-question 10-option set, single pass with regex extraction. The top AA scores are now in the high 80s: Claude Opus 5.5 (max) 87.7%, GPT-6 Astra (max) 86.9%, GPT-6.1 Sol (max) 86.0%.
The paper is credited to the "MMMU Team" and lists no institutions. It is filed under CMU because the corresponding author, Xiang Yue ([email protected]), and the senior author, Graham Neubig, are at CMU. The other authors include the original MMMU group at Ohio State (Huan Sun, Yu Su) and Waterloo (Wenhu Chen), which are not tracked. The original MMMU (2023) was led by Ohio State and is not filed separately. The GitHub repo is Apache 2.0.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| Claude Opus 5.5 (max; Artificial Analysis) | 87.7% | 2026-10-24 |
| GPT-6 Astra (max; Artificial Analysis) | 86.9% | 2026-10-24 |
| GPT-4o 0513 (paper, standard 10-option / vision 49.7%) | 54.0% | 2024-09-04 |