AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
evalYour notes
Models now clear high-school and competition mathematics, and the benchmarks that measured that progress score a final answer. Advanced mathematics does not have final answers — it has proofs, where the interesting failure is a step, not a result. AdvancedMathBench is built around that: a benchmark suite for reasoning process correctness in natural-language proofs at undergraduate (UG) and doctoral qualifying-exam (QE) level, drawn from examinations, competitions and textbooks. It has three pieces. ProverBench is 245 expert-reviewed proof problems (200 UG, 45 QE). VerifierBench is 888 model-generated proof trajectories (168 UG, 720 QE) carrying full-chain expert annotations that separate fatal from recoverable errors and include the expert's explanation. AutoVerifier is a trained proof verifier — a Qwen3.5-MoE checkpoint of roughly 68 GiB — that makes ProverBench scorable without a human, under a deliberately pessimistic protocol: a proof is accepted only when all eight verifier judgments accept it.
The paper preceded the artifacts by two and a half months. It first appeared on arXiv on 13 July 2026, describing an earlier revision with 296 ProverBench problems and 13 authors; the author-approved final manuscript is dated 28 September 2026, and the repository, the Hugging Face dataset and the AutoVerifier checkpoint all landed on 29 September 2026, alongside the matching arXiv v2 with 245 problems, 888 proofs and Zhouqi Hua added as fifth author. The release notes warn that counts and results from different manuscript versions should not be mixed.
Frontier models do not do well. The best prover, GPT-5.5-xhigh, reaches 64.5 on UG and 48.9 on QE; Claude-Opus-4.8 gets 59.0 / 40.0, DeepSeek-V4-Pro 54.0 / 40.0, and gpt-oss-120b 20.5 / 2.2. The lab scores its own model honestly at the bottom of the open-source block: Intern-S2-Preview-35B manages 27.0 / 16.7. Verification is where the suite earns its design. The best Balanced F1 (the harmonic mean of TPR and TNR) is only 65.1, and the TPR/TNR split exposes models that simply accept: Claude-Opus-4.8 has the highest true-positive rate at 96.4 but a 37.9 TNR, for a 54.4 Balanced F1, and Gemini-3.1-Pro-Preview runs 94.0 / 49.0. Adding meta-verification — does the model's error analysis agree with the expert annotation, judged by gpt-oss-120b — pulls every TNR down further (DeepSeek-V4-Pro 55.8, Claude-Opus-4.8 35.0, gpt-oss-120b 32.0), which is the suite's sharpest finding: agreeing with an expert's binary verdict is not the same as identifying the right error, and proof-generation rank does not predict proof-verification rank.
Caveats the release states itself. The published tables are paper-reported measurements, not reproductions from the standalone evaluator; the manuscript used four independent responses at temperature 1.0 with 128k maximum output and highest reasoning effort, while the CLI defaults to one sample and the example configs to 65,536 tokens, so "example settings are not a turnkey reproduction of paper results". No code license is declared, and dataset and checkpoint terms are separate. The repository ships only the evaluator, three task prompts and pinned download helpers — not the internal training framework or the dataset-construction pipeline — and full GPU inference of AutoVerifier is not covered by its offline tests. Where OpenBMB's MA-ProofBench tests the same two difficulty tiers through Lean 4 and Mathlib, AdvancedMathBench keeps the proofs in natural language and pays for it with a trained verifier and 888 annotated trajectories. Shanghai AI Lab; Lingkai Kong first author, with Wenwei Zhang and Kai Chen. The GitHub repository was still at 0 stars at filing, with 1 like on each Hugging Face artifact.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| GPT-5.5-xhigh | 64.5 UG / 48.9 QE (ProverBench) | — |
| Claude-Opus-4.8 | 59.0 UG / 40.0 QE (ProverBench) | — |
| DeepSeek-V4-Pro | 65.1 Meta Balanced F1 (VerifierBench) | — |
Related
Notes
Dated to the 2026-09-29 artifact release (GitHub repo, HF dataset and AutoVerifier), not to the 2026-07-13 arXiv v1, which described an earlier 296-problem ProverBench revision.