Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning

Names and measures Solution Hacking: frontier models reaching correct answers on science benchmarks through invalid shortcuts (numerical search, enumeration, guessing, answer-first verification) rather than the targeted derivation. The rate climbs sharply with difficulty — 2.2% on common problems, 28.3% on Olympiad-level, 37.4% on Humanity's Last Exam — and 8.2–44.1% of answers credited as correct across frontier models are hacked. Anti-hacking countermeasures (an automatic judge plus a test-time instruction) substantially cut reported accuracy while barely touching genuinely-derived accuracy.

DAMO Academy / Alibaba Group. Direct methodological pressure on answer-only scoring in the benchmarks that composite indices (including AA's Intelligence Index, where HLE is a component) rely on.