GameHorizon Suite
evalYour notes
A data-and-evaluation suite that measures gameplay capability in AAA titles at three nested temporal horizons, built to compare two families of models that are usually benchmarked separately: general-purpose VLMs and agents, which plan well but are rarely measured on action, and dedicated game agents, which are optimized for acting rather than planning. Three components. GameHorizon-Annotator is an automated pipeline that produces a bottom-up pyramid of natural-language instructions at short-horizon operations (L1), medium-horizon goals (L2) and long-horizon strategies (L3). GameHorizon-Data is 5,000 hours of gameplay from 100 trained human expert players across 21 game titles (open-world, action RPG, competitive shooter, sandbox survival and creature-collecting genres; Valorant is the largest share at 617.5 hours / 12.35%), with 411.03 million synchronized keyboard-and-mouse events (22.84 per second) and 6,184,036 instructions — 5,947,588 L1, 189,158 L2 and 47,290 L3, averaging 2.63 s, 82.6 s and 330.4 s of coverage respectively, so every frame is simultaneously aligned to all three horizons. After filtering cutscenes and window switches, 4,341 hours (86.8%) are kept for instruction annotation while actions remain available for all 5,000 hours. The paper presents it as the first corpus to jointly provide large-scale AAA gameplay, directly recorded human actions and dense multi-horizon instructions.
GameHorizon-Bench has a reproducible offline track and a stepwise online track. The offline track is 5,000 four-option MCQs — three primary tasks of 1,000 questions each plus ten diagnostic variants of 200 — derived from the aligned frames, instructions and actions, so scoring is independent of game environments and agent harnesses. T1 (single-horizon action) asks for the correct action sequence given frames and an L1 instruction, with ground-truth keyboard-mouse controls mapped to text through per-game keybinds; T2 (multi-horizon decomposition) asks for the ordered L1 sequence that realizes an L2 goal; T3 (cross-horizon consistency) asks for the aligned L3/L2/L1 instructions plus action sequence. The online track runs in Minecraft and comprises 10 order-dependent causal and 10 order-flexible thematic long-horizon tasks built from 62 verifiable short-horizon subtasks; on a subtask failure the environment resets to the corresponding success state so evaluation continues and failures localize to a stage, and a long-horizon task passes only if completed end-to-end with no resets (each subtask capped at 400 model calls).
The paper evaluates 47 models — 35 general-purpose VLMs, 4 unified multimodal models, 2 GUI agents, 3 coding agents and 3 dedicated game agents — in more than one million model inferences and API calls; the 3 game agents, which cannot answer questions, are scored on T1 only by Needleman-Wunsch alignment of their predicted action sequences against the four options. Across the 44 question-answering models mean accuracy is 64.7% against a 25% random baseline, with a 44.6%–80.2% range (35.6 points) and a clean difficulty gradient: T1 57.3% < T2 65.1% < T3 71.6% — fine-grained action prediction is the hardest, not cross-horizon understanding. GPT-6-Astra leads at 80.2% (T1 69.4, T2 79.6, T3 91.5), ahead of Gemini 3.8 Flash (77.3%) and Gemini 3.7 Flash (76.7%); proprietary models hold ten of the eleven Tier-1 slots and Kimi-K3 is the only open-weight model in Tier 1, sixth at 74.5%. Twelve models (three per tier) also ran online, where offline rank and online success correlate: GPT-6-Astra passes 66.1% of short-horizon subtasks (41/62) and 45.0% of long-horizon tasks (9/20), Gemini 3.6 Flash 56.5% / 30.0%, Kimi-K3 46.8% / 10.0%. A diagnostic variant shows that adding medium- and long-horizon instructions to vision-only input improves future-action planning by 7.2 percentage points.
From ARC Lab, Tencent — project lead and first author Yiran Wang, corresponding author Xiaodong Cun, senior author Ying Shan — with co-authors at the GVC Lab of Great Bay University, NUS, HUST, MMLab CUHK and the University of Macau. The date marks the arXiv v1 posting; the repository (Apache 2.0) currently carries the paper, project page and full leaderboard, with the annotator, offline and online benchmarks and the 5,000-hour corpus scheduled for release from around 25 October 2026. Companion work from the same lab on game and world video generation: Hunyuan-GameCraft and WorldCrafter.
Paper
Evaluation Details
Top Scores
| Model | Score | Date |
|---|---|---|
| GPT-6-Astra | 80.2% | 2026-09-21 |
| Gemini 3.8 Flash | 77.3% | 2026-09-21 |
| Gemini 3.7 Flash | 76.7% | 2026-09-21 |
| Kimi-K3 | 74.5% | 2026-09-21 |