A data-and-evaluation suite that measures gameplay capability in AAA titles at three nested temporal horizons, built to compare two families of models that are usually benchmarked separately: general-purpose VLMs and agents, which plan well but are rarely measured on action, and dedicated game agents, which are optimized for acting rather than planning. Three components. GameHorizon-Annotator is an automated pipeline that produces a bottom-up pyramid of natural-language instructions at short-horizon operations (L1), medium-horizon goals (L2) and long-horizon strategies (L3). GameHorizon-Data is 5,000 hours of gameplay from 100 trained human expert players across 21 game titles (open-world, action RPG, competitive shooter, sandbox survival and creature-collecting genres; Valorant is the largest share at 617.5 hours / 12.35%), with 411.03 million synchronized keyboard-and-mouse events (22.84 per second) and 6,184,036 instructions — 5,947,588 L1, 189,158 L2 and 47,290 L3, averaging 2.63 s, 82.6 s and 330.4 s of coverage respectively, so every frame is simultaneously aligned to all three horizons. After filtering cutscenes and window switches, 4,341 hours (86.8%) are kept for instruction annotation while actions remain available for all 5,000 hours. The paper presents it as the first corpus to jointly provide large-scale AAA gameplay, directly recorded human actions and dense multi-horizon instructions.

GameHorizon-Bench has a reproducible offline track and a stepwise online track. The offline track is 5,000 four-option MCQs — three primary tasks of 1,000 questions each plus ten diagnostic variants of 200 — derived from the aligned frames, instructions and actions, so scoring is independent of game environments and agent harnesses. T1 (single-horizon action) asks for the correct action sequence given frames and an L1 instruction, with ground-truth keyboard-mouse controls mapped to text through per-game keybinds; T2 (multi-horizon decomposition) asks for the ordered L1 sequence that realizes an L2 goal; T3 (cross-horizon consistency) asks for the aligned L3/L2/L1 instructions plus action sequence. The online track runs in Minecraft and comprises 10 order-dependent causal and 10 order-flexible thematic long-horizon tasks built from 62 verifiable short-horizon subtasks; on a subtask failure the environment resets to the corresponding success state so evaluation continues and failures localize to a stage, and a long-horizon task passes only if completed end-to-end with no resets (each subtask capped at 400 model calls).

The paper evaluates 47 models — 35 general-purpose VLMs, 4 unified multimodal models, 2 GUI agents, 3 coding agents and 3 dedicated game agents — in more than one million model inferences and API calls; the 3 game agents, which cannot answer questions, are scored on T1 only by Needleman-Wunsch alignment of their predicted action sequences against the four options. Across the 44 question-answering models mean accuracy is 64.7% against a 25% random baseline, with a 44.6%–80.2% range (35.6 points) and a clean difficulty gradient: T1 57.3% < T2 65.1% < T3 71.6% — fine-grained action prediction is the hardest, not cross-horizon understanding. GPT-6-Astra leads at 80.2% (T1 69.4, T2 79.6, T3 91.5), ahead of Gemini 3.8 Flash (77.3%) and Gemini 3.7 Flash (76.7%); proprietary models hold ten of the eleven Tier-1 slots and Kimi-K3 is the only open-weight model in Tier 1, sixth at 74.5%. Twelve models (three per tier) also ran online, where offline rank and online success correlate: GPT-6-Astra passes 66.1% of short-horizon subtasks (41/62) and 45.0% of long-horizon tasks (9/20), Gemini 3.6 Flash 56.5% / 30.0%, Kimi-K3 46.8% / 10.0%. A diagnostic variant shows that adding medium- and long-horizon instructions to vision-only input improves future-action planning by 7.2 percentage points.

From ARC Lab, Tencent — project lead and first author Yiran Wang, corresponding author Xiaodong Cun, senior author Ying Shan — with co-authors at the GVC Lab of Great Bay University, NUS, HUST, MMLab CUHK and the University of Macau. The date marks the arXiv v1 posting; the repository (Apache 2.0) currently carries the paper, project page and full leaderboard, with the annotator, offline and online benchmarks and the 5,000-hour corpus scheduled for release from around 25 October 2026. Companion work from the same lab on game and world video generation: Hunyuan-GameCraft and WorldCrafter.

Paper

Authors: Yiran Wang · Xingyilang Yin · Junfu Pu · Guangzhi Wang · Kaifeng Li · Mingyu Ouyang · Huiqiang Sun · Lingen Li

Evaluation Details

Questions 5,000
Tasks 13
Domains 5
Scoring Four-option MCQ accuracy (offline track); stepwise short-horizon subtask and end-to-end long-horizon task success rate (online track in Minecraft)
Random baseline 25% (four-option MCQ)
Saturation Not saturated (best overall 80.2%; 44-model mean 64.7%)
Domains: AAA video gameplay, action perception, goal decomposition, long-horizon planning, cross-horizon consistency

Top Scores

Model Score Date
GPT-6-Astra 80.2% 2026-09-21
Gemini 3.8 Flash 77.3% 2026-09-21
Gemini 3.7 Flash 76.7% 2026-09-21
Kimi-K3 74.5% 2026-09-21

Library

License Apache 2.0
evalbenchmarkevaluationgamingagentsagenticmultimodalvisionvideodatasetresearch

Related