WorldCrafter
modelYour notes
A camera-controllable autoregressive video world model whose long-horizon consistency comes from a camera-queryable implicit 3D-aware memory rather than from explicit geometry. Where depth-based world models reuse history by estimating depth and warping past frames, WorldCrafter learns a memory encoder that writes accumulated history latents plus their camera poses into a learned representation, and a pose-conditioned readout module that compresses that representation into a fixed set of target-view-specific tokens before denoising — so the requested viewpoint decides how multi-view evidence is spent against the generator's limited token budget. The DiT then processes [memory; recent context; noise] as one latent sequence per denoising step, preserving chunk-level causality. Encoder and readout are initialized from LagerNVS (CVPR 2026) to inherit a learned 3D inductive bias and are trained jointly with the video generator; camera control follows PRoPE relative-pose geometry implemented through UCPE's parallel camera-attention branch, applied only to the noisy tokens. At inference the encoder keeps the latest latent frame and greedily picks complementary frames whose joint field of view maximizes coverage of the upcoming trajectory.
The video DiT is initialized from Helios-base (arXiv 2603.04379, from Li Yuan's PKU group behind Open-Sora Plan) and trained in four stages: window adaptation on 760,000 Open-Sora-Plan videos (5,000 iterations, 32 GPUs); camera-branch training on 40,000 filtered OSP clips plus 6,000 DL3DV scenes with the backbone frozen; memory-encoder warm-up on a fixed 9-latent-frame input (5,000 iterations, 16 GPUs); then joint training of readout, encoder, DiT and camera branch for 8,000 iterations plus 1,000 more with synthetic MIND videos for dynamic subjects. Camera poses are annotated at metric scale with Depth Anything 3 and captions with Qwen2.5-VL. A pyramid distillation pass (three spatial resolutions, two denoising steps each, distribution matching distillation) yields WorldCrafter-Fast as a hybrid high-noise/low-noise pair, reaching 16 fps on a 4-GPU machine for streaming exploration.
Evaluation uses a purpose-built benchmark of 145 images (83 dynamic object-centric and 62 static scenes), each paired with five metric camera trajectories spanning 528–1,648 frames with closed-loop revisits — 725 videos per method — against eight recent camera-controllable world models including HY-WorldPlay, Lyra 2.0, Matrix-Game 3.5, SANA-WM, Echo-WM, Alaya-EVOKE, DreamX-World and LingBot-World 2. On the revisit-consistency protocol the two WorldCrafter models take the top two places on all four metrics (MEt3R, LPIPS, PSNR, SSIM); against Lyra 2.0, LPIPS falls from 0.487 to 0.255 and PSNR rises from 14.050 to 18.016 dB. WorldCrafter has the lowest error on all three camera-control metrics (RotErr, TransErr, CamMC) and the best VBench overall score (81.910), leading 5 of 8 VBench dimensions. The memory path is also cheaper than geometry warping: encoding plus readout costs 0.062 s per chunk versus 1.346 s for depth estimation and batched warping — a 21.7× reduction with no explicit depth or warping step. Ablations show the implicit memory beats a 4-frame retrieved-context baseline, joint optimization beats a frozen encoder, pose-guided readout beats pose-free readout, and max-coverage history selection beats FoV-similarity ranking.
Weights ship as WorldCrafter-Base and the distilled WorldCrafter-Fast, with the interactive demo code and serving infrastructure open-sourced alongside. GitHub reports no SPDX identifier; the repository's LICENSE.txt is a custom Tencent license that grants MIT-style permissions but requires users to "use the WorldCrafter only for academic purposes, and refrain from using it for any commercial or production purposes." From ARC Lab, Tencent IEG — Wangbo Yu and Kunhao Liu contributed equally, Wenbo Hu is the corresponding author and Ying Shan the senior author — with three co-authors at Peking University. The date marks the arXiv v1 posting; the GitHub repository went up on 14 September 2026.
Model Details
Benchmark Scores
| Benchmark | Score | Mode |
|---|---|---|
| Revisit LPIPS (vs. Lyra 2.0's 0.487) | 0.255 | WorldCrafter |
| Revisit PSNR (vs. Lyra 2.0's 14.050) | 18.016 dB | WorldCrafter |
| VBench overall (custom-input mode) | 81.910 | WorldCrafter |
Variants
| Name | Parameters | Notes |
|---|---|---|
| WorldCrafter-Base | — | Full-step base model; best camera-control accuracy and best VBench overall score |
| WorldCrafter-Fast | — | Pyramid-distilled high-noise/low-noise pair; best revisit consistency and 16 fps streaming generation on a 4-GPU machine |