GAE (Geometry-Native Autoencoder)
modelYour notes
A geometry-native latent space for video and 3D generation: instead of adding geometry as an extra output head, GAE reparameterizes a frozen geometry foundation model's features into a compact latent that is jointly decodable to appearance, depth, cameras and point maps. Stage 1 freezes Depth Anything 3 (DA3-GIANT) at both ends — its encoder emits a four-level feature hierarchy and its DPT head reads depth, rays and point maps back out — and trains a codec (GAECodec) that compresses the 3,072-channel feature stack to 64 or 128 channels, with a learned RGB head decoding appearance from the same state. Because the geometry head never trains, the bottleneck has to stay readable by the model the features came from. Two alignment losses shape the latent before any generative training: L_tok aligns each projected posterior token with C-RADIOv2.5-B, and L_struct matches DINOv2-L patch-patch similarity directly in the raw posterior mean — unlike REPA, which aligns intermediate denoiser states during generative training. Stage 2 freezes the codec and trains one conditional flow model over the standardized latent, conditioned on clean reference latents prepended at t=0, metric Plücker ray maps for scale-aware camera control, and text via cross-attention from a frozen Qwen3-0.6B. The same backbone covers text-to-image, camera-controlled video, and reference-conditioned novel-view synthesis.
The paper's central claim is tested in a controlled comparison that holds the flow architecture, data mixture, optimization schedule and sampling protocol fixed and swaps only the latent (SD-VAE, WAN2.1 video VAE, semantic RAEv2, raw DA3-GIANT L0 and L3, plus GLD and Gen3R). Against the best non-GAE controlled latent, GAE-64 cuts FVD by 12.7% on RealEstate10K (258.6 → 225.7) and by 23.1% on DL3DV (373.2 → 287.0), and leads the paired metrics (PSNR 20.02 / 18.00, LPIPS 0.143 / 0.194). On 3D coherence measured independently with VGGT — a backbone shared with none of the latents — GAE-64 roughly halves camera-trajectory error on RealEstate10K against the best non-GAE controlled latent (ATE 0.0072 → 0.0034) and posts the lowest MEt3R of the controlled set (0.1208, against 0.1263 for semantic RAEv2). On reconstruction, GAE-128 matches or slightly exceeds the raw DA3 L0 baseline (PSNR 28.76 single-view, 34.78 multi-view; rFID 5.4, rFVD 4.3) while using 24× fewer channels, and geometry decoded from GAE latents reaches AbsRel 0.090 / δ₁ 0.905 on RealEstate10K against a Pi3 reference.
The released checkpoint, TencentARC/GAE-D64-1B, is the GAE-64 configuration: a 0.93B-parameter flow transformer (28 encoder blocks of width 768, six decoder blocks of width 2048) over the 64-channel latent at 672×378 with 81-view clips, trained on 80 GPUs in BF16 with AdamW (base LR 5e-5, 500 warmup steps, cosine decay to 1e-6 by 60k steps, EMA 0.9995) on a mixture of RealEstate10K, DL3DV, ScanNet++, MVS-Synth, SpatialVID, OmniWorld and internally curated data; the parameter count covers the trainable Stage 2 generator and excludes the frozen DA3 encoder, codec and text encoder. The GAE-128 codec is evaluated in the paper but its weights are not on HuggingFace. Code and weights carry a custom Tencent license limited to academic use — GitHub reports no SPDX identifier, and LICENSE.txt grants MIT-style permissions but adds that users "agree to use [it] only for academic purposes, and refrain from using it for any non-academic, commercial or production purposes." From ARC Lab, Tencent IEG with HKUST, the University of Hong Kong and UT Austin: first author Jiahao Lu (HKUST) and Minghao Yin (Tencent ARC / HKU) contributed equally, Wenbo Hu (Tencent ARC) and Yuan Liu (HKUST) are corresponding authors, and Ying Shan is the senior author. The date marks the arXiv v1 posting (a v2 followed on 25 September 2026); all figures above are from v1.
Model Details
Benchmark Scores
| Benchmark | Score | Mode |
|---|---|---|
| FVD (RealEstate10K, controlled 9-view) | 225.7 | GAE-64 |
| FVD (DL3DV, controlled 9-view) | 287.0 | GAE-64 |
| VGGT ATE (RealEstate10K, generated views) | 0.0034 | GAE-64 |
Variants
| Name | Parameters | Notes |
|---|---|---|
| GAE-64 (GAE-D64-1B) | 0.93B | 64-channel geometry-native latent; the released checkpoint, at 672×378 with 81-view generation and Qwen3-0.6B text conditioning |
| GAE-128 | — | 128-channel latent; best RealEstate10K FID (25.6) and best codec reconstruction in the paper, but weights not released |