The interesting number in this release is 4.5 trillion. Dory 1.0 was, per the model card, "pre-trained on 6T tokens using NVFP4 for the first ~4.5T tokens and BF16 for the remainder," followed by "300 billion tokens of context extension" to a 256K window. That makes roughly three quarters of a 6T-token pretraining run executed natively in a 4-bit floating-point format — the largest such run NVIDIA has described, and the practical follow-through on the low-precision-pretraining line that papers like FP4 All the Way and Huawei's HiFloat4 argued for. The card is worth reading for what it does not contain: no loss or accuracy comparison against an equivalent BF16 run, and no statement of the training hardware. NVIDIA names Megatron-LM as the pretraining stack and leaves the cost saving implied rather than measured, so the headline result is "a 4-bit-pretrained model that works," not a quantified NVFP4-vs-BF16 gap.

Architecture: a looped model from a frontier lab. Dory is a "Looped (latent-recurrent) normalized-GPT with interleaved sliding-window attention (SWA) and full-attention layers" — 8 input layers, a shared 20-layer recurrent block applied repeatedly with the input reinjected at each pass, and 8 output layers, for 3.98B stored parameters. Loop count is a deployment knob set to 1, 2, 3 or 4 (NVIDIA recommends 2 by default, 1 to minimize compute, 4 when accuracy matters), which makes the same weights an elastic compute–accuracy dial. The usual objection to recurrent-depth models is KV memory; Dory answers it with "last-loop KV caching, which keeps a single cache per recurrent attention layer," so KV-cache footprint matches an equivalent non-looped model regardless of loop count. It is the first production-weight entry in the looped-transformer line this index has tracked mostly as papers — ByteDance's Ouro and Parallel Loop Transformer, Google's Reasoning with Latent Thoughts, and MBZUAI's Looped Models Done Right, Part II.

Scores scale with loops, unevenly. All figures are NVIDIA-measured, 1/2/4 loops: SWE-bench Verified 50.3 / 59.5 / 61.8, Terminal-Bench 2.1 12.5 / 18.1 / 22.2, SciCode 27.3 / 34.9 / 35.1, IFBench 74.3 / 77.9 / 79.0, AA-LCR 63 / 71 / 73. Agentic and long-context tasks gain the most from extra loops; knowledge-bound ones barely move or regress (GPQA Diamond 69.2 / 72.4 / 72.2, AA-Omniscience 10.0 / 10.1 / 10.0), which is the expected signature of added test-time depth rather than added knowledge. Against the card's own comparison set, 4-loop Dory beats Qwen3.5-4B and Qwen3.5-9B on SWE-bench Verified and IFBench while trailing both on GPQA Diamond and AA-Omniscience. Post-training is SFT on an NVIDIA-curated dataset plus multi-environment GRPO over math, code, science, instruction following and multi-step tool use, using an asynchronous RL setup in NeMo RL and NeMo Gym.

Weights ship in BF16, MXFP8 (W8A16) and NVFP4 (W4A16) plus a GGUF Q5_K_M export for llama.cpp, under the OpenMDW License Agreement v1.1 and marked "ready for commercial or non-commercial use." No paper or technical report at filing, and no Artificial Analysis entry — the AA v4.3.2 figures on the card are NVIDIA's own internal estimates, not AA measurements.

Model Details

Architecture DENSE
Parameters 4B
Context window 262,144
Training tokens 6T
License OpenMDW v1.1

Benchmark Scores

Benchmark Score Mode
SWE-bench Verified (4 loops) 61.8 —
Terminal-Bench 2.1 (4 loops) 22.2 —
GPQA Diamond (4 loops, no tools) 72.2 —
IFBench (4 loops) 79.0 —
SciCode (4 loops) 35.1 —
AA-LCR (4 loops) 73 —

Variants

Name Parameters Notes
NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 4B reference checkpoint, W16A16; 3,975,229,440 stored parameters
NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 — W8A16 quantized export
NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4 — W4A16 quantized export
NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF — Q5_K_M export for llama.cpp
open-weightefficiencyreasoningagentsquantizationarchitecture

Related