Nemotron-Labs Dory 1.0 4B
modelYour notes
The interesting number in this release is 4.5 trillion. Dory 1.0 was, per the model card, "pre-trained on 6T tokens using NVFP4 for the first ~4.5T tokens and BF16 for the remainder," followed by "300 billion tokens of context extension" to a 256K window. That makes roughly three quarters of a 6T-token pretraining run executed natively in a 4-bit floating-point format — the largest such run NVIDIA has described, and the practical follow-through on the low-precision-pretraining line that papers like FP4 All the Way and Huawei's HiFloat4 argued for. The card is worth reading for what it does not contain: no loss or accuracy comparison against an equivalent BF16 run, and no statement of the training hardware. NVIDIA names Megatron-LM as the pretraining stack and leaves the cost saving implied rather than measured, so the headline result is "a 4-bit-pretrained model that works," not a quantified NVFP4-vs-BF16 gap.
Architecture: a looped model from a frontier lab. Dory is a "Looped (latent-recurrent) normalized-GPT with interleaved sliding-window attention (SWA) and full-attention layers" — 8 input layers, a shared 20-layer recurrent block applied repeatedly with the input reinjected at each pass, and 8 output layers, for 3.98B stored parameters. Loop count is a deployment knob set to 1, 2, 3 or 4 (NVIDIA recommends 2 by default, 1 to minimize compute, 4 when accuracy matters), which makes the same weights an elastic compute–accuracy dial. The usual objection to recurrent-depth models is KV memory; Dory answers it with "last-loop KV caching, which keeps a single cache per recurrent attention layer," so KV-cache footprint matches an equivalent non-looped model regardless of loop count. It is the first production-weight entry in the looped-transformer line this index has tracked mostly as papers — ByteDance's Ouro and Parallel Loop Transformer, Google's Reasoning with Latent Thoughts, and MBZUAI's Looped Models Done Right, Part II.
Scores scale with loops, unevenly. All figures are NVIDIA-measured, 1/2/4 loops: SWE-bench Verified 50.3 / 59.5 / 61.8, Terminal-Bench 2.1 12.5 / 18.1 / 22.2, SciCode 27.3 / 34.9 / 35.1, IFBench 74.3 / 77.9 / 79.0, AA-LCR 63 / 71 / 73. Agentic and long-context tasks gain the most from extra loops; knowledge-bound ones barely move or regress (GPQA Diamond 69.2 / 72.4 / 72.2, AA-Omniscience 10.0 / 10.1 / 10.0), which is the expected signature of added test-time depth rather than added knowledge. Against the card's own comparison set, 4-loop Dory beats Qwen3.5-4B and Qwen3.5-9B on SWE-bench Verified and IFBench while trailing both on GPQA Diamond and AA-Omniscience. Post-training is SFT on an NVIDIA-curated dataset plus multi-environment GRPO over math, code, science, instruction following and multi-step tool use, using an asynchronous RL setup in NeMo RL and NeMo Gym.
Weights ship in BF16, MXFP8 (W8A16) and NVFP4 (W4A16) plus a GGUF Q5_K_M export for llama.cpp, under the OpenMDW License Agreement v1.1 and marked "ready for commercial or non-commercial use." No paper or technical report at filing, and no Artificial Analysis entry — the AA v4.3.2 figures on the card are NVIDIA's own internal estimates, not AA measurements.
Model Details
Benchmark Scores
| Benchmark | Score | Mode |
|---|---|---|
| SWE-bench Verified (4 loops) | 61.8 | — |
| Terminal-Bench 2.1 (4 loops) | 22.2 | — |
| GPQA Diamond (4 loops, no tools) | 72.2 | — |
| IFBench (4 loops) | 79.0 | — |
| SciCode (4 loops) | 35.1 | — |
| AA-LCR (4 loops) | 73 | — |
Variants
| Name | Parameters | Notes |
|---|---|---|
| NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 | 4B | reference checkpoint, W16A16; 3,975,229,440 stored parameters |
| NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 | — | W8A16 quantized export |
| NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4 | — | W4A16 quantized export |
| NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF | — | Q5_K_M export for llama.cpp |