Nanbeige4-3B
modelYour notes
Nanbeige's first small-model generation and the start of its open Apache 2.0 line. The report's title is "Exploring the Frontier of Small Language Models". It is a dense, Llama-style model trained from scratch: 32 layers, hidden size 2560, 20 query heads and 4 KV heads (GQA). It has a 166K-token vocabulary and 3.93B parameters including embeddings.
Pretraining covers a 23T-token corpus. It uses a fine-grained warmup-stable-decay (FG-WSD) schedule:
- 0.1T warmup;
- 12.4T and 6.5T stable stages;
- 4T of decay and long-context training, extending the context to 64K through ABF.
Post-training uses more than 30 million instructions. It then distills from the lab's larger "flagship reasoning model" with Dual Preference Distillation (DPD), followed by RL.
Two reasoning checkpoints were released:
- Thinking-2510 (October 22, 2025): self-reported AIME24 87.5, AIME25 81.7 and GPQA 77.2.
- Thinking-2511 (November 20): self-reported GPQA 82.2, Arena-Hard-v2 60.0 and BFCL-V4 53.8.
The base weights followed on October 31, and the report on December 6. Its base model was later reused for Nanbeige4.1-3B, and Thinking-2511 underlies the ToolMind-Web-3B search agent.
Model Details
Benchmark Scores
| Benchmark | Score | Mode |
|---|---|---|
| GPQA | 82.2 | Thinking-2511 |
| Arena-Hard-v2 | 60.0 | Thinking-2511 |
| AIME 2025 | 81.7 | Thinking-2510 |
Variants
| Name | Parameters | Notes |
|---|---|---|
| Nanbeige4-3B-Base | — | Base model, 32K context |
| Nanbeige4-3B-Thinking-2510 | — | First reasoning checkpoint |
| Nanbeige4-3B-Thinking-2511 | — | Updated reasoning checkpoint, 64K context |