Nanbeige's first small-model generation and the start of its open Apache 2.0 line. The report's title is "Exploring the Frontier of Small Language Models". It is a dense, Llama-style model trained from scratch: 32 layers, hidden size 2560, 20 query heads and 4 KV heads (GQA). It has a 166K-token vocabulary and 3.93B parameters including embeddings.

Pretraining covers a 23T-token corpus. It uses a fine-grained warmup-stable-decay (FG-WSD) schedule:

  • 0.1T warmup;
  • 12.4T and 6.5T stable stages;
  • 4T of decay and long-context training, extending the context to 64K through ABF.

Post-training uses more than 30 million instructions. It then distills from the lab's larger "flagship reasoning model" with Dual Preference Distillation (DPD), followed by RL.

Two reasoning checkpoints were released:

  • Thinking-2510 (October 22, 2025): self-reported AIME24 87.5, AIME25 81.7 and GPQA 77.2.
  • Thinking-2511 (November 20): self-reported GPQA 82.2, Arena-Hard-v2 60.0 and BFCL-V4 53.8.

The base weights followed on October 31, and the report on December 6. Its base model was later reused for Nanbeige4.1-3B, and Thinking-2511 underlies the ToolMind-Web-3B search agent.

Model Details

Architecture DENSE
Parameters 4B
Context window 65,536
Training tokens 23T
License Apache 2.0

Benchmark Scores

Benchmark Score Mode
GPQA 82.2 Thinking-2511
Arena-Hard-v2 60.0 Thinking-2511
AIME 2025 81.7 Thinking-2510

Variants

Name Parameters Notes
Nanbeige4-3B-Base — Base model, 32K context
Nanbeige4-3B-Thinking-2510 — First reasoning checkpoint
Nanbeige4-3B-Thinking-2511 — Updated reasoning checkpoint, 64K context

Paper

small-modelsreasoningopen-weight

Related