A new base trained from scratch on 28T tokens as a looped transformer. The report is titled "Unlocking Agentic Capabilities in a Compact Model". Its 22 physical layers run twice per forward pass, giving 44 effective layers, with hidden size 3072, 48 query heads and 8 KV heads. The model has 4.17B parameters in total, or 3B without embeddings. The report says it chose looping from scratch over upcycling an existing model, and a full two-pass loop over sharing the KV cache across loops. The instruct model has a 256K context.

Self-reported results:

  • Reasoning: GPQA-Diamond 87.4 and HMMT Feb 2026 82.8.
  • Agentic coding: SWE-bench Verified 63.6, SWE-bench Pro 46.9 and Terminal-Bench 2.0 44.1.
  • Other: GDPval rubrics 74.3 and AA-LCR 58.7.

Artificial Analysis's August 2026 study of models that fit on a phone scored it 63 on its mobile-device benchmark set at 16K context, tied for first. That set is separate from the Intelligence Index.

The release added:

  • FP8 and GPTQ-Int8 checkpoints.
  • Nanbeige forks of vLLM, SGLang, llama.cpp and Ollama to serve the looped architecture.
  • DSpark (August 31): an 848M speculative-decoding draft model that reads hidden states from five target layers and drafts seven tokens at a time.

The card says its modeling code already contains LoopSplit, mHC with depth attention and concatenated n-gram embeddings. These are features meant for Nanbeige4.5, which was in training for release later in 2026.

Model Details

Architecture DENSE
Parameters 4B
Context window 262,144
Training tokens 28T
License Apache 2.0

Benchmark Scores

Benchmark Score Mode
GPQA Diamond 87.4 —
SWE-bench Verified 63.6 —
SWE-bench Pro 46.9 —
Terminal-Bench 2.0 44.1 —
AA mobile-device benchmark set (16K) 63 —

Variants

Name Parameters Notes
Nanbeige4.2-3B-Base — Base model, 128K context
Nanbeige4.2-3B — Instruct/reasoning model, 256K context; FP8 and GPTQ-Int8 checkpoints
Nanbeige4.2-3B-DSpark — 848M speculative-decoding draft model, 7-token blocks (Aug 31)

Paper

small-modelslooped-transformeragentscodingopen-weight

Related