K2 Horizon
modelYour notes
K2 Horizon is a fleet of six fully open models from MBZUAI's Institute of Foundation Models (IFM), released September 3, 2026: 375B-A23B, MoVA 36B-A4B, 32B, 7B, 3.7B, and 0.9B, all under Apache 2.0 with a shared architecture, a 250K-token vocabulary, and a native 524,288-token context from mid-training onward (the 0.9B uses a 64K vocabulary and 128K context). Each model is pretrained on roughly 20 trillion tokens (the 3.7B, 7B, 32B, and 36B-A4B on an identical 22T-token stream); nearly 17% of the corpus is problem-solving trajectories with explicit reasoning, and about 10T tokens are synthetic. IFM commits to releasing intermediate checkpoints, training data or construction recipes, mixture compositions, the xLLM training stack, post-training and RL code, fine-grained logs, and evaluation results, extending the LLM360 "360-open" line behind K2-V2 and K2 Think V2. At launch the final weights are public and the checkpoints, data, and code are marked as released from August 24 as TxT360-v2 and the Code-, Math-, and SFT-Reasoning and Pretrain-Behaviors sets; training hardware and cost are not disclosed.
The flagship K2-Horizon-375B-A23B is a sparse MoE with 375B total and about 23B active parameters: 61 layers, hidden size 6,144, 48 query and 8 KV heads, 192 routed experts with top-8 sigmoid routing plus one shared expert, and dense MLPs in the first three layers. Post-training chains mid-training, SFT, model merging, and RL with agent-specialized branches over more than 100 million synthesized tasks, yielding a "development tree" of reasoning and agentic variants rather than a single chat model. At high reasoning effort it scores Terminal-Bench 2.1 70.2, GPQA Diamond 87.3, Humanity's Last Exam 32.0, SWE-Bench Pro 42.6, BrowseComp 72.8, Toolathlon 65.3, MCPMark 67.7, and a GDPval-AA Elo of 1,441, ahead of Nemotron 3 Ultra (550B-A55B) on every listed agentic benchmark and trailing GLM 5.2 (753B-A40B) on most rows while leading it on Toolathlon. IFM audited the Terminal-Bench run for reward hacking with Artificial Analysis's procedure: 24 of 500 passing trials were flagged, which lowers the score from 70.2 to 66.9. On the Artificial Analysis Intelligence Index v5.0 it scores 18 (31 on v4.3.2, then 20 points above K2 Think V2's 11; 34 when v4.3 launched on 8 Sep 2026; 38 on v4.2; 47 on v4.1.1 at release).
Two architectural additions ship with the fleet. MoVA (Mixture-of-Value Attention) brings expert routing into multi-head attention the way MoE sparsifies feed-forward layers; the 36B-A4B model pairs 100 feed-forward experts (top-8) with 64 attention-value experts (top-4) to land just below the dense 32B at 4B active parameters. Uno is a conditional LoRA adapter trained by "diffusion distillation" to emit blocks of tokens in parallel while the frozen autoregressive weights keep the output distribution, a lossless decode speedup that IFM's release puts at roughly 3×; adapters ship for the 7B and 0.9B models. Serving recipes exist for vLLM and SGLang (BF16, TP=8 on one 8×H200 node for the 375B), with FP8 and GGUF checkpoints for most sizes. NVFP4 checkpoints of the 375B-A23B and 32B followed on September 22 (routed experts in NVFP4, all other layers in BF16; Blackwell-generation GPUs required).
Through September IFM kept opening the stack behind the release: the post-training code (renamed RL360), attention kernels for xLLM (xattn, with FP32 attention-output state for the backward pass), checkpoint conversion and vLLM serving (xbridges), a pretraining-data toolkit (model-based annotation, quality bucketing, distributed shuffling), a quality gate that every mid- and post-training conversation had to pass (process_entry), and long-context QA and agent-task synthesis pipelines (LCQA, task-factory). Each repo has only a handful of stars so far, and the pretraining and mid-training datasets were still gated at the end of September.
Model Details
Benchmark Scores
| Benchmark | Score | Mode |
|---|---|---|
| Terminal-Bench 2.1 | 70.2 | high reasoning effort |
| GPQA Diamond | 87.3 | high reasoning effort |
| Humanity's Last Exam (no tools) | 32.0 | high reasoning effort |
| SWE-Bench Pro (strict) | 42.6 | high reasoning effort |
| BrowseComp | 72.8 | high reasoning effort |
| Toolathlon Verified | 65.3 | high reasoning effort |
| MCPMark | 67.7 | high reasoning effort |
| tau3-Banking | 34.0 | high reasoning effort |
| SciCode | 42.7 | high reasoning effort |
| AA-LCR | 76.0 | high reasoning effort |
| GDPval-AA | 1441 Elo | high reasoning effort |
Variants
| Name | Parameters | Notes |
|---|---|---|
| K2-Horizon-375B-A23B | 375B | Flagship MoE (61 layers, 192 routed + 1 shared expert, top-8). AA Intelligence Index v5.0 score 18 (31 on v4.3.2; file's top-level anchor). FP8 checkpoint also released. |
| K2-Horizon-MoVA-36B-A4B | 36B | 48 layers; 100 FFN experts (top-8) plus MoVA with 64 attention-value experts (top-4). Terminal-Bench 2.1 58.6, GPQA Diamond 80.8, HLE 25.2, tau3-Banking 26.8. GGUF released. |
| K2-Horizon-32B | 32B | Dense, 64 layers, hidden 5,120. Hugging Face hosts the Stage-1 checkpoint at launch; the final checkpoint is marked as forthcoming. FP8 and GGUF released. |
| K2-Horizon-7B | 7B | IFM claims the best-performing model under 10B. K2-Horizon-7B-Uno diffusion-distillation LoRA adapter, FP8, and GGUF released. |
| K2-Horizon-3.7B | 3.7B | IFM claims the best reasoning under 4B. GGUF released. |
| K2-Horizon-0.9B | 0.9B | Dense, 28 layers, 64K vocabulary, 128K context; AIME 2026 above 48 per IFM. Uno adapter and GGUF released. |