Marin's 67B-total, roughly 2B-active mixture-of-experts model, built in Marin's own "Grug MoE" architecture and pretrained from scratch on 10T tokens on TPU v4 by the Marin team at Open Athena. It has 26 layers, a hidden size of 2,560, grouped-query attention with 20 query and 5 key-value heads, and 256 routed experts with four selected per token plus a shared expert, over a 128,256-token vocabulary; router biases from Marin's quantile-balancing scheme are folded into the exported weights.

Context was then extended to 262,144 tokens in a 1,000-step run (steps 156,000 to 157,000), and on September 9, 2026 Open Athena published five BF16 base checkpoints from that comparison on HuggingFace under the OpenMDW-1.1 license: two attention-scaling settings (QK multiplier 1.57 and 1.75) and three that upsample long documents 2×, 4× or 8×. They are base models with no instruction tuning, and serving them needs Marin's vLLM fork, which registers the GrugMoE architecture. Supervised fine-tuning at 262K context and RLVR runs on top of Snowball followed as further public experimental checkpoints in September; the team also built an end-to-end post-training pipeline for it. Snowball sits between the dense Marin 32B and the 535B-total, 23B-active MoE that Marin began pretraining in August 2026.

The first named post-trained release, Marin Snowball 67B-A2B v0.1, appeared on October 19. The card labels it a "research artifact" that "has not been properly tested or evaluated." It starts from a DARE-TIES merge of three Grug-67B-A2B / Snowball RLVR checkpoints, then runs 14 updates of asynchronous RLOO-N (a leave-one-out policy gradient with truncated importance sampling) on 64 H100s for policy training plus 48 for inference. The 69,487 training tasks are 12,123 NVIDIA Nemotron tasks and 57,364 "structural closure" tasks, and both the task files and the configs are public. On the team's own development validation set, which it has inspected repeatedly, combined pass@1 rose from 41.2% to 50.2%: NVIDIA-task reward improved and structural reward fell from 39.1 to 32.9. The card states plainly that its inline BFCL-Parity, SWE-bench Verified and OT-TBLite runs "encountered endpoint errors and timeouts. They do not establish valid benchmark performance." v0.1.1 (October 22) adds a low-learning-rate identity SFT. It raises the 60-question identity panel from 7/60 to 45/60 but regresses MATH500 from 461 to 449 of 500 and GSM8K from 1,224 to 1,038 of 1,319, and the card calls it "not a regression-free replacement." Both are OpenMDW-1.1 and need Marin's vLLM build.

Open Athena's October 29 post, "Post-Training in the Open" by Ben Feuer, presents v0.1 as Marin's first model to get serious post-training. Its lineage is a graph of 14 models and 15 connections, out of more than 100 post-trained Snowball models the team tracks, using four techniques: SFT, RLVR, final-token preference optimization (FTPO) and model merging. The in-house stack builds on SkyRL and Harbor. Ranked on Marin's own eval policy against its peers, v0.1 has a mean rank of 8.13 against GPT-OSS's 8.93, a gap well inside the 4.14 critical difference. With a FLOP-controlled adjustment it ranks first (5.18 against GPT-OSS-20B's 8.29), though not significantly ahead of the second-best model. Open Athena calls that "GPT-OSS-level quality", nine to fifteen months behind the open-weights frontier at this size. It names agentic skill, factuality and complex instruction following as the main gaps. The post also says snowball-base beats Marin 32B on Marin 32B's own launch eval card with 10× fewer FLOPs and 16× fewer active parameters. Open Athena published a 4-bit MLX build for Macs with 32GB or more of memory, and says a full technical report will follow in November.

Model Details

Architecture MOE
Parameters 67B
Active params 2B
Experts 256 (top-4)
Context window 262,144
Training tokens 10T
Training hardware TPU v4
License OpenMDW-1.1

Variants

Name Parameters Notes
snowball-67b-a2b-base-262k-qk157 — Naive long-context extension, QK multiplier 1.57
snowball-67b-a2b-base-262k-qk175 — Raised attention scale for 262K context (QK multiplier 1.75)
snowball-67b-a2b-base-262k-qk175-skew2 / skew4 / skew8 — QK 1.75 with long-context documents upsampled 2x, 4x or 8x during the extension
Marin-Snowball-67B-A2B-v0.1 — 2026-10-19; RLOO-N RL (14 steps) from a DARE-TIES merge of Grug-67B-A2B/Snowball RLVR checkpoints; native dev-set pass@1 41.2 -> 50.2; card: external benchmark runs hit endpoint errors and do not establish valid benchmark performance
Marin-Snowball-67B-A2B-v0.1-MLX-4bit — 2026-10-20; 4-bit MLX quantization of v0.1 for Apple silicon (32GB+); runs with mlx-lm
Marin-Snowball-67B-A2B-v0.1.1 — 2026-10-22; identity SFT on v0.1 (LR 1e-6, step 50); identity panel 7/60 -> 45/60; MATH500 461 -> 449/500, GSM8K 1224 -> 1038/1319 (regressions)
moeopen-weightpretraininglong-context

Related