Olmo-core 3
libraryYour notes
The usual case for mixture-of-experts is that per-token FLOPs depend only on the active experts, so total capacity is nearly free. The technical report's framing is that this is an accounting error: "training cost depends not only on active compute but also on many other factors, including bytes moved, memory held, and kernels launched; many of those costs follow total parameter capacity rather than active compute." On infrastructure built for dense models the mismatch bites hard — in Ai2's own early baselines throughput fell as stored expert capacity grew even though useful work per token stayed nearly fixed. Olmo-core's previous MoE path used FSDP, gathering and resharding model weights for every micro-batch of data.
Version 3.0.0 replaces that with a stack built on DDP: experts stay resident on the GPUs and the data is routed to them, so the repeated weight gathering disappears. Layered on top are expert parallelism, pipeline parallelism, a distributed optimizer and activation recompute. The MoE-specific optimizations are where most of the engineering sits: NVSHMEM-based rowwise expert parallelism writes each token straight into its expert's input buffer, removing permutation overhead from dispatch and combine; expert parallelism is made fully synchronization-free by keeping routing metadata on the GPU and using device-scheduled grouped GEMMs; MXFP8 is integrated closely enough to cut compute, activation memory and transport together; and checkpoints are topology-agnostic, so the parallel layout can change between runs.
Reported throughput, all on NVIDIA B300s. A 47B MoE on eight GPUs runs at 52,000 tok/s/GPU against 19,400 on the earlier FSDP implementation — about 2.7×. Growing the expert pool from 8 to 128 while still selecting four experts per token holds active parameters at ~3.2B and raises total capacity from 4.6B to 47B for under 5% throughput loss. In a controlled four-GPU benchmark with work spread uniformly across experts, MXFP8 beats BF16 by about 21% while peak active memory falls from 103 GiB to 95 GiB, with most of the gain coming from feed-forward compute and inter-expert data movement rather than attention. The stack has been benchmarked from 12.9B up to a 1.2T-parameter configuration with 58.36B active per token on 512 GPUs, peaking at 858 useful-model TFLOP/s/GPU; an experimental run with the optional DeepEP v2 backend reached 2.38T. Both headline figures come with caveats Ai2 states plainly: these tests "used random routing to measure system performance, rather than the quality of a trained model," and the 2.38T configuration was "a short-capacity test rather than a full training run."
The report is unusually generous with things that did not work. It names token gerrymandering — a load-balancing score that improves while the actual workload becomes less balanced, i.e. the router learning to hack its own auxiliary loss. Lowering experts' learning rates to compensate for seeing fewer tokens did not help in the model family tested. GPU kernels took different amounts of time on the same matrix shapes with different values, so comparisons need matching inputs, not just matching dimensions. And overlapping communication with computation on separate streams sometimes slowed end-to-end execution by more than it hid. v3.0.0 is also a breaking release: the olmo_core.model_ladder API and its eleven ladder training scripts are removed and the minimum PyTorch is raised to 2.10. Ai2 frames the stack as the foundation for a next-generation MoE Olmo, extending sparse work that goes back to OLMoE after the dense Olmo 3. Apache 2.0; ~1,704 stars. Note that OlmPool also ships Olmo-core-format checkpoints but is a separate long-context study, not part of this release.
Paper
Library
pip install ai2-olmo-core