From MBZUAI-IFM and USC-ISI, on a gap between what sparse MoE promises and what it costs. Per-token FLOPs depend only on the k active experts, not the total pool E, so adding experts is supposed to be free. Wall-clock is not: all-to-all communication, aggregate memory for parameters, gradients and optimizer states, and GPU underutilization from small per-expert GEMMs and routing kernels all grow with E. The paper measures it — at 1.1B activated parameters, going from 8 experts to 128 inflates step time by 1.08×; at 4B activated, the same expansion costs 1.72×, and the gap is expected to widen at larger activated sizes. They name this the MoE efficiency paradox.

EMO's argument is that the waste is front-loaded: early-stage data cannot use a large expert pool, so training should not pay for one from step zero. It treats MoE capacity as expandable memory and grows the pool during training, with sparsity carried into the scaling law to derive stage-wise compute-optimal token budgets rather than a hand-set schedule. The fitted allocation puts 23.5%, 9.5%, 16.5%, 21.8% and 28.6% of the budget into the E=8, 16, 32, 64 and 128 stages — enough tokens early to build representations, then the majority late where marginal returns are largest. Each stage resumes from the last checkpoint; a warm-stable-decay schedule with a 500-step warmup per intermediate stage keeps expansion from introducing any discontinuity, and the learning rate decays only in the final stage. An ablation on the 16→32 boundary shows why the derived point is the right one: expanding at 25% of the budget gives loss 1.069, at 50% 1.071 and at 75% 1.076 — nearly flat from 25–50%, then steepening — and the scaling law lands at ~45%, inside the flat region.

The scale test grows a 1.1B dense model into a 9.6B-A1.1B MoE with E=128 over five stages on 1.92T tokens (8→16→32→64→128, fixed top-8 routing), on 32 H200s under a deliberately optimized stack: BF16, FlashAttention 3, fused transformer-block kernels, selective activation recomputation, best-fit-decreasing sequence packing with document masking, and entropy-based load balancing. Final pretraining loss is 1.017 against 0.994 for a fixed-E=128 baseline on the same token budget — a 2.3% relative gap — and 1.065 for fixed-E=64, while saving 10% of GPU hours. Across eight downstream benchmarks it clearly beats the smaller fixed pools and is competitive with the E=128 ceiling, though the paper reports that fixed-E=128 stays ahead on GSM8K at the final checkpoint, suggesting reasoning-heavy tasks may want the full pool exposed throughout. Each expansion produces a transient loss spike that recovers within roughly 10K steps, and the method is robust to how new experts and router weights are initialized — the choice moves the spike, not the final loss. Co-authored by MBZUAI President Eric Xing.

Paper

Authors: Linghao Jin · Chufan Shi · Huijuan Wang · Nuan Wen · Zhengzhong Liu · Eric Xing · Xuezhe Ma
moeefficiencypretrainingscaling-lawsresearch

Related