MoonEP
libraryYour notes
The expert-parallel communication library behind Kimi K3's MoE pretraining, which removes routing imbalance rather than tolerating it. Every rank receives exactly S×K tokens however skewed the router is, so all ranks compute the same amount; the balance comes from a small number of dynamic redundant experts planned online from the current router outputs and prefetched before expert computation, with their gradients reduced back to their home ranks in the backward pass. The K3 report proves the result that makes this safe: a balanced plan always exists with at most E/R redundant experts per rank (E experts, R EP ranks), and that bound is essentially tight. Reserving E/R slots therefore guarantees a feasible plan at every step, so training is never interrupted — unlike prior schemes such as ECHO and UltraEP, which preset a redundant-expert count or a per-rank token cap, stall when no plan fits the cap, and need the cap tuned by hand while still leaving residual imbalance. Exact optima are computed offline with integer linear programming as references for a GPU planning kernel that is near-optimal, costs almost nothing, and always respects the bound.
Perfect balance also simplifies the data path. A fused permute/unpermute operator sends tokens straight to their expert-grouped positions on remote ranks, with the planner precomputing every destination, and hands views of the communication buffer back to the computation — so the communication-to-user-buffer copy that dominates the epilogue disappears, and a fixed S×K buffer suffices. Statically known shapes remove per-layer MoE host synchronization. Benchmarks on H20 at EP=8, sweeping router imbalance, put MoonEP's communication time below DeepEP v2 at every imbalance level and nearly flat as imbalance grows, where DeepEP degrades because its latency is set by the hottest rank; total dispatch time including MoonEP's extra planning and prefetch kernels matches DeepEP v2's dispatch alone, and combine is faster throughout. End to end, DeepEP's iteration time climbs with imbalance and eventually runs out of memory as shifting activation shapes fragment the allocator, while MoonEP's iteration time stays flat and its static shapes never fragment. MIT-licensed, NVIDIA GPUs today with Zhenwu PPU support listed as coming; about 1,160 GitHub stars.