Dynamic MoE — letting easy tokens activate fewer experts — is an old idea with a practical gap: the existing methods want pre-training from scratch or task-specific adaptation, which leaves the case that actually matters untouched. You already have a fully post-trained static MoE and you want its inference bill cut. Zero-Expert Self-Distillation Adaptation (ZEDA) converts one into the other cheaply, and the trick is that it changes nothing about how many experts the router picks. It injects parameterless zero experts whose output is identically zero into each layer's existing pool, so the router's candidate set grows with zero-computation options while the activation count stays fixed — the number of active real experts falls out of that on its own.

The augmented model is then adapted by two-stage self-distillation — SFT then on-policy distillation — against the original MoE as a frozen teacher, so the target is the model's own prior behaviour rather than an external one. Stability is the hard part of this conversion, and ZEDA adds a Group Auxiliary Loss that regulates the relative activation frequency of normal versus zero experts, keeping the original routing distribution from being disrupted. Adaptation is genuinely low-cost: under 31 hours for Qwen3-30B-A3B and 62 for GLM-4.7-Flash on 8 NVIDIA H200s.

Across 11 benchmarks spanning maths, code and instruction following, it removes over half the expert FLOPs for a marginal accuracy loss against the unmodified model, beating the strongest dynamic-MoE baseline by 6.1 points on Qwen and 4.0 on GLM, and turning the saved compute into roughly a 1.20× end-to-end speedup — a gap between FLOPs removed and wall-clock gained that the paper is upfront about, listing speedup decay at long sequence lengths among its limitations, along with no larger MoE deployments and no long-horizon agentic tasks. The analysis is the part worth keeping: how much compute a token gets correlates with the teacher–student distribution gap, model uncertainty and response patterns, not with overall task difficulty — so "easy tokens" turns out to mean tokens the model is already confident about, not tokens from easy problems. The zero-expert construction is borrowed from LongCat-Flash and MoE++, applied here as a post-hoc conversion rather than a pre-training choice. By Tsinghua University and Shanghai AI Lab, with Frontis.AI, Kuaishou and WeChat AI (lead Xingtai Lv; senior authors Ning Ding and Bowen Zhou); code released as TsinghuaC3I/ZEDA under MIT.

Paper

Authors: Xingtai Lv · Li Sheng · Kaiyan Zhang · Yichen You · Siyan Gao · Xueheng Luo · Yuxin Zuo · Yuchen Fan · Junlin Yang · Ganqu Cui · Bingning Wang · Fan Yang · Youbang Sun · Ning Ding · Bowen Zhou
moeefficiencydistillationpost-traininginferenceresearch

Related