A long-context training system for learned sparse attention, the DeepSeek-V3.2-style design in which a lightweight indexer scores every earlier token and attention reads only the top-k. At million-token lengths, the selection step becomes the training bottleneck, not the attention it feeds. The indexer scores all previous tokens while attention reads only k of them, current systems materialize the full score table, and activation recomputation repeats the selection. The paper's observation is that the output of selection is a set of indices, not an activation. Two components follow from it:

  • FlashTopK: a fused kernel that finds each query's exact top-k on the fly without materializing a score table. It is deterministic and compatible with context parallelism.
  • Selection replay: reuses the forward pass's indices during recomputation instead of selecting again.

Results:

  • End to end: training GLM-5.1 on 1M-token sequences runs 2.07× faster than Megatron-LM on 512 H20 GPUs.
  • Other sparse-attention layers, at 1M tokens on 32 H800 GPUs: 2.01× on the CSA layers of DeepSeek-V4-Flash and 1.57× on the QSA layers of Qwen3.8-Flash-Next.
  • FlashTopK alone: its README reports full-prefill selection 1.7–3.3× faster than dense scoring plus FlashInfer top-k on trained indexers, and 1.2–2.0× on untrained ones.

The authors are from the University of Edinburgh and Tencent Hunyuan, with Luo Mai as senior author. The paper was first submitted on October 23 and revised to v3 on November 4. FlashTopK was released as open source on November 5 under Tencent-Hunyuan. The paper uses other labs' models, and it does not say whether Tencent's own Hy4 was trained with it.

Paper

Authors: Yangshen Deng · Xinjing Huang · Yulong Wang · Yunzhuo Liu · Jun Hu · Shuyong Tan · Chun-Chih Tseng · Mingji Han · Ray Ying · Key Zhang · Luo Mai
attentionlong-contextinfrastructuregpu-kernelsefficiencyresearch

Related