The Sparse Frontier

"The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs" — the largest empirical study of training-free sparse attention (6 methods, contexts up to 128K), showing that large-sparse models beat small-dense ones on the compute Pareto frontier. Adopted as an evaluation framework by follow-on work. Edinburgh-led (Nawrot, Ponti) with Cohere and Meta coauthors.