Benchmarking Optimizers for LLM Pretraining

A systematic, controlled comparison of LLM-pretraining optimizers — AdamW, Muon, Scion, SOAP, Sophia and others — under matched budgets and careful tuning, making it a go-to reference in the "optimizer wars" around Muon-style geometry-aware methods. From Martin Jaggi's MLO lab. General training science (non-SNAI-branded), though the same group's data/optimizer work feeds the Swiss AI Apertus pipeline.