Fantastic Pretraining Optimizers

"Fantastic Pretraining Optimizers and Where to Find Them" (Wen, Hall, Ma, Liang): a controlled re-evaluation of 11 optimizers (Adam, Muon, Soap, ...) run inside the Marin lab, deflating many claimed speedups to <1.4× at scale. Companion: "Pre-training under infinite compute" (2509.14786) — 5.2× data-efficiency from regularization + ensembling when compute is abundant. The pretraining-science pair of 2025.