Loss Deceleration & Zero-Sum Learning

"Loss Deceleration & Zero-Sum Learning" gives a mechanistic account of why scaling-law loss curves bend — loss improvement decelerates when learning on some examples comes at the expense of others (zero-sum learning) — from Irina Rish's lab. A companion result, "Beyond Cosine Decay: an Infinite Learning-Rate Schedule for continual pre-training" (arXiv 2503.02844, CoLLAs 2025), introduces a learning-rate schedule that enables stable continual pretraining of foundation models.