Knowledge Entropy Decay

ICLR 2025 Oral: defines "knowledge entropy" — how broadly a model integrates its parametric memory sources — and shows it decays monotonically during pretraining, mechanistically explaining why late-stage models acquire new knowledge poorly and forget more; resuscitating inactive memory vectors restores acquisition. Pretraining science from the LK Lab (Minjoon Seo) that informs continual-pretraining and mid-training recipes.