Goldfish Loss

"Be Like a Goldfish, Don't Memorize!" (NeurIPS 2024): a memorization-mitigating pretraining loss that pseudo-randomly drops a subset of tokens from the loss computation, preventing verbatim reproduction of training sequences with negligible impact on downstream benchmarks. Edge-of-window work from Tom Goldstein's group on copyright- and privacy-safer LLM pretraining.