Rho-1: Not All Tokens Are What You Need

Selective Language Modeling (SLM) — a pretraining method that abandons uniform next-token prediction and instead trains only on the tokens that matter. A reference model trained on high-quality data scores every pretraining token by its excess loss; the target model then applies its language-modeling loss only to the highest-scoring tokens, skipping noisy or already-learned ones. The paper argues that much of a web corpus is low-utility for a given objective and that filtering at the token level (not just the document level) is a strong efficiency lever. Introduced with the Rho-1 models (1B and 7B).

Continual pretraining on 15B OpenWebMath tokens yielded up to a 30% absolute few-shot gain across 9 math tasks; after fine-tuning, Rho-1-1B and Rho-1-7B reached 40.6% and 51.8% on MATH — matching DeepSeekMath while using only 3% of the pretraining tokens. On 80B general tokens SLM gave a 6.8% average lift across 15 diverse tasks, establishing token-level data selection as a durable technique for pretraining efficiency.

A collaboration across Xiamen University, Tsinghua University, Shanghai AI Laboratory, and Microsoft (senior authors Weizhu Chen and Nan Duan). NeurIPS 2024 Oral and Best Paper Runner-Up. Code and models at microsoft/rho.