ProX (Programming Every Example)

"Programming every example": small LMs emit per-document cleaning programs (normalize, strip, keep/drop) instead of hand-crafted rules, refining pretraining data at scale (ICML 2025). The >100B-token refined corpora released on HuggingFace were reused by later pretraining efforts — a pre-SII GAIR contribution to the data-refinement toolchain from Pengfei Liu's group.