"Enhancing Multilingual LLM Pretraining with Model-Based Data Selection": a model-based multilingual data-selection pipeline that filters FineWeb/FineWeb2 into a higher-quality subset, improving downstream pretraining. The released FineWeb-HQ / FineWeb2-HQ datasets are widely used (~73K downloads/mo) with community language forks. From Martin Jaggi's MLO lab; the pipeline feeds the Swiss AI Apertus data pipeline (cross-referenced, non-SNAI-branded artifact).

Two follow-ups landed in October 2026. FineWeb2-HQ-Plus (5 October, odc-by, 100 languages) keeps the top 10% of FineWeb2 v2.1.1 per language but swaps the per-language classifiers for a single cross-lingual XLM-RoBERTa-embedding classifier. It is trained jointly on all languages with multilingual instruction data (MURI-IT, Tagengo, WikiQA) added to the positives, lower-resource languages upsampled at most 3×, and negatives drawn from the 50th–75th percentile of a proxy classifier rather than uniformly. The method is from "Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection" (arXiv 2604.20549, DATA-FM @ ICLR 2026), whose ablations train 1B models on 103B tokens. FineWeb2-HQ-QA (7 October) is a raw, unfiltered synthetic set of question–answer pairs generated natively in 20 languages from 188.8M FineWeb2-HQ documents, as an alternative to translating English synthetic data. The filtering code is still to be released.

Dataset

datasettraining-datamultilingualresearch