HPLT
datasetThe flagship open multilingual pretraining corpus family from the multi-national HPLT consortium (Edinburgh a key partner, with Oslo, CUNI, Turku, Helsinki, and others — a multi-institution artifact, not solely Edinburgh's). HPLT v2 (ACL 2025) shipped ~7.6T whitespace tokens over 21B documents in 193 languages plus 380M parallel sentence pairs across 51 languages. HPLT 3.0 (LREC 2026) scales to ~30T whitespace tokens (≈13.5T Gemma-3 subword tokens, ≈40T characters — the counts differ by tokenizer, not by data), ~11.5B documents in 198 languages, distilled from 7.2PB of Internet Archive and Common Crawl (2012–2024) and released CC0 with monolingual and parallel data, a 9-language evaluation suite, and 57 encoder-decoder models. DocHPLT adds a document-aligned parallel corpus over ~50 language pairs.
Outputs 3
HPLT v2
dataset<p>~7.6T whitespace tokens / 21B documents across 193 languages (deduplicated 21TB + cleaned 15TB variants) plus 380M parallel sentence pairs over 51 languages; ACL 2025 long paper.</p>
HPLT 3.0
dataset<p>~30T whitespace tokens (~13.5T Gemma-3 subword; 40T characters), ~11.5B documents, 198 languages, distilled from 7.2PB of Internet Archive and Common Crawl (2012–2024), released CC0. Ships monolingual and parallel data, a 9-language eval suite, and 57 encoder-decoder models; LREC 2026.</p>
DocHPLT
dataset<p>Document-aligned parallel corpus covering ~50 language pairs with hundreds of millions of aligned document pairs with quality scores, released CC0.</p>