UltraData 2609 release (Code, RL, SFT-Agent)
datasetYour notes
OpenBMB's September 2026 UltraData drop, released alongside MiniCPM5-2B, whose pretraining corpus it feeds. The drop now carries both the L2 and L3 tiers, split by language (C++, C#, Go, Java, JavaScript, PHP, Python and more) as parquet shards, together with the L2 quality classifier used to build it. It follows Ultra-FineWeb-L3 as the second tier-labelled UltraData release. The same week added UltraData-RL-2609, 85,995 verifiable RL problems (math 32.4K, code 23.7K, long-context 18.0K, knowledge 11.9K) on which JustRL II was run, and UltraData-SFT-Agent-2609, 483,661 agent trajectories across code, search, general, and tool-use agents; the pipeline itself is described in the tiered data-management paper (arXiv 2602.09003). Apache 2.0.
The technical report arrived on 2026-10-09, five weeks after the data — UltraData-Code: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data (arXiv:2610.23960), sixteen authors across Peking, Tsinghua and ModelBest. Its premise is that corpus expansion has run out of room — "as obtaining public code of sufficient quality becomes more difficult, further progress cannot rely on corpus expansion alone" — so the four tiers apply progressively narrower processing to progressively smaller pools. L0 keeps raw repositories and provenance so the pipeline can be re-run; L1 is filtered, normalised, near-deduplicated code for broad pretraining; L2 selects algorithmically relevant files by language-adaptive relevance signals, about 400B tokens across 11 languages; L3 converts selected implementations into structured programming exercises, about 150B tokens over the same 11.
The ablation is controlled, and worth reading with its scale attached. On 10B-token continual pretraining of a 1B model, L2 beats L1 by +7.80pp pass@1 on EvalPlus and +5.13pp on MultiPL-E; replacing half the L2 tokens with L3 adds a further +8.42pp and +8.07pp. Both gains are reported to grow at a 100B-token budget, which is the direction a data paper needs — though 1B/10B is still far below the scale this corpus is built to serve. The result to take from it is that the synthesis tier outperforms the selection tier at a third of the token count.