Almost every LLM reads text through subword tokenization, which the paper argues "obscures fine-grained information — which is problematic, especially for scientific data, such as computer code or biological sequences, where meaning depends on the individual characters." Models that read raw UTF-8 bytes avoid that, but had consistently lagged subword models, and the reason was economic rather than architectural: matching a subword model meant pretraining a byte-level one from scratch. Bolmo's claim is that you can skip that bill by converting an already-capable subword model — a process Ai2 calls byteifying.

The architecture keeps the source transformer intact as a "global model" over patches and wraps it in new byte-level machinery: each byte embedding residually adds the longest subword-token embedding ending at that position; a single mLSTM local encoder contextualizes bytes; a non-causal boundary predictor with one byte of lookahead reproduces the source tokenizer's patch boundaries >99% of the time; the last byte of each patch is pooled as the patch representation; and after depooling, a four-layer mLSTM local decoder feeds a 512-way byte head. Conversion runs in two stages. Stage 1 freezes the global model and trains only the new components to distill the source subword LM exactly — 9.8B tokens (~43B bytes) under a combined boundary, layer-4 representation-alignment and patch-likelihood objective. Stage 2 trains everything end to end on 39.3B tokens (~173B bytes) with boundary plus cross-entropy loss, so the model can finally exploit byte-level detail. Total: under 1% of a typical pretraining budget.

Bolmo 7B, converted from Olmo 3 7B, scores 78.6 on CUTE character understanding against 56.9 for its own source model and 52.3 / 50.8 for BLT 7B and EvaByte 6.5B — a +21.7-point gain over the model it came from, and the best of any prior byte-level LM at comparable size. It pays for that elsewhere: GSM8K 68.0 vs 73.1, MATH −7.7, HumanEval pass@1 40.6 vs 49.0 (though pass@16 is higher, which the authors read as more diverse continuations), non-STEM multiple choice 75.8 vs 77.7. Measured in wallclock on H100s rather than FLOPs, decoding runs at ~125 bytes/s against ~150 for the subword model and a 72K-byte prefill takes ~1s against ~0.8s, with the efficiency crossover at ~6.6 bytes per patch. Two honest constraints: the boundary predictor sees one future byte where a real tokenizer has unrestricted lookahead, and post-training by task arithmetic requires "embedding resettability" and does not always transfer reliably.

The family predates this entry's date — Bolmo 1B and 7B went up in December 2025 alongside the arXiv preprint, and now carry ~50 and ~60 HuggingFace likes. What the October 2026 Nature publication adds is byteification applied beyond Olmo: Bwen 8B from Qwen3-8B-Base (see Qwen3), which Ai2 calls "our strongest byteified model yet" and reports outperforming Bolmo 7B across their aggregate evaluation suite, and Llama-B 8B from Meta-Llama-3-8B, which "comes close to matching" its source. All four models also ship Stage-1 checkpoints, which keep the original global model frozen while training only the byte-level components — a cheaper starting point for researchers who want to run just the second half of the recipe. Apache 2.0, except Llama-B 8B under the Llama 3 license; data (bolmo_mix) and bolmo-core training code are open.

Paper

Venue Nature
Authors: Benjamin Minixhofer · Tyler Murray · Tomasz Limisiewicz · Anna Korhonen · Luke Zettlemoyer · Noah A. Smith · Edoardo M. Ponti · Luca Soldaini · Valentin Hofmann
architecturetokenizerefficiencyopen-weightresearchdistillation

Related