Thunder-Tok

Linguistics-aware Korean tokenizer: builds a large seed vocabulary from corpus substrings, filters structurally invalid candidates (broken Unicode bytes, word-boundary violations), and prunes toward frequent whole-word units aligned with Korean morphology — cutting token counts ~44% vs the Llama tokenizer at comparable downstream performance, which translates directly into faster, cheaper inference. Used inside Thunder-LLM; the arXiv paper now carries the title "Less Is More: Reducing Token Counts Without Compromising Performance".