mmBERT

Modern multilingual encoders from JHU-CLSP (Marone, Weller, Fleshman, Yang, Lawrie, Van Durme): ModernBERT-style architecture (Flash Attention 2, 8K context, Gemma 2 tokenizer / 256K vocab) pretrained from scratch on 3T+ tokens with annealed language learning — progressively expanding from 60 to 1,833 languages, saving the low-resource long tail for the decay phase. The first real successor to XLM-R as the default multilingual encoder; mmBERT-base runs ~468K monthly HF downloads.