mmBERT: ModernBERT goes Multilingual

Summary

mmBERT is a massively multilingual encoder model trained on more than 3T tokens across over 1,800 languages to improve upon existing multilingual architectures like XLM-R. Built upon ModernBERT with a Gemma 2 tokenizer, mmBERT employs a three-phase training curriculum consisting of pre-training on 60 languages, mid-training on 110 languages, and a final decay phase covering 1,833 languages. The training pipeline integrates an inverse mask ratio schedule, dynamic language temperature annealing, and TIES merging across three decay variants. Benchmark evaluations demonstrate strong natural language understanding on English GLUE and multilingual XTREME, as well as competitive retrieval performance on MTEB v2 and CoIR. The release includes standard base and small models alongside open training data and checkpoints.

Takeaways

1. 1

   mmBERT uses an inverse mask ratio schedule that reduces masking from 30% to 15% to 5% across its three training phases.

2. 2

   Over 1,700 low-resource languages are introduced exclusively in the final 100B token decay phase, enabling effective learning without excessive data repetition.

3. 3

   Three distinct variants trained during the decay phase (English-focused, 110-language, and all-language) are combined using TIES model merging.