# mmBERT: ModernBERT goes Multilingual

[Hugging Face](https://yomu.fyi/company/hugging-face) · Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, Ben Van Durme · Sep 9, 2025

**Type:** Announcement

## Summary

mmBERT is a massively multilingual encoder model trained on more than 3T tokens across over 1,800 languages to improve upon existing multilingual architectures like XLM-R. Built upon ModernBERT with a Gemma 2 tokenizer, mmBERT employs a three-phase training curriculum consisting of pre-training on 60 languages, mid-training on 110 languages, and a final decay phase covering 1,833 languages. The training pipeline integrates an inverse mask ratio schedule, dynamic language temperature annealing, and TIES merging across three decay variants. Benchmark evaluations demonstrate strong natural language understanding on English GLUE and multilingual XTREME, as well as competitive retrieval performance on MTEB v2 and CoIR. The release includes standard base and small models alongside open training data and checkpoints.

## Context

Previous multilingual encoder models faced limitations in speed, performance, and strategies for effectively learning low-resource languages, with mmBERT aiming to improve upon models like XLM-R.

## Approach / What changed

mmBERT adapts the ModernBERT architecture with a Gemma 2 tokenizer and trains across three phases (pre-training, mid-training with context extension to 8192 tokens, and a decay phase). It uses progressive language addition (60 to 110 to 1,833 languages), an inverse mask ratio schedule (30% to 15% to 5%), annealed language sampling temperatures (tau 0.7 to 0.5 to 0.3), and TIES model merging.

## Takeaways

- mmBERT uses an inverse mask ratio schedule that reduces masking from 30% to 15% to 5% across its three training phases.
- Over 1,700 low-resource languages are introduced exclusively in the final 100B token decay phase, enabling effective learning without excessive data repetition.
- Three distinct variants trained during the decay phase (English-focused, 110-language, and all-language) are combined using TIES model merging.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Search](https://yomu.fyi/topic/search)

[Read original post](https://huggingface.co/blog/mmbert)