mmBERT: ModernBERT goes Multilingual

Summary

mmBERT is a massively multilingual encoder model trained on more than 3T tokens across over 1,800 languages to improve upon existing multilingual architectures like XLM-R. Built upon ModernBERT with a Gemma 2 tokenizer, mmBERT employs a three-phase training curriculum consisting of pre-training on 60 languages, mid-training on 110 languages, and a final decay phase covering 1,833 languages. The training pipeline integrates an inverse mask ratio schedule, dynamic language temperature annealing, and TIES merging across three decay variants. Benchmark evaluations demonstrate strong natural language understanding on English GLUE and multilingual XTREME, as well as competitive retrieval performance on MTEB v2 and CoIR. The release includes standard base and small models alongside open training data and checkpoints.

Context

Previous multilingual encoder models faced limitations in speed, performance, and strategies for effectively learning low-resource languages, with mmBERT aiming to improve upon models like XLM-R.

Approach / What changed

mmBERT adapts the ModernBERT architecture with a Gemma 2 tokenizer and trains across three phases (pre-training, mid-training with context extension to 8192 tokens, and a decay phase). It uses progressive language addition (60 to 110 to 1,833 languages), an inverse mask ratio schedule (30% to 15% to 5%), annealed language sampling temperatures (tau 0.7 to 0.5 to 0.3), and TIES model merging.

Takeaways

• mmBERT uses an inverse mask ratio schedule that reduces masking from 30% to 15% to 5% across its three training phases.
• Over 1,700 low-resource languages are introduced exclusively in the final 100B token decay phase, enabling effective learning without excessive data repetition.
• Three distinct variants trained during the decay phase (English-focused, 110-language, and all-language) are combined using TIES model merging.

Related reading

• Hugging Face · Sep 4, 2025

  Welcome EmbeddingGemma, Google's new efficient embedding model

  Google DeepMind released EmbeddingGemma, a multilingual embedding model with 308 million parameters and a 2048-token context window designed for on-device applications. Based on the Gemma3 transformer backbone, the architecture replaces causal attention with bidirectional attention to function as an encoder, followed by mean pooling and two dense layers producing 768-dimensional vectors. The model incorporates Matryoshka Representation Learning, allowing outputs to be truncated down to 512, 256, or 128 dimensions for reduced memory and storage footprints. Trained on approximately 320 billion multilingual tokens across more than 100 languages, the quantized model operates under 200 MB of RAM. In domain-specific evaluations on the MIRIAD dataset, fine-tuning increased NDCG@10 from 0.8340 to 0.8862, outperforming larger baselines.

  Tom Aarsen, Joshua, Alvaro Bartolome, Aritra Roy Gosthipaty, Pedro Cuenca, Sergio Paniego

• huggingface.co · Nov 9, 2020

  Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

  Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.