Introducing Falcon-H1-Arabic: Pushing the Boundaries of Arabic Language AI with Hybrid Architecture

Summary

The Technology Innovation Institute introduced Falcon-H1-Arabic, a family of Arabic language models released in 3B, 7B, and 34B parameter sizes. Built on a hybrid architecture, each block runs Mamba State Space Models and Transformer attention in parallel before fusing their representations at the output projection. Pre-training incorporated around 300 billion tokens across Arabic dialects, English, and multilingual content filtered via linguistic analysis. A post-training pipeline combining supervised fine-tuning and direct preference optimization addresses long-context utilization and alignment. Across evaluations on the Open Arabic LLM Leaderboard, the 34B model achieves roughly 75%, exceeding the performance of larger baseline systems like Llama-3.3-70B.

Context

Feedback on the previous Falcon-Arabic release identified critical weaknesses in long-context understanding, Arabic dialectal variations, mathematical reasoning, and domain-specific knowledge.

Approach / What changed

Engineers developed Falcon-H1-Arabic using a hybrid Mamba-Transformer architecture, expanding context limits to 128K tokens for the 3B model and 256K tokens for the 7B and 34B models. They rebuilt the pre-training pipeline with deep linguistic filtering across approximately 300 billion tokens of Arabic, English, and multilingual data, followed by supervised fine-tuning and direct preference optimization.

Takeaways

• Falcon-H1-Arabic runs Mamba and Transformer attention layers in parallel within each block, fusing representations before the block output projection.
• Context capacities scale up to 128K tokens for the 3B model and 256K tokens for both the 7B and 34B model variants.
• On the Open Arabic LLM Leaderboard, the 34B model reaches approximately 75%, outperforming larger models such as Llama-3.3-70B.

Related reading

• Hugging Face · Sep 9, 2025

  mmBERT: ModernBERT goes Multilingual

  mmBERT is a massively multilingual encoder model trained on more than 3T tokens across over 1,800 languages to improve upon existing multilingual architectures like XLM-R. Built upon ModernBERT with a Gemma 2 tokenizer, mmBERT employs a three-phase training curriculum consisting of pre-training on 60 languages, mid-training on 110 languages, and a final decay phase covering 1,833 languages. The training pipeline integrates an inverse mask ratio schedule, dynamic language temperature annealing, and TIES merging across three decay variants. Benchmark evaluations demonstrate strong natural language understanding on English GLUE and multilingual XTREME, as well as competitive retrieval performance on MTEB v2 and CoIR. The release includes standard base and small models alongside open training data and checkpoints.

  Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, Ben Van Durme

• Hugging Face · Jan 27, 2026

  Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs