Introducing Falcon-H1-Arabic: Pushing the Boundaries of Arabic Language AI with Hybrid Architecture

Summary

The Technology Innovation Institute introduced Falcon-H1-Arabic, a family of Arabic language models released in 3B, 7B, and 34B parameter sizes. Built on a hybrid architecture, each block runs Mamba State Space Models and Transformer attention in parallel before fusing their representations at the output projection. Pre-training incorporated around 300 billion tokens across Arabic dialects, English, and multilingual content filtered via linguistic analysis. A post-training pipeline combining supervised fine-tuning and direct preference optimization addresses long-context utilization and alignment. Across evaluations on the Open Arabic LLM Leaderboard, the 34B model achieves roughly 75%, exceeding the performance of larger baseline systems like Llama-3.3-70B.

Takeaways

1. 1

   Falcon-H1-Arabic runs Mamba and Transformer attention layers in parallel within each block, fusing representations before the block output projection.

2. 2

   Context capacities scale up to 128K tokens for the 3B model and 256K tokens for both the 7B and 34B model variants.

3. 3

   On the Open Arabic LLM Leaderboard, the 34B model reaches approximately 75%, outperforming larger models such as Llama-3.3-70B.