# Introducing Falcon-H1-Arabic: Pushing the Boundaries of Arabic Language AI with Hybrid Architecture

[Hugging Face](https://yomu.fyi/company/hugging-face) · Basma Boussaha, Mohammed Alyafeai, Ahmed Alzubaidi, Leen AlQadi, Shaikha Alsuwaidi, Omar saif alkaabi, Hamza Alobeidli, Hakim Hacid · Jan 5, 2026

**Type:** Announcement

## Summary

The Technology Innovation Institute introduced Falcon-H1-Arabic, a family of Arabic language models released in 3B, 7B, and 34B parameter sizes. Built on a hybrid architecture, each block runs Mamba State Space Models and Transformer attention in parallel before fusing their representations at the output projection. Pre-training incorporated around 300 billion tokens across Arabic dialects, English, and multilingual content filtered via linguistic analysis. A post-training pipeline combining supervised fine-tuning and direct preference optimization addresses long-context utilization and alignment. Across evaluations on the Open Arabic LLM Leaderboard, the 34B model achieves roughly 75%, exceeding the performance of larger baseline systems like Llama-3.3-70B.

## Context

Feedback on the previous Falcon-Arabic release identified critical weaknesses in long-context understanding, Arabic dialectal variations, mathematical reasoning, and domain-specific knowledge.

## Approach / What changed

Engineers developed Falcon-H1-Arabic using a hybrid Mamba-Transformer architecture, expanding context limits to 128K tokens for the 3B model and 256K tokens for the 7B and 34B models. They rebuilt the pre-training pipeline with deep linguistic filtering across approximately 300 billion tokens of Arabic, English, and multilingual data, followed by supervised fine-tuning and direct preference optimization.

## Takeaways

- Falcon-H1-Arabic runs Mamba and Transformer attention layers in parallel within each block, fusing representations before the block output projection.
- Context capacities scale up to 128K tokens for the 3B model and 256K tokens for both the 7B and 34B model variants.
- On the Open Arabic LLM Leaderboard, the 34B model reaches approximately 75%, outperforming larger models such as Llama-3.3-70B.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning)

[Read original post](https://huggingface.co/blog/tiiuae/falcon-h1-arabic)