Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs

Summary

Standard Arabic language model evaluations predominantly assess Modern Standard Arabic while overlooking regional spoken dialects that feature distinct syntax, vocabulary, and cultural grounding. To address this evaluation gap, researchers created Alyah, a benchmark comprising 1,173 manually curated multiple-choice questions collected from native Emirati speakers across seven cultural and linguistic categories. Distractor answer choices were synthetically generated using language models and manually reviewed for semantic plausibility. Evaluating base and instruction-tuned Arabic-native, adapted, and multilingual models revealed that instruction-tuned systems consistently outperformed base variants, with falcon-h1-arabic-7b-instruct achieving the top instruction accuracy at 82.18%. However, questions involving implicit meanings, rare expressions, and specific dialect phenomena remained difficult across all tested model families and sizes.

Context

Most existing Arabic large language model benchmarks evaluate Modern Standard Arabic, leaving regional dialects under-evaluated. Dialects differ from Modern Standard Arabic in syntax, vocabulary, phonology, and cultural grounding, causing models that succeed on formal texts to struggle with dialectal greetings, idioms, and conversational usage.

Approach / What changed

Researchers introduced Alyah, an Emirati-centric benchmark of 1,173 multiple-choice questions collected manually from native Emirati speakers across seven categories. Distractor choices were generated with language models and reviewed for plausibility, and dozens of base and instruction-tuned Arabic-native, multilingual, and adapted models were evaluated on accuracy.

Takeaways

• The Alyah benchmark consists of 1,173 manually curated multiple-choice questions spanning seven categories, with Language & Dialect being the largest category at 619 samples.
• Instruction-tuned models generally outperformed base counterparts on conversational norms and values, with falcon-h1-arabic-7b-instruct scoring highest among instruct models at 82.18% accuracy.
• Generic multilingual pretraining struggled on dialect-specific questions, with Language and Dialect alongside Greetings and Daily Expressions proving consistently difficult across model sizes.

Related reading

• Hugging Face · Jan 5, 2026

  Introducing Falcon-H1-Arabic: Pushing the Boundaries of Arabic Language AI with Hybrid Architecture

  The Technology Innovation Institute introduced Falcon-H1-Arabic, a family of Arabic language models released in 3B, 7B, and 34B parameter sizes. Built on a hybrid architecture, each block runs Mamba State Space Models and Transformer attention in parallel before fusing their representations at the output projection. Pre-training incorporated around 300 billion tokens across Arabic dialects, English, and multilingual content filtered via linguistic analysis. A post-training pipeline combining supervised fine-tuning and direct preference optimization addresses long-context utilization and alignment. Across evaluations on the Open Arabic LLM Leaderboard, the 34B model achieves roughly 75%, exceeding the performance of larger baseline systems like Llama-3.3-70B.

  Basma Boussaha, Mohammed Alyafeai, Ahmed Alzubaidi, Leen AlQadi, Shaikha Alsuwaidi, Omar saif alkaabi, Hamza Alobeidli, Hakim Hacid

• AWS · Jul 22, 2026