# Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs

[Hugging Face](https://yomu.fyi/company/hugging-face) · Omar saif alkaabi, Ahmed Alzubaidi, Hamza Alobeidli, Shaikha Alsuwaidi, Mohammed Alyafeai, Leen AlQadi, Basma Boussaha, Hakim Hacid · Jan 27, 2026

**Type:** Benchmark

## Summary

Standard Arabic language model evaluations predominantly assess Modern Standard Arabic while overlooking regional spoken dialects that feature distinct syntax, vocabulary, and cultural grounding. To address this evaluation gap, researchers created Alyah, a benchmark comprising 1,173 manually curated multiple-choice questions collected from native Emirati speakers across seven cultural and linguistic categories. Distractor answer choices were synthetically generated using language models and manually reviewed for semantic plausibility. Evaluating base and instruction-tuned Arabic-native, adapted, and multilingual models revealed that instruction-tuned systems consistently outperformed base variants, with falcon-h1-arabic-7b-instruct achieving the top instruction accuracy at 82.18%. However, questions involving implicit meanings, rare expressions, and specific dialect phenomena remained difficult across all tested model families and sizes.

## Context

Most existing Arabic large language model benchmarks evaluate Modern Standard Arabic, leaving regional dialects under-evaluated. Dialects differ from Modern Standard Arabic in syntax, vocabulary, phonology, and cultural grounding, causing models that succeed on formal texts to struggle with dialectal greetings, idioms, and conversational usage.

## Approach / What changed

Researchers introduced Alyah, an Emirati-centric benchmark of 1,173 multiple-choice questions collected manually from native Emirati speakers across seven categories. Distractor choices were generated with language models and reviewed for plausibility, and dozens of base and instruction-tuned Arabic-native, multilingual, and adapted models were evaluated on accuracy.

## Takeaways

- The Alyah benchmark consists of 1,173 manually curated multiple-choice questions spanning seven categories, with Language & Dialect being the largest category at 619 samples.
- Instruction-tuned models generally outperformed base counterparts on conversational norms and values, with falcon-h1-arabic-7b-instruct scoring highest among instruct models at 82.18% accuracy.
- Generic multilingual pretraining struggled on dialect-specific questions, with Language and Dialect alongside Greetings and Daily Expressions proving consistently difficult across model sizes.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://huggingface.co/blog/tiiuae/emirati-benchmarks)