Loading post…
Hugging FaceLeen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai, Maitha Alhammadi, Shaikha Alsuwaidi, Omar saif alkaabi, Basma Boussaha, Hakim HacidBenchmark
QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard
Summary
QIMMA is an Arabic LLM leaderboard built to address fragmented evaluation, translation artifacts, weak benchmark quality checks, limited coverage, and reproducibility gaps. It treats benchmark validation as a prerequisite for reported scores rather than an afterthought. The suite consolidates 109 subsets from 14 source benchmarks into more than 52,000 samples across seven domains, with 99% native Arabic content, Arabic-adapted HumanEval+ and MBPP+ coding tasks, and six prompt templates. Before model evaluation, Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B assess samples using a 10-point rubric, flagged cases receive native-speaker review, and code statements are refined without changing task identifiers, reference solutions, or tests. In the April 2026 rankings, Qwen/Qwen3.5-397B-A17B-FP8 ranks first, while Arabic-specialized models lead cultural and linguistic tasks and multilingual models lead coding.
Context
Arabic NLP evaluation is fragmented across isolated tasks and narrow domains, with translation artifacts, insufficient validation, annotation and answer-quality issues, cultural bias, limited reproducibility, and few unified assessments. The stated motivation is to determine whether Arabic benchmarks measure genuine Arabic language capability and whether their reported results remain reliable after quality problems are identified.
Approach / What changed
QIMMA combines 109 subsets from 14 source benchmarks into a unified suite and validates samples before evaluating models. Two LLMs independently score each sample against a 10-point rubric; samples flagged by both are dropped, while disagreements proceed to review by native Arabic speakers with cultural and dialectal familiarity. For code benchmarks, Arabic problem statements are refined while task identifiers, reference solutions, and test suites remain unchanged. Evaluation uses LightEval, EvalPlus, and FannOrFlop with standardized Arabic prompt templates and task-specific metrics.
Takeaways
QIMMA’s validation pipeline uses Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B; a sample scoring below 7/10 from either model is eliminated, while cases flagged by only one model receive human review.
The suite contains more than 52,000 samples across seven domains, with 99% native Arabic content and Arabic-adapted HumanEval+ and MBPP+ tasks for coding evaluation.
The April 2026 leaderboard places Qwen/Qwen3.5-397B-A17B-FP8 first overall; Jais-2-70B-Chat leads ArabicMMLU and ArabCulture, while Qwen3.5-397B leads HumanEval+ and MBPP+.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.