---
title: "QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard"
description: "QIMMA is an Arabic LLM leaderboard built to address fragmented evaluation, translation artifacts, weak benchmark quality checks, limited coverage, and reproducibility gaps. It treats benchmark validation as a prerequisite for reported scores rather than an afterthought. The suite consolidates 109 subsets from 14 source benchmarks into more than 52,000 samples across seven domains, with 99% native Arabic content, Arabic-adapted HumanEval+ and MBPP+ coding tasks, and six prompt templates. Before model evaluation, Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B assess samples using a 10-point rubric, flagged cases receive native-speaker review, and code statements are refined without changing task identifiers, reference solutions, or tests. In the April 2026 rankings, Qwen/Qwen3.5-397B-A17B-FP8 ranks first, while Arabic-specialized models lead cultural and linguistic tasks and multilingual models lead coding."
---

# QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard

[Hugging Face](https://yomu.fyi/company/hugging-face) · Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai, Maitha Alhammadi, Shaikha Alsuwaidi, Omar saif alkaabi, Basma Boussaha, Hakim Hacid · Apr 21, 2026

**Type:** Benchmark

## Summary

QIMMA is an Arabic LLM leaderboard built to address fragmented evaluation, translation artifacts, weak benchmark quality checks, limited coverage, and reproducibility gaps. It treats benchmark validation as a prerequisite for reported scores rather than an afterthought. The suite consolidates 109 subsets from 14 source benchmarks into more than 52,000 samples across seven domains, with 99% native Arabic content, Arabic-adapted HumanEval+ and MBPP+ coding tasks, and six prompt templates. Before model evaluation, Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B assess samples using a 10-point rubric, flagged cases receive native-speaker review, and code statements are refined without changing task identifiers, reference solutions, or tests. In the April 2026 rankings, Qwen/Qwen3.5-397B-A17B-FP8 ranks first, while Arabic-specialized models lead cultural and linguistic tasks and multilingual models lead coding.

## Context

Arabic NLP evaluation is fragmented across isolated tasks and narrow domains, with translation artifacts, insufficient validation, annotation and answer-quality issues, cultural bias, limited reproducibility, and few unified assessments. The stated motivation is to determine whether Arabic benchmarks measure genuine Arabic language capability and whether their reported results remain reliable after quality problems are identified.

## Approach / What changed

QIMMA combines 109 subsets from 14 source benchmarks into a unified suite and validates samples before evaluating models. Two LLMs independently score each sample against a 10-point rubric; samples flagged by both are dropped, while disagreements proceed to review by native Arabic speakers with cultural and dialectal familiarity. For code benchmarks, Arabic problem statements are refined while task identifiers, reference solutions, and test suites remain unchanged. Evaluation uses LightEval, EvalPlus, and FannOrFlop with standardized Arabic prompt templates and task-specific metrics.

## Takeaways

- QIMMA’s validation pipeline uses Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B; a sample scoring below 7/10 from either model is eliminated, while cases flagged by only one model receive human review.
- The suite contains more than 52,000 samples across seven domains, with 99% native Arabic content and Arabic-adapted HumanEval+ and MBPP+ tasks for coding evaluation.
- The April 2026 leaderboard places Qwen/Qwen3.5-397B-A17B-FP8 first overall; Jais-2-70B-Chat leads ArabicMMLU and ArabCulture, while Qwen3.5-397B leads HumanEval+ and MBPP+.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard)
- Source URL: https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard
- Ingested by Yomu: 2026-08-29T01:48:07.297Z

[Read original post](https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard)