---
title: "Community Evals: Because we're done trusting black-box leaderboards over the community"
description: "Hugging Face is introducing Community Evals, a system for decentralized and transparent reporting of model evaluation results on the Hub. It responds to saturated benchmarks and inconsistent scores across model cards, papers, and evaluation platforms, while acknowledging that benchmark performance does not necessarily reflect real-world capabilities. Benchmark dataset repositories can register as benchmarks, define reproducible eval specifications in eval.yaml, aggregate scores from model repositories and community submissions, and display leaderboards in dataset cards. Model results are stored in .eval_results/*.yaml, while users can submit sourced scores through pull requests that remain visible as community results, with Git history and APIs supporting tracking and further aggregation. The beta feature aims to expose what was evaluated, how, when, and by whom, but it does not claim to solve benchmark saturation, test-set training, or the benchmark-reality gap."
---

# Community Evals: Because we're done trusting black-box leaderboards over the community

[Hugging Face](https://yomu.fyi/company/hugging-face) · ben burtenshaw, Nathan Habib, Bertrand Chevrier, merve, Daniel van Strien, Niels Rogge, Julien Chaumond · Feb 4, 2026

**Type:** Announcement

## Summary

Hugging Face is introducing Community Evals, a system for decentralized and transparent reporting of model evaluation results on the Hub. It responds to saturated benchmarks and inconsistent scores across model cards, papers, and evaluation platforms, while acknowledging that benchmark performance does not necessarily reflect real-world capabilities. Benchmark dataset repositories can register as benchmarks, define reproducible eval specifications in eval.yaml, aggregate scores from model repositories and community submissions, and display leaderboards in dataset cards. Model results are stored in .eval\_results/\*.yaml, while users can submit sourced scores through pull requests that remain visible as community results, with Git history and APIs supporting tracking and further aggregation. The beta feature aims to expose what was evaluated, how, when, and by whom, but it does not claim to solve benchmark saturation, test-set training, or the benchmark-reality gap.

## Context

The initiative responds to saturated benchmark results, inconsistent scores reported across model cards, papers, and evaluation platforms, and a gap between benchmark performance and reported real-world model behavior. It also addresses the lack of a single source of truth for evaluation results.

## Approach / What changed

Benchmark dataset repositories can register as benchmarks and define task specifications in eval.yaml using the Inspect AI format. Model repositories store scores in .eval\_results/\*.yaml, while community members submit sourced results through pull requests. The Hub aggregates model-author and community results, displays them on benchmark dataset cards and model cards, preserves Git-based history, and exposes scores through APIs.

## Takeaways

- Benchmark dataset repositories can aggregate reported results from across the Hub and display leaderboards in their dataset cards, with MMLU-Pro, GPQA, and HLE already live.
- Evaluation scores stored in .eval\_results/\*.yaml appear on model cards and feed into benchmark datasets; community-submitted results can remain visible without model-author approval.
- Community Evals is intended to make evaluation details and provenance visible, but it does not solve benchmark saturation, test-set training, or the benchmark-reality gap.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Testing](https://yomu.fyi/topic/testing)

- Source: [Hugging Face](https://huggingface.co/blog/community-evals)
- Source URL: https://huggingface.co/blog/community-evals
- Ingested by Yomu: 2026-08-28T08:01:02.253Z

[Read original post](https://huggingface.co/blog/community-evals)