Community Evals: Because we're done trusting black-box leaderboards over the community

Summary

Hugging Face is introducing Community Evals, a system for decentralized and transparent reporting of model evaluation results on the Hub. It responds to saturated benchmarks and inconsistent scores across model cards, papers, and evaluation platforms, while acknowledging that benchmark performance does not necessarily reflect real-world capabilities. Benchmark dataset repositories can register as benchmarks, define reproducible eval specifications in eval.yaml, aggregate scores from model repositories and community submissions, and display leaderboards in dataset cards. Model results are stored in .eval_results/*.yaml, while users can submit sourced scores through pull requests that remain visible as community results, with Git history and APIs supporting tracking and further aggregation. The beta feature aims to expose what was evaluated, how, when, and by whom, but it does not claim to solve benchmark saturation, test-set training, or the benchmark-reality gap.

Takeaways

1. 1

   Benchmark dataset repositories can aggregate reported results from across the Hub and display leaderboards in their dataset cards, with MMLU-Pro, GPQA, and HLE already live.

2. 2

   Evaluation scores stored in .eval_results/*.yaml appear on model cards and feed into benchmark datasets; community-submitted results can remain visible without model-author approval.

3. 3

   Community Evals is intended to make evaluation details and provenance visible, but it does not solve benchmark saturation, test-set training, or the benchmark-reality gap.