# Announcing Evaluation on the Hub

huggingface.co · Lewis Tunstall, Abhishek, Tristan Thrush, Sasha Luccioni, Leandro von Werra, Nazneen Rajani, Aleksandra Piktus, Omar Sanseviero, Douwe Kiela · Jun 28, 2022

**Type:** Announcement

## Summary

Hugging Face introduced Evaluation on the Hub, a no-code tool designed to streamline the evaluation of machine learning models across diverse datasets. Traditional evaluation approaches often suffer from reproducibility issues, implementation inconsistencies, and cumbersome workflows across varied metrics and datasets. Powered by AutoTrain and Hugging Face Spaces, the service allows users to configure tasks, map dataset columns, select evaluation metrics, and run tests directly from dataset pages. Completed evaluations automatically open pull requests on the respective model cards to encode standardized verification metadata and update public dataset leaderboards. This system establishes consistent benchmarking pipelines across tasks like image classification, text summarization, and named entity recognition without requiring local code execution.

## Context

Evaluating machine learning models across multiple datasets and metrics is often cumbersome, and self-reported performance numbers frequently suffer from bugs, subtle implementation differences, and lack of reproducibility. Additionally, traditional benchmark paradigms struggle with saturation, brittleness, and unclear data distribution boundaries.

## Approach / What changed

Evaluation on the Hub provides a no-code evaluation interface built on Hugging Face Spaces and powered by AutoTrain. Users select a dataset, configure splits, map columns to standard formats, and select task metrics. Once triggered, the AutoTrain backend evaluates selected models and automatically submits pull requests that encode the verified evaluation results directly into model card metadata, updating leaderboard rankings.

## Takeaways

- Evaluation on the Hub uses AutoTrain as its execution backend to run evaluations and automatically open pull requests that store verified results in model card metadata.
- The tool enables no-code model evaluation by letting users map dataset columns to standard formats, select default or custom metrics, and launch jobs directly through Hugging Face Spaces.
- Results generated by the system are aggregated into dataset leaderboards to standardize baseline comparisons across tasks such as text classification, summarization, and image classification.

**Tags:** [Developer Experience](https://yomu.fyi/topic/developer-experience), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://huggingface.co/blog/eval-on-the-hub)