Announcing Evaluation on the Hub

Summary

Hugging Face introduced Evaluation on the Hub, a no-code tool designed to streamline the evaluation of machine learning models across diverse datasets. Traditional evaluation approaches often suffer from reproducibility issues, implementation inconsistencies, and cumbersome workflows across varied metrics and datasets. Powered by AutoTrain and Hugging Face Spaces, the service allows users to configure tasks, map dataset columns, select evaluation metrics, and run tests directly from dataset pages. Completed evaluations automatically open pull requests on the respective model cards to encode standardized verification metadata and update public dataset leaderboards. This system establishes consistent benchmarking pipelines across tasks like image classification, text summarization, and named entity recognition without requiring local code execution.

Context

Evaluating machine learning models across multiple datasets and metrics is often cumbersome, and self-reported performance numbers frequently suffer from bugs, subtle implementation differences, and lack of reproducibility. Additionally, traditional benchmark paradigms struggle with saturation, brittleness, and unclear data distribution boundaries.

Approach / What changed

Evaluation on the Hub provides a no-code evaluation interface built on Hugging Face Spaces and powered by AutoTrain. Users select a dataset, configure splits, map columns to standard formats, and select task metrics. Once triggered, the AutoTrain backend evaluates selected models and automatically submits pull requests that encode the verified evaluation results directly into model card metadata, updating leaderboard rankings.

Takeaways

• Evaluation on the Hub uses AutoTrain as its execution backend to run evaluations and automatically open pull requests that store verified results in model card metadata.
• The tool enables no-code model evaluation by letting users map dataset columns to standard formats, select default or custom metrics, and launch jobs directly through Hugging Face Spaces.
• Results generated by the system are aggregated into dataset leaderboards to standardize baseline comparisons across tasks such as text classification, summarization, and image classification.

Related reading

• huggingface.co · Apr 13, 2022

  Machine Learning Experts - Lewis Tunstall

  In this interview with Britney Muller, Hugging Face machine learning engineer Lewis Tunstall discusses his background, his book Natural Language Processing with Transformers, and tooling optimizations for deploying transformer models. Tunstall notes that large model parameter sizes often lead to high latency, making off-the-shelf transformers impractical for responsive applications like chatbots. To resolve this issue, he developed single-line export tooling within the transformers library to convert models into the ONNX format. This conversion enables engineers to run PyTorch models in TensorFlow or on specialized hardware, achieving faster latency and higher throughput. Tunstall also recounts co-authoring the O'Reilly book with Leandro von Werra and Thom Wolf, collaborating on the Hugging Face Course, and training a GPT-2 model named code parrot on Python code.

  Britney Muller

• Hugging Face · Oct 1, 2025

  Introducing RTEB: A New Standard for Retrieval Evaluation

  Public embedding benchmarks often fail to measure true generalization because training sets overlap with public evaluation data, rewarding memorization over general search quality. The Retrieval Embedding Benchmark (RTEB) beta introduces a standardized evaluation framework designed for real-world enterprise retrieval applications. To prevent overfitting while preserving transparency, RTEB combines fully public datasets with private evaluation sets managed exclusively by MTEB maintainers. The suite spans 20 languages across domains like healthcare, law, finance, and code, measuring ranked retrieval quality using NDCG@10 on datasets of at least 1,000 documents. While currently limited to text-only retrieval and containing repurposed question-answering datasets, the benchmark provides clear signals when models degrade on unseen data.