# Introducing RTEB: A New Standard for Retrieval Evaluation

[Hugging Face](https://yomu.fyi/company/hugging-face) · Frank Liu, Kenneth Enevoldsen, Solomatin Roman, Isaac Chung, Tom Aarsen, Fődi, Zoltán · Oct 1, 2025

**Type:** Announcement

## Summary

Public embedding benchmarks often fail to measure true generalization because training sets overlap with public evaluation data, rewarding memorization over general search quality. The Retrieval Embedding Benchmark (RTEB) beta introduces a standardized evaluation framework designed for real-world enterprise retrieval applications. To prevent overfitting while preserving transparency, RTEB combines fully public datasets with private evaluation sets managed exclusively by MTEB maintainers. The suite spans 20 languages across domains like healthcare, law, finance, and code, measuring ranked retrieval quality using NDCG@10 on datasets of at least 1,000 documents. While currently limited to text-only retrieval and containing repurposed question-answering datasets, the benchmark provides clear signals when models degrade on unseen data.

## Context

Existing embedding benchmarks struggle to measure true generalization because training datasets often overlap with evaluation data, encouraging models to memorize test sets. Additionally, many benchmarks rely on academic or question-answering datasets that do not reflect enterprise retrieval use cases, or they focus too narrowly on a single domain.

## Approach / What changed

RTEB introduces a hybrid evaluation model combining public datasets for reproducibility with private datasets evaluated exclusively by MTEB maintainers to assess performance on unseen data. The benchmark focuses on real-world enterprise domains such as law, healthcare, finance, and code across 20 languages, using NDCG@10 as its default ranking metric for datasets containing at least 1,000 documents and 50 queries.

## Takeaways

- RTEB uses a hybrid strategy of public datasets and private, MTEB-evaluated datasets to measure true generalization and identify model overfitting.
- The benchmark targets enterprise use cases across 20 languages in domains like law, healthcare, code, and finance, standardizing on NDCG@10 as the primary metric.
- Current limitations include text-only modality, missing languages such as Chinese and Arabic, and roughly half of the datasets being repurposed from question-answering tasks.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Search](https://yomu.fyi/topic/search), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://huggingface.co/blog/rteb)