Introducing RTEB: A New Standard for Retrieval Evaluation

Summary

Public embedding benchmarks often fail to measure true generalization because training sets overlap with public evaluation data, rewarding memorization over general search quality. The Retrieval Embedding Benchmark (RTEB) beta introduces a standardized evaluation framework designed for real-world enterprise retrieval applications. To prevent overfitting while preserving transparency, RTEB combines fully public datasets with private evaluation sets managed exclusively by MTEB maintainers. The suite spans 20 languages across domains like healthcare, law, finance, and code, measuring ranked retrieval quality using NDCG@10 on datasets of at least 1,000 documents. While currently limited to text-only retrieval and containing repurposed question-answering datasets, the benchmark provides clear signals when models degrade on unseen data.

Takeaways

1. 1

   RTEB uses a hybrid strategy of public datasets and private, MTEB-evaluated datasets to measure true generalization and identify model overfitting.

2. 2

   The benchmark targets enterprise use cases across 20 languages in domains like law, healthcare, code, and finance, standardizing on NDCG@10 as the primary metric.

3. 3

   Current limitations include text-only modality, missing languages such as Chinese and Arabic, and roughly half of the datasets being repurposed from question-answering tasks.