Introducing RTEB: A New Standard for Retrieval Evaluation

Summary

Public embedding benchmarks often fail to measure true generalization because training sets overlap with public evaluation data, rewarding memorization over general search quality. The Retrieval Embedding Benchmark (RTEB) beta introduces a standardized evaluation framework designed for real-world enterprise retrieval applications. To prevent overfitting while preserving transparency, RTEB combines fully public datasets with private evaluation sets managed exclusively by MTEB maintainers. The suite spans 20 languages across domains like healthcare, law, finance, and code, measuring ranked retrieval quality using NDCG@10 on datasets of at least 1,000 documents. While currently limited to text-only retrieval and containing repurposed question-answering datasets, the benchmark provides clear signals when models degrade on unseen data.

Context

Existing embedding benchmarks struggle to measure true generalization because training datasets often overlap with evaluation data, encouraging models to memorize test sets. Additionally, many benchmarks rely on academic or question-answering datasets that do not reflect enterprise retrieval use cases, or they focus too narrowly on a single domain.

Approach / What changed

RTEB introduces a hybrid evaluation model combining public datasets for reproducibility with private datasets evaluated exclusively by MTEB maintainers to assess performance on unseen data. The benchmark focuses on real-world enterprise domains such as law, healthcare, finance, and code across 20 languages, using NDCG@10 as its default ranking metric for datasets containing at least 1,000 documents and 50 queries.

Takeaways

• RTEB uses a hybrid strategy of public datasets and private, MTEB-evaluated datasets to measure true generalization and identify model overfitting.
• The benchmark targets enterprise use cases across 20 languages in domains like law, healthcare, code, and finance, standardizing on NDCG@10 as the primary metric.
• Current limitations include text-only modality, missing languages such as Chinese and Arabic, and roughly half of the datasets being repurposed from question-answering tasks.

Related reading

• huggingface.co · Jul 3, 2020

  The Reformer - Pushing the limits of language modeling

  Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.

  Patrick von Platen

• huggingface.co · Mar 31, 2021

  Understanding BigBird's Block Sparse Attention

  Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.