---
title: "Adding Benchmaxxer Repellant to the Open ASR Leaderboard"
description: "The Open ASR Leaderboard is adding private English speech-recognition datasets from Appen Inc. and DataoceanAI to evaluate scripted and conversational speech across several accents. To reduce benchmark-specific optimization (“benchmaxxing”) and test-set contamination, the datasets remain private, while the default Average WER continues to use public datasets only. A toggle exposes the private data, with macroaverages for overall, scripted, conversational, American, and non-American subsets; individual split scores are withheld to discourage provider- or accent-specific optimization. The leaderboard uses a shared Hub dataset, normalized transcripts and outputs, and open-source evaluation code, while private evaluation is run after public-set verification when models are submitted."
---

# Adding Benchmaxxer Repellant to the Open ASR Leaderboard

[Hugging Face](https://yomu.fyi/company/hugging-face) · Eric Bezzam, Steven Zheng, Eustache Le Bihan, Sergio Bruccoleri, Jeanine Sinanan-Singh, Casey Ford, Guanbo Wang, Yukai Huang, Ke Li, Yufeng Hao, Liao Xiaoling · May 6, 2026

**Type:** Announcement

## Summary

The Open ASR Leaderboard is adding private English speech-recognition datasets from Appen Inc. and DataoceanAI to evaluate scripted and conversational speech across several accents. To reduce benchmark-specific optimization (“benchmaxxing”) and test-set contamination, the datasets remain private, while the default Average WER continues to use public datasets only. A toggle exposes the private data, with macroaverages for overall, scripted, conversational, American, and non-American subsets; individual split scores are withheld to discourage provider- or accent-specific optimization. The leaderboard uses a shared Hub dataset, normalized transcripts and outputs, and open-source evaluation code, while private evaluation is run after public-set verification when models are submitted.

## Context

The leaderboard faces risks from benchmark-specific optimization, including models improving scores through use of public test sets or training data closely resembling a particular dataset. It also needs to represent differences across accents, scripted versus conversational speech, and application-specific capabilities rather than treating one score as a catch-all measure.

## Approach / What changed

Appen Inc. and DataoceanAI supplied private scripted and conversational English ASR datasets spanning American, Australian, Canadian, Indian, and British accents. The leaderboard adds a Private data tab with macroaverages for overall, scripted, conversational, US, and non-US data, keeps private data out of the default Average WER, and withholds scores for individual splits. Submitted models report public-set results, which are verified before private-set metrics are computed.

## Takeaways

- Private datasets cover scripted and conversational English speech across multiple accents, with samples including punctuated text, casing, proper nouns, disfluencies, and spontaneous conversation.
- Average WER is a macroaverage of provider averages weighted equally; private datasets are excluded by default and can be included with a toggle, with Rank Δ showing ordering changes.
- Individual private split scores are intentionally withheld to limit optimization toward a specific data provider or accent; multiple providers are used to balance possible training-data advantages.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/open-asr-leaderboard-private-data)
- Source URL: https://huggingface.co/blog/open-asr-leaderboard-private-data
- Ingested by Yomu: 2026-08-29T01:48:40.562Z

[Read original post](https://huggingface.co/blog/open-asr-leaderboard-private-data)