Adding Benchmaxxer Repellant to the Open ASR Leaderboard

Summary

The Open ASR Leaderboard is adding private English speech-recognition datasets from Appen Inc. and DataoceanAI to evaluate scripted and conversational speech across several accents. To reduce benchmark-specific optimization (“benchmaxxing”) and test-set contamination, the datasets remain private, while the default Average WER continues to use public datasets only. A toggle exposes the private data, with macroaverages for overall, scripted, conversational, American, and non-American subsets; individual split scores are withheld to discourage provider- or accent-specific optimization. The leaderboard uses a shared Hub dataset, normalized transcripts and outputs, and open-source evaluation code, while private evaluation is run after public-set verification when models are submitted.

Takeaways

1. 1

   Private datasets cover scripted and conversational English speech across multiple accents, with samples including punctuated text, casing, proper nouns, disfluencies, and spontaneous conversation.

2. 2

   Average WER is a macroaverage of provider averages weighted equally; private datasets are excluded by default and can be included with a toggle, with Rank Δ showing ordering changes.

3. 3

   Individual private split scores are intentionally withheld to limit optimization toward a specific data provider or accent; multiple providers are used to balance possible training-data advantages.