Measuring benchmark optimization in speech recognition

Summary

Public voice AI benchmark scores can overstate speech-recognition ability when models learn benchmark-specific patterns rather than improve underlying transcription. Traditional evaluations may miss reliability, naturalness, contextual appropriateness, and other real-world conditions, motivating held-out sets in Real World VoiceEQ, the Open-ASR Leaderboard, and the Far-field ASR Leaderboard. The study introduces three tests and evaluates 11 widely used open-source ASR models, finding that several reproduce VoxPopuli and LibriSpeech reference transcripts when audio contradicts them, words are silenced, or two written forms sound identical. Models also sometimes use acoustic cues to identify dataset membership and select its expected transcription policy, while fresh post-cutoff data and other interventions often restore more audio-faithful output; the findings support fully held-out evaluation and broader metrics than a single public WER score.

Takeaways

1. 1

   An ensemble-based consensus disagreement probe flagged potential reference errors in 40% of analyzed VoxPopuli test clips, affecting roughly 3% of all reference words.

2. 2

   Models exhibiting benchmark-optimized behavior reproduced erroneous VoxPopuli reference transcripts 18–30% of the time, and lower-WER models were more likely to reproduce those errors.

3. 3

   Some models achieved roughly 90% accuracy when switching between spelling conventions associated with different datasets, exceeding the 50% random-choice baseline.