The Open ASR Leaderboard Adds Its First Global South Language

Summary

Voice Arena and Hugging Face have added two Monsoon evaluation sets, en-IN and hi-IN, to the Open ASR Leaderboard, introducing Indian English and Hindi coverage. The initiative addresses a limitation of aggregate WER: existing test sets record what was said but little about who spoke, making disparities across geography, age, gender, accent, devices, and other conditions difficult to see. The four speaker-disjoint public and private splits contain 4,888 speakers and vary across nine collection axes, using unscripted dual-channel conversations, broad district and device coverage, and 12 speaker metadata attributes. Because Hindi has extensive valid orthographic variation, its references use manually curated lattices and are scored with Orthographically-Informed Word Error Rate (OIWER); comparisons with flattened references show higher, non-uniform error rates and changed system rankings.

Takeaways

1. 1

   The four Monsoon splits comprise 4,888 speakers, and more than half of the speakers in each split appear exactly once; the ten largest contributors account for 2.8% to 6.8% of total duration.

2. 2

   Hindi lattices admit multiple written forms for each transcript span, with variants gathered from ASR outputs and language models and then judged by native-speaker linguists.

3. 3

   Scoring hypotheses against flattened Hindi references increases error rates for every tested system, not uniformly, and changes some system rankings compared with lattice-based scoring.