---
title: "The Open ASR Leaderboard Adds Its First Global South Language"
description: "Voice Arena and Hugging Face have added two Monsoon evaluation sets, en-IN and hi-IN, to the Open ASR Leaderboard, introducing Indian English and Hindi coverage. The initiative addresses a limitation of aggregate WER: existing test sets record what was said but little about who spoke, making disparities across geography, age, gender, accent, devices, and other conditions difficult to see. The four speaker-disjoint public and private splits contain 4,888 speakers and vary across nine collection axes, using unscripted dual-channel conversations, broad district and device coverage, and 12 speaker metadata attributes. Because Hindi has extensive valid orthographic variation, its references use manually curated lattices and are scored with Orthographically-Informed Word Error Rate (OIWER); comparisons with flattened references show higher, non-uniform error rates and changed system rankings."
---

# The Open ASR Leaderboard Adds Its First Global South Language

[Hugging Face](https://yomu.fyi/company/hugging-face) · Eric Bezzam, Shobhit Banga, Manas Dhir, Bhaskar Singh, Manmeet Kaur, Aaditya Pareek, Walecha, Sagar Jain, Hanuman Sidh, Vanshika Chhabra · Aug 28, 2026

**Type:** Announcement

## Summary

Voice Arena and Hugging Face have added two Monsoon evaluation sets, en-IN and hi-IN, to the Open ASR Leaderboard, introducing Indian English and Hindi coverage. The initiative addresses a limitation of aggregate WER: existing test sets record what was said but little about who spoke, making disparities across geography, age, gender, accent, devices, and other conditions difficult to see. The four speaker-disjoint public and private splits contain 4,888 speakers and vary across nine collection axes, using unscripted dual-channel conversations, broad district and device coverage, and 12 speaker metadata attributes. Because Hindi has extensive valid orthographic variation, its references use manually curated lattices and are scored with Orthographically-Informed Word Error Rate (OIWER); comparisons with flattened references show higher, non-uniform error rates and changed system rankings.

## Context

Aggregate ASR leaderboard scores can conceal differences in error rates across populations because existing test sets generally record the speech content but little information about who spoke. Hindi also presents substantial orthographic variation, including multiple valid spellings, code-mixed terms, and joined or separated compound forms that a conventional normaliser cannot reliably collapse.

## Approach / What changed

Voice Arena and Hugging Face collected four speaker-disjoint Monsoon splits for Indian English and Hindi, with public and private portions, broad geographic and device coverage, spontaneous conversations, and speaker metadata. Hindi references use manually curated lattices of valid orthographic variants and are scored with OIWER. The Indian English set joins the main leaderboard, while Hindi appears in its multilingual evaluation.

## Takeaways

- The four Monsoon splits comprise 4,888 speakers, and more than half of the speakers in each split appear exactly once; the ten largest contributors account for 2.8% to 6.8% of total duration.
- Hindi lattices admit multiple written forms for each transcript span, with variants gathered from ASR outputs and language models and then judged by native-speaker linguists.
- Scoring hypotheses against flattened Hindi references increases error rates for every tested system, not uniformly, and changes some system rankings compared with lattice-based scoring.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/open-asr-leaderboard-global-south)
- Source URL: https://huggingface.co/blog/open-asr-leaderboard-global-south
- Ingested by Yomu: 2026-08-29T05:42:17.107Z

[Read original post](https://huggingface.co/blog/open-asr-leaderboard-global-south)