Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks

Summary

The Open ASR Leaderboard expanded its evaluation framework by introducing dedicated multilingual and long-form transcription tracks across eleven datasets. Benchmarking more than sixty open- and closed-source models from eighteen organizations revealed clear operational trade-offs between accuracy and inference speed. Systems combining Conformer encoders with large language model decoders achieved the lowest word error rates in short-form English speech recognition. In contrast, connectionist temporal classification and token-and-duration transducer decoders yielded tenfold to hundredfold improvements in throughput with only slight accuracy penalties. While closed-source commercial systems maintained performance leads in long-form audio transcription, specialized single-language models consistently outperformed generalized systems supporting hundreds of languages.

Context

Most speech recognition benchmarks focus on short-form English transcription under 30 seconds, neglecting critical variables such as multilingual performance and inference throughput for long-form audio like meetings and podcasts.

Approach / What changed

The Open ASR Leaderboard incorporated multilingual and long-form transcription tracks, comparing more than 60 open- and closed-source models from 18 organizations across 11 datasets using word error rate and inverse real-time factor metrics.

Takeaways

• Combining Conformer or Fast Conformer encoders with LLM decoders achieves top English transcription accuracy, as demonstrated by Canary-Qwen-2.5B, Granite-Speech-3.3-8B, and Phi-4-Multimodal-Instruct.
• CTC and TDT decoders deliver 10x to 100x faster throughput than LLM decoders, making them suitable for real-time or batch transcription despite slightly higher error rates.
• Closed-source systems maintain a lead over open-source alternatives in long-form transcription, while fine-tuned or single-language models consistently outperform massively multilingual architectures in accuracy.

Related reading

• Hugging Face · Nov 13, 2025

  Building for an Open Future - our new partnership with Google Cloud

  Hugging Face and Google Cloud announced an expanded strategic partnership designed to streamline the deployment and management of open models on Google Cloud infrastructure. Prompted by a tenfold increase in Hugging Face usage on Google Cloud over three years, the collaboration introduces a joint CDN Gateway using Hugging Face Xet technology to cache models and datasets directly on Google Cloud. This caching mechanism aims to shorten download times, strengthen model supply chain resilience, and accelerate time-to-first-token across Vertex AI, Google Kubernetes Engine, Cloud Run, and Compute Engine virtual machines. Hugging Face plans to integrate native library support for Google Cloud TPUs, lower instance prices on Inference Endpoints, and enhance Hub security scanning via VirusTotal, Google Threat Intelligence, and Mandiant.

  Jeff Boudier, Simon Pagezy

• huggingface.co · Jan 12, 2022

  Boosting Wav2Vec2 with n-grams in 🤗 Transformers

  Wav2Vec2 models fine-tuned with Connectionist Temporal Classification transcribe speech without external language models, but decoding can still suffer from spelling inaccuracies. Hugging Face Transformers addressed this by integrating Kensho Technologies' pyctcdecode library to support decoding with n-gram language models. Instead of decoding using simple argmax operations over logits, the Wav2Vec2ProcessorWithLM class feeds full probability matrices into beam search guided by KenLM n-gram probabilities. KenLM's build_binary utility compresses language model ARPA files into binary formats, reducing file size by more than half for faster loading and hub deployment. In Swedish xls-r-300m-sv benchmarks on Common Voice 7, this 5-gram boosted decoding setup achieved an 18.85% word error rate, delivering an approximate 30% relative performance gain.