Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks

Summary

The Open ASR Leaderboard expanded its evaluation framework by introducing dedicated multilingual and long-form transcription tracks across eleven datasets. Benchmarking more than sixty open- and closed-source models from eighteen organizations revealed clear operational trade-offs between accuracy and inference speed. Systems combining Conformer encoders with large language model decoders achieved the lowest word error rates in short-form English speech recognition. In contrast, connectionist temporal classification and token-and-duration transducer decoders yielded tenfold to hundredfold improvements in throughput with only slight accuracy penalties. While closed-source commercial systems maintained performance leads in long-form audio transcription, specialized single-language models consistently outperformed generalized systems supporting hundreds of languages.

Takeaways

1. 1

   Combining Conformer or Fast Conformer encoders with LLM decoders achieves top English transcription accuracy, as demonstrated by Canary-Qwen-2.5B, Granite-Speech-3.3-8B, and Phi-4-Multimodal-Instruct.

2. 2

   CTC and TDT decoders deliver 10x to 100x faster throughput than LLM decoders, making them suitable for real-time or batch transcription despite slightly higher error rates.

3. 3

   Closed-source systems maintain a lead over open-source alternatives in long-form transcription, while fine-tuned or single-language models consistently outperform massively multilingual architectures in accuracy.