Streaming ASR vs. Batch ASR: The Buffer-Latency Trade-off
Streaming and batch ASR pipelines compared: how input buffer length, speaker diarization, and where audio is processed trade latency against transcript quality and privacy.
Loading post…
Hugging FaceFrancesco, Ivan Medennikov, Taejin Park, Jagadeesh Balam, Adi- margolin, Maryam MotamediAnnouncement
**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**
Summary
NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on VoiceArena’s Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). It classifies who spoke when in conversations, handling overlapping speech and supporting up to eight speakers across live and recorded audio. The model uses a Sortformer approach with arrival-order speaker labeling, a 31-layer Transformer encoder, and two memory mechanisms for streaming inference. It accepts 16 kHz single-channel audio and outputs a floating-point tensor of speaker-activity probabilities at a configurable time step. Developers can combine its timestamps with automatic speech recognition to produce speaker-attributed transcripts, with recommended input-buffer latencies ranging from 0.32 to 30.4 seconds.
Context
Speaker diarization classifies who spoke when in conversations, enabling speaker-attributed transcripts for meetings, customer calls, and podcasts. Without it, transcripts lack speaker attribution, reducing the usefulness of search, summaries, action items, and voice-agent memory.
Approach / What changed
The model uses a Sortformer architecture with arrival-order speaker labeling, a 31-layer Transformer encoder with rotary positional embeddings, and two memory mechanisms (Arrival-Order Speaker Cache and FIFO queue) to support both offline and streaming diarization for up to eight speakers.
Takeaways
Nemotron 3 Diarization achieves a 14.72% DER on VoiceArena’s Diarization-Bench, a ~24% relative reduction over the next-ranked system.
The model uses arrival-order speaker labeling to maintain stable speaker channels across streaming chunks, avoiding the need to solve new speaker permutations per chunk.
Adding licensed David AI multi-speaker data to training decreased compound DER by 0.77 absolute points, from 11.19% to 10.42%.
FAQ
How many speakers does Nemotron 3 Diarization support?
The model supports up to eight speaker channels. If a recording contains more speakers, speech can be missed or assigned to the wrong channel.
Does the model identify specific people or just label speakers?
The model uses anonymous labels (e.g., speaker_2) based on arrival order, not real-world identities. Downstream applications must map these to explicit identities using meeting metadata, user profiles, or active speaker verification models.
What latency configurations are available for streaming?
The model supports input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds. Shorter buffers reduce latency but generally lower accuracy and throughput. The 0.32-second configuration is the lowest recommended.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.