---
title: "**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**"
description: "NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on VoiceArena’s Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). It classifies who spoke when in conversations, handling overlapping speech and supporting up to eight speakers across live and recorded audio. The model uses a Sortformer approach with arrival-order speaker labeling, a 31-layer Transformer encoder, and two memory mechanisms for streaming inference. It accepts 16 kHz single-channel audio and outputs a floating-point tensor of speaker-activity probabilities at a configurable time step. Developers can combine its timestamps with automatic speech recognition to produce speaker-attributed transcripts, with recommended input-buffer latencies ranging from 0.32 to 30.4 seconds."
---

# \*\*Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization\*\*

[Hugging Face](https://yomu.fyi/company/hugging-face) · Francesco, Ivan Medennikov, Taejin Park, Jagadeesh Balam, Adi- margolin, Maryam Motamedi · Sep 23, 2026

**Type:** Announcement

## Summary

NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on VoiceArena’s Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). It classifies who spoke when in conversations, handling overlapping speech and supporting up to eight speakers across live and recorded audio. The model uses a Sortformer approach with arrival-order speaker labeling, a 31-layer Transformer encoder, and two memory mechanisms for streaming inference. It accepts 16 kHz single-channel audio and outputs a floating-point tensor of speaker-activity probabilities at a configurable time step. Developers can combine its timestamps with automatic speech recognition to produce speaker-attributed transcripts, with recommended input-buffer latencies ranging from 0.32 to 30.4 seconds.

## Context

Speaker diarization classifies who spoke when in conversations, enabling speaker-attributed transcripts for meetings, customer calls, and podcasts. Without it, transcripts lack speaker attribution, reducing the usefulness of search, summaries, action items, and voice-agent memory.

## Approach / What changed

The model uses a Sortformer architecture with arrival-order speaker labeling, a 31-layer Transformer encoder with rotary positional embeddings, and two memory mechanisms (Arrival-Order Speaker Cache and FIFO queue) to support both offline and streaming diarization for up to eight speakers.

## Takeaways

- Nemotron 3 Diarization achieves a 14.72% DER on VoiceArena’s Diarization-Bench, a ~24% relative reduction over the next-ranked system.
- The model uses arrival-order speaker labeling to maintain stable speaker channels across streaming chunks, avoiding the need to solve new speaker permutations per chunk.
- Adding licensed David AI multi-speaker data to training decreased compound DER by 0.77 absolute points, from 11.19% to 10.42%.

## FAQ

### How many speakers does Nemotron 3 Diarization support?

The model supports up to eight speaker channels. If a recording contains more speakers, speech can be missed or assigned to the wrong channel.

### Does the model identify specific people or just label speakers?

The model uses anonymous labels (e.g., speaker\_2) based on arrival order, not real-world identities. Downstream applications must map these to explicit identities using meeting metadata, user profiles, or active speaker verification models.

### What latency configurations are available for streaming?

The model supports input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds. Shorter buffers reduce latency but generally lower accuracy and throughput. The 0.32-second configuration is the lowest recommended.

**Tags:** [AI](https://yomu.fyi/topic/ai), [ASR](https://yomu.fyi/topic/asr), [NVIDIA](https://yomu.fyi/topic/nvidia), [Real-time](https://yomu.fyi/topic/real-time), [Speaker Diarization](https://yomu.fyi/topic/speaker-diarization)

- Source: [Hugging Face](https://huggingface.co/blog/nvidia/nemotron-diarization)
- Source URL: https://huggingface.co/blog/nvidia/nemotron-diarization
- Ingested by Yomu: 2026-09-23T15:00:09.574Z

[Read original post](https://huggingface.co/blog/nvidia/nemotron-diarization)