Engineering

Speaker diarization explained: how AI knows who said what

By the exma team · September 13, 2026 · 8 min read

Two different problems wearing one transcript

"Transcribe this recording" is really two tasks bundled together. The first is automatic speech recognition (ASR): converting audio into words. The second is diarization: figuring out which words belong to which speaker. A transcript without diarization reads as one continuous block of text — every speaker's words run together with no indication of who said what. Add diarization and the same audio becomes a labeled dialogue: Speaker 1, Speaker 2, back and forth, which is what makes a multi-person recording actually readable as a transcript rather than a wall of text.

These are scored differently, too. ASR accuracy is measured by word error rate (WER) — the percentage of words inserted, deleted, or substituted incorrectly, which we cover in speech to text, explained. Diarization has its own metric, described below, and a system can do well on one while doing poorly on the other: get every word right and still put half of them under the wrong speaker.

How diarization actually works

At a high level, modern diarization systems do three things:

1. Segment the audio into short windows, typically a second or two long.
2. Embed each window into a numeric vector — a "voiceprint" that captures characteristics like pitch, timbre, and cadence, independent of what words were spoken.
3. Cluster those embeddings to decide how many distinct voices are present and which windows belong to the same speaker.

A batch system that processes a whole recording at once can look at all the embeddings together before deciding on clusters. A real-time system, streaming audio live, has to make that call incrementally — deciding, as each new segment arrives, whether it matches a speaker cluster already seen in this session or represents a new speaker, without the benefit of hearing the rest of the conversation first. That's a meaningfully harder version of the same problem, which is why live diarization in a deposition or a multi-party call is a genuine engineering feat, not just transcription with a label attached.

Diarization error rate: the metric that actually matters

Diarization error rate (DER) is the standard way to score diarization accuracy, expressed as the percentage of total audio time that's mislabeled. It combines three error types:

Speaker confusion is the error that matters most in practice — it's the one that puts a real, correctly transcribed sentence under the wrong name in a deposition or interview transcript, which is a substantive problem even though every word is correct.

What actually breaks diarization

• Overlapping speech. When two people talk at once, most systems assign that moment to one speaker — interruptions and fast back-and-forth are where error concentrates. A well-built system flags overlap rather than silently guessing.
• Similar-sounding voices. Two speakers with close pitch and timbre are harder to separate into distinct clusters, especially over a short recording with limited data per speaker.
• Phone and low-bandwidth audio. Compression and narrow frequency bands strip some of the vocal detail the embedding relies on.
• Many speakers, short turns. A roundtable with quick, single-sentence turns gives the clustering step less signal per speaker to work with than a two-person interview.
• Speaker rejoining after a long gap. If someone leaves and returns much later, or a new person joins mid-session, the system has to decide whether it's a known cluster or a new one — long gaps make that harder.

"Speaker 1" vs. an actual name

Diarization on its own tells you that Speaker 1 and Speaker 2 are different people — it doesn't know their names. Getting from "Speaker 1" to "Judge Alvarez" requires either a human labeling the transcript (usually done once, in the first exchange, and applied to the rest) or speaker enrollment: a reference voice sample provided in advance that the system matches new audio against. Enrollment is common in settings with the same recurring speakers — a courtroom with a regular judge, a company's recurring meeting participants — and much less practical for one-off interviews or depositions with people the system has never heard before.

Where diarization quality actually shows up

For a single-speaker dictation, diarization barely matters. For everything else it's the difference between a usable transcript and a wall of text: depositions and multi-party interviews (see what makes a transcript court-admissible), insurance recorded statements with an adjuster and a claimant, police interviews with multiple people in the room, and any live courtroom proceeding with a judge, counsel, and witnesses all speaking in turn — covered in real-time speech-to-text in the courtroom. In all of these, "what was said" is only half the record; "who said it" is the other half, and it's the half diarization is responsible for getting right.

Frequently asked questions

What is speaker diarization?

The process of splitting audio into segments and labeling who was speaking in each one — "who spoke when" — separately from transcription, which handles "what was said."

How does speaker diarization actually work?

Audio is split into short windows, each converted into a voice embedding, and those embeddings are clustered to determine how many speakers are present and which segments belong to each. Real-time systems do this incrementally as audio streams in.

What is diarization error rate (DER)?

The standard accuracy metric: the percentage of audio time mislabeled, combining missed speech, false-alarm speech, and speaker confusion — separate from word error rate, which scores transcription.

Why does diarization struggle with overlapping speech?

Two voices in the same time window usually get assigned to one speaker by most systems, which is why interruptions and cross-talk are where diarization error concentrates.

Can diarization identify speakers by name?

Only with enrollment — a reference voice sample provided in advance. Without it, diarization distinguishes speakers from each other but relies on a human to label which one is which.

This article is general technical information. Diarization accuracy varies by vendor, audio quality, and number of speakers — evaluate any tool on audio representative of your actual use case before relying on it for a formal record.