The words can be right while the meeting is wrong

“Please send the report.”

“Cancel the meeting now.”

Eight ordinary words. A transcription system can get every one right and still give the wrong person credit for saying them. That is an awkward feature for a meeting assistant. Especially when the person who said “cancel” gets assigned the action item. 🙃

NVIDIA’s Nemotron 3.5 Transcribe launch brings transcription and speaker attribution into the same checkpoint. Its release discusses word error rate, diarization error rate and concatenated minimum-permutation word error rate. Those numbers answer different questions. [1]

I built a tiny scoring example to make that difference visible. It uses invented text, not model-generated transcripts. The result is zero chronological word errors and 50% speaker-attributed word error for the same eight words.

Original diagram of the worked example below. These are synthetic transcripts, not Nemotron outputs or benchmark scores.

Give the words to the wrong people

The reference has two speakers:

A: please send the report
B: cancel the meeting now

Now imagine the recognized words stay in their correct chronological order, but the speaker labels switch around the ends of the turns. After collecting each predicted speaker’s words, the hypothesis becomes:

X: please send the meeting now
Y: cancel the report

The word “report” has moved to Y. “Meeting now” has moved to X. No word disappeared from the overall recording. The attribution changed.

If I flatten the recognized words in their original time order and compare them with the spoken sequence, the word error rate is zero. That comparison has deliberately thrown away the speaker labels.

This is a useful transcription check. It is an incomplete meeting check.

Let the names change. Keep the assignments honest.

Speaker labels are usually arbitrary. A system might call the first person speaker_0, Alice or X. A scorer should not punish a correct transcript just because A became Y.

Concatenated minimum-permutation WER, or cpWER, addresses that naming problem. Collect each speaker’s words in order, try assignments between reference speakers and predicted speakers, and take the assignment with the fewest word edits. Divide that edit count by the total number of reference words. [2]

For this example, A→X and B→Y costs four edits:

• A’s “report” becomes “meeting now”: one substitution and one insertion.
• B’s “meeting now” becomes “report”: one substitution and one deletion.

The other speaker assignment costs more. The minimum is therefore 4 / 8 = 50%.

Here is the complete little scorer. It exhaustively tries permutations, which is convenient for two speakers and a terrible plan for a large meeting. MeetEval uses an assignment algorithm for the practical version. [2]

from itertools import permutations
import json
def edits(a, b):
    previous = list(range(len(b) + 1))
    for i, word in enumerate(a, 1):
        row = [i]
        for j, other in enumerate(b, 1):
            row.append(min(row[-1]+1, previous[j]+1, previous[j-1]+(word != other)))
        previous = row
    return previous[-1]
def cpwer(reference, hypothesis):
    # Exact exhaustive assignment is suitable for this tiny illustration only.
    n = max(len(reference), len(hypothesis))
    refs = [x.split() for x in reference] + [[]] * (n-len(reference))
    hyps = [x.split() for x in hypothesis] + [[]] * (n-len(hypothesis))
    words = sum(map(len, refs))
    if not words:
        raise ValueError('This example requires reference words')
    error = min(sum(edits(r, h) for r, h in zip(refs, order)) for order in permutations(hyps))
    return {'errors': error, 'reference_words': words, 'rate': error/words}
reference = ['please send the report', 'cancel the meeting now']
wrong_speakers = ['please send the meeting now', 'cancel the report']
spoken_order = 'please send the report cancel the meeting now'.split()
print('Chronological word error:', edits(spoken_order, spoken_order), '/', len(spoken_order))
print('Wrong speaker attribution:', json.dumps(cpwer(reference, wrong_speakers)))
print('Only speaker names swapped:', json.dumps(cpwer(reference, reference[::-1])))
print('Second speaker missing:', json.dumps(cpwer(reference, reference[:1])))
assert cpwer(reference, wrong_speakers)['rate'] == .5
assert cpwer(reference, reference[::-1])['rate'] == 0
assert cpwer(reference, reference[:1])['rate'] == .5
assert edits('a b'.split(), 'a x b'.split()) == 1
assert edits([], ['a']) == 1
try:
    cpwer([], [])
except ValueError:
    pass
else:
    raise AssertionError('Empty reference must be explicit')

The important lines of output are:

Chronological word error: 0 / 8
Wrong speaker attribution: {"errors": 4, "reference_words": 8, "rate": 0.5}
Only speaker names swapped: {"errors": 0, "reference_words": 8, "rate": 0.0}
Second speaker missing: {"errors": 4, "reference_words": 8, "rate": 0.5}

I checked this implementation against MeetEval 0.4.3 on the three displayed speaker cases and 30 additional generated cases. The edit counts and reference-word denominators matched. That checks the scoring code; it says nothing about an ASR model’s accuracy.

The label-swap control matters. If swapping the two names alone made the score worse, the scorer would be measuring the spelling of labels rather than who said what.

Read the leaderboard with the task still attached

NVIDIA reports 5.33% average English WER on the Open ASR comparison and roughly 32.4% cpWER on VoiceArena’s full-session comparison. These come from different evaluation setups. Dividing one by the other would not measure a speaker-attribution penalty. [1]

The same distinction applies to diarization error rate, or DER. That is a speaker-time measure. cpWER counts word edits after assigning speaker streams. Neither is a universal conversion of the other.

For a meeting product, I would keep several checks beside each other: words, speaker-attributed words, missed speakers, and an actual listen to the difficult turns. A transcript can read beautifully while assigning an objection, promise or decision to the wrong person.

Common mistake: treating a strong plain-WER result as proof that speaker-labelled minutes are reliable. The denominator may be words, but the product failure is often attribution.

My toy scorer also ignores timestamps and overlap. A speaker can receive the right words in the wrong temporal arrangement without this particular check telling the whole story. Text normalization needs an explicit policy too: punctuation, numbers and domain terms can change an edit count before speaker assignment even starts.

The release is worth trying for its combined speech tasks. The evaluation lesson is useful before downloading any checkpoint: decide whether the application needs the words, the speaker timeline, or both—and score that actual output.

References and assets

[1] NVIDIA’s Nemotron 3.5 Transcribe and Granary v2 release. Published model results are NVIDIA’s reported results; I did not rerun its inference or leaderboard evaluations.

[2] MeetEval’s cpWER implementation. The independent check used MeetEval 0.4.3.

Diagram and scoring example are original. No recorded conversation or personal data was used.

AI disclosure

AI was used to draft the article.