Two speech models can tie overall and serve different audiences

The new Open ASR Leaderboard update adds private conversational English splits from Egyptian and Nigerian speakers, along with broader accent breakdowns. [1] That makes this a good moment to ask what an aggregate score is weighting. My numerical example is synthetic; it is not a measurement of those datasets or any named speech model.

Imagine choosing a speech recognizer from a row labeled 7.5% word error rate. Two systems share that number, so the shortlist looks settled.

In the invented scorecard below, one system makes many more errors on the smaller group. The overall tie hides that difference completely.

Computed illustration, not leaderboard results. A and B are fictional systems; group sizes are assumed reference-word counts. Image by the author.

Follow the denominator

Word error rate, or WER, divides substitutions, deletions and insertions by the number of words in the reference transcript. I use “errors” below for that summed edit count. [3]

Suppose group one contains 900 reference words and group two contains 100. System A makes 45 and 30 errors; system B makes 63 and 12.

Both have 75 errors across 1,000 reference words. Their pooled WER is therefore 7.5%.

But the per-group views are different:

Fictional system	Group one	Group two	Pooled WER	Equal-group mean
A	5.0%	30.0%	7.5%	17.5%
B	7.0%	12.0%	7.5%	9.5%

A is better on the larger group. B is better on the smaller group. Pooling gives each reference word equal weight, so the larger group contributes more to the result.

The equal-group mean, often called a macro average, answers a different question: give each of these two groups the same weight, regardless of its word count. Neither definition is secretly the one correct summary. The denominator needs to match the question.

Run the arithmetic

This uses Python's standard library. Save it as scorecard.py and run python3 scorecard.py:

from fractions import Fraction

# Invented error counts and reference-word counts, not model measurements.
reference_words = (900, 100)
errors = {'A': (45, 30), 'B': (63, 12)}
for model, counts in errors.items():
    rates = [Fraction(e, n) for e, n in zip(counts, reference_words)]
    pooled = Fraction(sum(counts), sum(reference_words))
    macro = sum(rates) / len(rates)
    print(f'{model}: groups={float(rates[0]*100):.1f}%,{float(rates[1]*100):.1f}% '
          f'pooled={float(pooled*100):.1f}% macro={float(macro*100):.1f}%')
    assert pooled == Fraction(3, 40)

It prints:

A: groups=5.0%,30.0% pooled=7.5% macro=17.5%
B: groups=7.0%,12.0% pooled=7.5% macro=9.5%

I checked the sums and both averaging rules separately with exact fractions. No audio was transcribed, no speech model was called, and no existing model's error count was substituted into this example.

These are invented word counts, not counts of speakers, recordings or users. A dataset with more words need not represent more people. The example also does not reconstruct the live leaderboard's aggregation scheme.

What the new breakdown makes easier to ask

The release describes spontaneous conversational recordings, with cased, punctuated transcripts and disfluencies. It adds US versus non-US groupings, and Inner Circle versus Outer & Expanding Circle groupings based on Kachru's framework. [1] Those broad categories are useful views, while more detailed per-accent results can expose differences within them.

Before using a ranking for an application, I would check the reference conventions and normalization, the sample sizes behind each group, and whether the relevant speakers resemble the people who will use the system. Private test data can reduce direct tuning against visible test examples; it also means an outside reader cannot simply download the split and rerun every transcription.

Nadine's Nigerian-speech evaluation write-up raises another part of the scoring contract: whether dialect forms are transcribed faithfully or normalized into standard spellings. [2] Its reported model scores are not reused here. A careful reference policy matters alongside the averaging rule.

An aggregate is useful for comparing the declared mixture. A subgroup scorecard helps explain who contributes to that mixture and where the errors sit. I would keep both views visible when selecting a recognizer.

References

1. Eric Bezzam, Ashley Mo and Aoi Otani, Egyptian and Nigerian English join the Open ASR Leaderboard.

2. Nadine, Whisper Keeps Correcting Nigerian Speech. Here's How I Measured It.

3. Hugging Face Evaluate contributors, WER metric implementation and definition.

Asset credits

Original chart from the stated synthetic counts and checked arithmetic. It contains no private audio, actual benchmark transcripts or third-party charts.

AI disclosure

AI was used to draft the article.