DEV Community

EveryLocalAI
EveryLocalAI

Posted on

Seven speech models on real news broadcasts: Leaderboard #1 ties a 0.6B model

#ai #machinelearning #performance

The Open ASR Leaderboard has a clear #1: a 30B mixture-of-experts model. Below it sit models a tenth its size.

So, a simple production question: point all of them at real news broadcasts (studio anchors, field reporters, press conferences, background room noise). Does the leaderboard order hold?

It doesn't. On an NVIDIA H100, the seven models land within 0.94 WER points of each other, and a Holm-corrected paired test cannot tell any two apart. What does differ is speed and cost.

The Setup 

Wikimedia Commons news (VOA, White House, Al Jazeera)
   │ ffmpeg once → 16 kHz mono Opus (byte-identical audio)
   ▼
human subtitles ──► 15–30 s windows cut at cue boundaries
   ▼
7 models × same windows · 1× H100 · batch 16 · no language hint
   ▼
corpus WER · 95% CI by bootstrapping videos · paired test vs best
Enter fullscreen mode Exit fullscreen mode

Bootstrapping videos, not individual windows, matters: twelve windows from one briefing share a room and a microphone. Clustered confidence intervals are wider and honest.

Results Summary

Model Released WER (%) 95% CI RTF (Speed) Cost / Audio Hr
Parakeet Unified EN 0.6B 2026-04 7.34% [5.16, 9.97] 596x ~$0.005
Qwen3-ASR 1.7B 2026-06 7.58% [5.53, 9.85] 89x ~$0.034
Whisper large-v3-turbo 2024-10 7.64% [5.10, 10.57] 115x ~$0.026
Parakeet TDT 0.6B v3 2025-08 7.97% [5.88, 10.41] 658x ~$0.005
Nemotron-3.5-Transcribe 30B 2026-10 8.00% [5.85, 10.46] 49x ~$0.061
Granite Speech 5.0 2B 2026-10 8.02% [5.92, 10.47] 13x ~$0.220
freetranscribe parakeet-ft 2026-10 8.28% [6.14, 10.79] 1,117x ~$0.003

Statistical verdict: Smallest Holm-corrected $p = 0.654$. At this sample size (31 broadcasts, 2.6 hours), none of the differences between the #1 model and the 0.6B models are statistically significant.

The Dirty Laundry (What Broke)

  1. Whisper memorized White House transcripts: On crosstalk, Whisper repeatedly outputted The Press-. The Press-. The Press-., matching the exact format of the web transcripts it was trained on. It did this across 7 windows; no other model did.
  2. One long window crashed an entire batch: A cue over 30s crashed Whisper's batched call and failed all 16 windows in it (WER spiked to 15.5% instead of 7.6%). When deploying, always retry failed batches item-by-item.
  3. Reference subtitles vs verbatim transcript bias: Human subtitles edit out stutter words. Verbatim models (Nemotron inserted 628 words, 9% stutter repeats) get billed for faithfulness, while 'cleaner' models matched the human captioner style.

Key Takeaways for Audio Pipelines

  • Don't pay for size on English broadcasts. At ~$3/hr for a spot H100, an hour of news audio costs ~$0.003 with the fastest model and ~$0.06 with the slowest, at virtually identical accuracy.
  • Select by throughput and serving stack: 0.6B to 1.7B models run comfortably on consumer cards with massive throughput headroom.

Full interactive dot-whisker charts, dataset details, and reproduction methodology:
https://everylocalai.com/benchmark/asr-news-broadcasts-2026-10

Top comments (0)