Disclosure: I build IraVoice, a commercial Mac dictation app. IraVoice currently ships Apple's recognition path, so this comparison is not independent product-review coverage. The purpose of publishing it is to make a narrow engineering result inspectable instead of turning it into a universal accuracy claim.
Speech-recognition comparisons often collapse several different questions into one number: Which corpus was used? Was audio streamed or uploaded as a batch? Was punctuation normalized? Did a cleanup model rewrite the output? Was latency measured from the first byte, the last audio frame, or the final transcript?
We wanted a smaller question that we could answer and reproduce:
On one disclosed Apple-silicon Mac, how do Apple's current SpeechAnalyzer API and the whisper.cpp
small.enmodel compare on the same deterministic clean-English snapshot when both are exercised through IraVoice's production recognition adapters?
The full benchmark, method, downloads, and limitations are public. This post explains the decisions behind the test and what the result does—and does not—support.
The result in one table
| Engine | WER | CER | Median post-speech latency | p95 post-speech latency | Completed clips |
|---|---|---|---|---|---|
| Apple SpeechAnalyzer | 1.98% | 1.02% | 125–132 ms | 194–201 ms | 40/40 per run |
whisper.cpp 1.8.4, ggml-small.en
|
4.28% | 1.79% | 122–125 ms | 152–161 ms | 40/40 per run |
Those latency ranges are the minimum and maximum statistic across two complete runs per engine. They are not confidence intervals, and they are not selectively chosen fastest runs.
Across the same 607 reference words, Apple recorded 12 word edits and whisper.cpp recorded 26. Accuracy was identical across the two repeated runs for each engine. No measured clip failed.
The narrow conclusion is that Apple was more accurate on this particular snapshot. The latency result was close, with whisper.cpp slightly lower on the disclosed post-speech measure. Nothing in this test establishes that Apple is universally more accurate, that either engine is better for every dictation workflow, or that the result transfers to other Whisper models.
How the snapshot was built
The source corpus is LibriSpeech test-clean. We selected one deterministic clip from each of 40 speakers: 20 female and 20 male speakers, with clips between 4 and 10 seconds and references between 8 and 20 words.
That created a deliberately small, speaker-balanced snapshot. It is not the complete 2,620-clip test-clean split.
For reproducibility, the pipeline:
- verifies the official source-archive checksum;
- selects clips from a committed manifest rather than at random during a run;
- converts only the selected clips to 16 kHz mono, 16-bit PCM WAV;
- warms each engine with one unreported clip;
- feeds every measured clip in 100 ms chunks at real-time pace; and
- repeats the complete 40-clip run twice per engine.
The test machine was an Apple M5 Max Mac with 128 GiB of memory, running macOS 26.5 build 25F71 with an English (United States) locale.
Apple was tested through SpeechAnalyzer's progressive transcription path. whisper.cpp was tested through the production batch-server path used by IraVoice, using whisper.cpp 1.8.4 and ggml-small.en.
That difference matters. One engine produces progressive streaming recognition; the other tested path returns a batch transcript. We compared final raw text and a specific interval after the final real-time-paced audio chunk. We did not pretend the interaction models were identical.
What we removed from the product pipeline
The benchmark invokes the same recognition adapters used by the app, but it intentionally excludes product features that would make the engines harder to compare.
There was no custom vocabulary, formatting pass, AI cleanup, digit rewriting, or hand correction. The output scored by the verifier is the raw final transcript returned by each recognition path.
Normalization is explicit: case, punctuation, Unicode width, and diacritics are normalized; letters, digits, and intra-word apostrophes are preserved. Digits are not rewritten as words.
WER and CER are micro-averaged across the reference units. Clip-level substitutions, insertions, deletions, hypotheses, references, and timing are retained rather than publishing only an aggregate score.
This is important because a single WER can hide different failure modes. A developer evaluating a recognizer for commands may care more about substitutions in short phrases than about the aggregate edit rate. The 160 clip-run CSV rows and complete four-run JSON make those differences inspectable.
Latency needs a boundary
“Latency” is not useful unless the start and end events are defined.
Here, post-speech latency begins after the final real-time-paced audio chunk and ends when the tested adapter returns its final transcript. It excludes application startup, model acquisition, UI formatting, any optional cleanup, and text insertion into another app.
Under that definition, Apple produced 125 ms and 132 ms median latency across its two runs, with p95 values of 201 ms and 194 ms. whisper.cpp produced 122 ms and 125 ms medians, with p95 values of 152 ms and 161 ms.
The honest reading is not “Apple won latency.” It did not. The measured ranges were close, and whisper.cpp was slightly lower on this interval. A larger test across more machines and more realistic speech would be needed before making a broader responsiveness claim.
What this benchmark does not prove
The largest limitation is the speech itself. LibriSpeech test-clean is read audiobook speech recorded under relatively clean conditions. It is useful for reproducibility, but it is not everyday desktop dictation.
This study does not establish performance for:
- spontaneous prompts or self-corrections;
- noisy rooms or far-field microphones;
- accents, disabilities, or population-wide fairness;
- domain-specific vocabulary;
- multilingual recognition;
- Intel Macs, other Apple-silicon generations, or other macOS builds;
- other whisper.cpp models or runtime configurations; or
- the complete end-to-end experience of either product path.
It also comes from an interested publisher. IraVoice publishes the study and currently ships Apple's recognition path. The production app source and benchmark runner remain proprietary. The public package contains the complete output artifacts, manifest, checksums, citation files, and scoring-consistency verifier—but not the runner source.
That is why the affiliation and missing runner are stated beside the result rather than hidden in a footnote.
How to audit or reuse it
The benchmark is versioned as 1.0.0 under CC BY 4.0.
The manifest SHA-256 is 171639597cda2a98311fab77b169aa7086e95321f402b50c1f93b597dbd7fc36, and the tested production revision is 81ecbb9582d0990827c76be0e685124d2be651c8.
The included verifier recomputes the published aggregate WER and CER from the clip-level records and checks scoring consistency. That does not reproduce the proprietary runner, but it does let a reviewer confirm that the headline numbers match the released hypotheses, references, and edit counts.
The practical takeaway for Mac developers
The useful lesson is less about declaring a permanent winner and more about constructing an evidence boundary.
If you are choosing a local speech engine for a Mac app, test through the same adapter and audio path you will ship. Publish the model label, machine, OS build, locale, normalization rules, timing boundary, failures, and raw per-clip outputs. Keep formatting or language-model cleanup outside the recognizer comparison unless that combined system is the actual subject of the test.
For this one clean-English snapshot, Apple's current SpeechAnalyzer path produced fewer recognition edits than whisper.cpp small.en, while the disclosed post-speech latency stayed close. That is a useful baseline. It is not a substitute for testing spontaneous speech, noise, vocabulary, accessibility, and the real machines your users have.
Top comments (0)