I build Plainsay, an open-source dictation app for the Mac. You hold a key, speak, release, and the text is pasted at your cursor. It can run two speech engines entirely on the Mac: OpenAI's Whisper large-v3-turbo through WhisperKit, and NVIDIA's Parakeet TDT 0.6B v3 through FluidAudio. Both run on Core ML.
When I had to choose a default, I found plenty of claims about which model is "faster" or "more accurate", and very few measurements I could rerun. So I wrote a small harness, committed the audio and the results, and wrote down what it does and doesn't show. Here's how it works and what I learned from reading the errors one by one.
What I wanted to measure
Two things matter for dictation:
- Accuracy, as word error rate (WER): substitutions + deletions + insertions, divided by the number of words in the reference.
- Latency: how long after you stop talking until the text is ready. In push-to-talk dictation you transcribe the whole clip when the key is released, so that wait is what you feel.
I deliberately did not benchmark other dictation apps. Driving someone else's proprietary app in an unattended script isn't something I can do fairly, and I didn't want to republish other vendors' numbers I couldn't reproduce. This measures my own two engines, with enough detail that anyone can check it.
The dataset: a small, disclosed sample of LibriSpeech
I used LibriSpeech test-clean (CC BY 4.0), downloaded from OpenSLR and checked against its published MD5. The full set has 2,620 utterances from 40 speakers. I used a 50-utterance stratified sample: 10 speakers spread evenly through the sorted speaker list (every 4th), and 5 utterances spread across each speaker's first chapter.
That's 1,002 reference words and about 356 seconds of audio. It's small enough that I can read every hypothesis by hand, and it spans 10 different voices. It's also small enough that one or two words move the WER by 0.1 to 0.2 points, which matters later.
The 50 FLAC files and their reference transcripts are committed to the repo, so the benchmark runs without downloading the 346 MB archive.
The harness
The harness is a ~240-line Swift command-line target, Sources/BenchmarkCLI/main.swift. The design choices that matter:
-
Same engine code as the app. It calls
WhisperKitEngineandParakeetEnginefrom the app's core module directly, not a reimplementation. Dependencies were pinned (WhisperKit 1.1.0, FluidAudio 0.15.6). -
Same audio format as the microphone path. Each FLAC is decoded with
AVAudioFileand resampled withAVAudioConverterto 16 kHz mono Float32, which is what the app's recorder produces. All clips are decoded up front, so file I/O isn't timed. -
Only transcription is timed. The clock wraps
transcribe(samples:prompt:)after the engine has been prepared. Model download and Core ML preparation are outside the timed region. -
No hidden warm-up. The first timed clip is the first inference after
prepare(), and the same model instance is reused for all 50 clips. I'd rather include the slow first call than quietly drop it. - No prompt. Both engines run with an empty vocabulary prompt.
- Simple normalization. Before scoring, both the reference and the hypothesis are lowercased, punctuation except apostrophes is stripped, and whitespace is collapsed. LibriSpeech references are bare uppercase words, and Whisper outputs capitalized, punctuated text, so without this you'd count formatting as errors.
- WER is a plain Levenshtein alignment over words.
It writes a JSON file with every reference, hypothesis and per-clip latency.
Results
Run on 2026-08-22, MacBook Air (M4, 10-core CPU, 16 GB), macOS 26.5.1:
| Engine | WER | Avg latency | Real-time factor |
|---|---|---|---|
| Parakeet TDT 0.6B v3 | 2.50% (25 / 1,002) | 0.10 s | 0.014 |
| Whisper large-v3-turbo | 2.89% (29 / 1,002) | 0.68 s | 0.095 |
Neither engine failed on any clip.
The latency gap is the clear result. From the per-clip data, Parakeet's median clip took 0.088 s and its slowest 0.19 s. Whisper's median was 0.64 s and its slowest 1.04 s. The first inference after preparation was 0.13 s for Parakeet and 0.93 s for Whisper, so the cold first call matters a lot more for Whisper. For push-to-talk, Parakeet's result is effectively instant. Whisper's is a short but noticeable pause.
The accuracy gap is not a clear result, and reading the errors shows why.
Reading the errors
With per-clip hypotheses committed, you can see what each counted "error" actually was. A good share of them aren't recognition mistakes:
- British vs American spelling, in both directions. Some references use British spelling (HONOUR, ENQUIRE) and both engines wrote "honor" and "inquire". Other references use American spelling (COLOR, CHECK) and Parakeet wrote "colour" and "cheque". Each counts as a full word error.
- Abbreviations. Whisper wrote "St. Francis" for SAINT FRANCIS, and "Mr Watson" for MISTER WATSON.
-
Quotation marks. On two clips Whisper produced nested quotes (
"'We wish to talk with him,' answered Kenneth. "'Talk!'). My normalizer keeps apostrophes so "don't" stays one word, which means'weandhim'no longer match the reference. The clip above accounts for 4 of Whisper's 29 errors, and quote artifacts cost it 6 in total. - Rare proper nouns. "Margolotte" became "Margolot", "Donnithorne" became "donnythorne" (Whisper) or "Donny Thorne" (Parakeet, two errors), and "Silvia" became "Sylvia" in both engines.
There are real mistakes too. Both engines heard BOOKKEEPER as "bit keeper", and Whisper turned the French word BADAUDERIE into "Bad-Dot-Grey".
The takeaway: on clean read speech, the 0.39-point WER difference between these engines is about the same size as the normalization noise. An earlier committed run (Aug 17) had Parakeet at 23 errors instead of 25 on the same clips. Before you conclude one model is more accurate, look at what the errors are. A more complete normalizer (spelling variants, abbreviations, edge apostrophes, like Whisper's English text normalizer) would shrink both numbers. I haven't committed that yet, so I'm not quoting a re-scored figure.
What this benchmark doesn't tell you
-
It's English only. LibriSpeech
test-cleansays nothing about Polish, German or any other language. Parakeet v3's multilingual coverage and Whisper's language breadth are separate, unmeasured claims. - It's clean audiobook narration. Real dictation means a laptop mic, a room, hesitations, "no wait, I mean". WER will be higher for both engines.
- It skips the capture path. Clips are decoded from files, so microphone behavior and voice-activity handling aren't tested.
- It's one machine. An M4 MacBook Air. Older M1 machines will be slower, especially for Whisper.
- It's a snapshot. The app has since moved to FluidAudio 0.15.8, which fixed dropped endings on some long recordings. Long dictations aren't in this sample at all, since LibriSpeech clips are short.
Why Parakeet ended up the recommended default
For push-to-talk dictation in English, latency decides a lot: a tenth of a second against two-thirds of a second, on clips of a few seconds. Accuracy on this sample is a tie within noise. Whisper is still there for the languages and accents where it does better, and because some people simply prefer its punctuation style.
Run it yourself
git clone https://github.com/conrader/plainsay
cd plainsay
git checkout 2bc1552b19a5ae4094bd81714461baf4ebaa7202
swift build --product BenchmarkCLI
.build/debug/BenchmarkCLI Benchmark/data/librispeech-test-clean-manifest.json
It prints per-clip progress, then WER, latency and real-time factor per engine, and writes results.json next to the manifest. The full method is in BENCHMARK.md, and the committed results are in Benchmark/data/.
If you run it on a different Mac, I'd like to see the numbers, especially from M1/M2 machines. Open an issue or a discussion on the repo. Pull requests that add a better normalizer or a noisier test set are welcome.
Top comments (0)