Parakeet v3 vs Whisper vs Apple's on-device model in 25 languages, every transcript published

yapapp — Mon, 26 Oct 2026 09:20:46 +0000

We make Yap, a dictation app that runs NVIDIA's Parakeet v3 on the device (Mac, Windows and phones). Before saying anything about accuracy in our own marketing, we wanted numbers anyone can check, so we ran three on-device speech recognizers on the same recordings, on one Mac, and published every transcript.

What we compared

Parakeet TDT 0.6B v3 (NVIDIA), as Core ML through FluidAudio, the same native recognizer our app ships
Whisper large-v3-turbo (OpenAI), q5_0 quantized, through whisper.cpp with Metal
Apple SpeechTranscriber, the new on-device model in macOS 26, in the six languages it covers here

All three ran on one Mac with an M2 Pro. Nothing was sent to a server.

Method

The recordings are the FLEURS test split at a pinned revision: 100 recordings per language, taken at evenly spaced positions, in the 25 European languages Parakeet v3 supports. Each engine was given the language. Each transcript and the FLEURS reference sentence were normalized the same way (Unicode NFKC, lowercase, words as runs of letters and digits) and compared word by word. Word error rate is substitutions, deletions and insertions over the number of reference words. Every transcript, timing and the SHA-256 of every recording are in the repository, so every number can be recomputed.

Results: word error rate (lower is better)

Language	Parakeet v3	Whisper turbo	Apple (macOS 26)
Bulgarian	13.4%	15.9%	not supported
Croatian	12.4%	11.3%	not supported
Czech	10.4%	11.5%	not supported
Danish	15.8%	14.6%	not supported
Dutch	6.3%	6.6%	not supported
English	4.5%	3.9%	6.7%
Estonian	14.2%	19.9%	not supported
Finnish	10.1%	7.7%	not supported
French	5.3%	6.6%	6.6%
German	4.8%	5.1%	6.9%
Greek	31.7%	12.0%	not supported
Hungarian	14.1%	17.2%	not supported
Italian	2.5%	2.7%	4.2%
Latvian	17.3%	18.6%	not supported
Lithuanian	17.8%	25.0%	not supported
Maltese	22.0%	72.3%	not supported
Polish	8.2%	7.7%	not supported
Portuguese	3.6%	3.3%	5.9%
Romanian	10.0%	11.8%	not supported
Russian	4.7%	4.6%	not supported
Slovak	8.4%	11.5%	not supported
Slovenian	19.8%	23.5%	not supported
Spanish	2.8%	2.5%	4.3%
Swedish	12.0%	9.9%	not supported
Ukrainian	5.7%	6.8%	not supported

The lowest rate in each row is in bold. Parakeet had the lower rate in 15 of 25 languages, Whisper in 10.

What stood out

Greek is Whisper's by a mile (12.0% against 31.7%). If you dictate in Greek, use Whisper.
Maltese and the Baltic languages favor Parakeet: Maltese 22.0% against 72.3%, Lithuanian 17.8% against 25.0%, Estonian 14.2% against 19.9%.
Parakeet beat Apple's new on-device model in all six languages Apple covers here, and Whisper in five: for example German at 4.8% (Parakeet), 5.1% (Whisper) and 6.9% (Apple). In French, Apple edged out Whisper (6.6% against 6.6%, 6.59 to 6.62) while Parakeet led at 5.3%.
Speed: the median language ran at about 20 times real time with Parakeet and 13 times with Whisper turbo on the same Mac. Apple's model was faster still where it applies.

Limits

FLEURS is read Wikipedia sentences in quiet rooms. Real dictation, conversation and noise are harder for every engine.
Numbers and spelling variants are not normalized, so "25" against "twenty-five" counts as an error for all three alike.
One Mac, one run.
We make an app that uses Parakeet. That is why everything needed to check us is public.

Data

Repository with every transcript, timing and recording hash (CC BY 4.0): github.com/founden-bot/on-device-asr-benchmark
The same data with charts by language: yapapp.cc/data/on-device-asr-benchmark

If you rerun it and get different numbers, or spot a mistake in the scoring, open an issue on the repository.