whisper.cpp v1.9.5, one recording, four builds (x86 CPU, Intel iGPU, Arm CPU, RTX Spark CUDA): medium was faster on both CPUs, turbo was faster or tied on both GPUs, and the per-phase timings had the same shape on all four.
This is an English adaptation of an article I originally published in Japanese on Zenn: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.
Short version: on an x86 CPU and an Arm CPU, medium was faster than large-v3-turbo; on two different GPUs, turbo was faster (or tied in total), and the per-phase timings explained the flip the same way on all four.
A 67-second video summary (narration is synthetic, English captions available):
TL;DR
- Same recording, same arguments, whisper.cpp v1.9.5, four builds: x86 CPU (AVX2), x86 integrated GPU (Intel Arc 140V, Vulkan), Arm CPU (NEON), Arm GPU (RTX Spark N1X, CUDA).
- medium was faster on both CPUs (1.27× and 1.43×). turbo was faster on CUDA (by 257ms). On the integrated GPU the totals were a tie (15ms apart), but turbo was faster once model loading is excluded.
- The breakdown had the same shape on all four: turbo gains in the decoder and loses in the encoder, and which is larger depends on the hardware.
- Building whisper.cpp with MSVC on Windows on Arm didn't use NEON and was 5.6–10.2× slower than a clang build. One line of
system_infoshows it. - The faster the GPU, the more of the total is model loading, which hides the compute difference. On CUDA, 1,118ms of turbo's 1,537ms was loading.
Scope
-
What: transcribing one short Japanese clip with whisper.cpp, comparing
large-v3-turbo(fp16),medium, and turbo's q5_0 quantization. - On what: an x86 laptop (Intel Core Ultra 7 258V with the Arc 140V integrated GPU) and a Surface Laptop Ultra (Windows 11 Arm64, NVIDIA RTX Spark N1X). One CPU build and one GPU build on each.
-
Measured: CLI total time and per-phase time (load, encode, batchd), averaged over runs 2–5; round-trip time to a resident server;
-t 8vs.-t 20on the Arm CPU. - Not measured: see "Where this stops" near the end.
- Versions are as recorded in October 2026, when I wrote the original.
This continues an earlier article (in Japanese): whisper の turbo と medium、速いほうが CPU と内蔵 GPU で入れ替わった. That one covers the method (same recording, same arguments, each condition run 5 times in a row with the first run discarded) and how to read the phase timings. There, on one PC, medium was faster on the CPU and turbo was faster on the integrated GPU. This time I added a second PC with a different CPU architecture and a different GPU family.
The result first
whisper.cpp at v1.9.5 on both machines, same recording, same arguments. Average of runs 2–5, in ms.
| Model | x86 CPU | x86 iGPU | Arm CPU | Arm CUDA |
|---|---|---|---|---|
| large-v3-turbo (fp16) | 13,192 | 2,393 | 13,873 | 1,537 |
| medium | 10,370 | 2,408 | 9,698 | 1,794 |
| large-v3-turbo (q5_0) | 12,021 | 1,371 | 7,108 | 795 |
| Faster of turbo and medium | medium (1.27×) | tie (15ms apart) | medium (1.43×) | turbo (by 257ms) |
- On both CPUs, medium was faster. x86 uses AVX2 and Arm uses NEON: different instruction sets.
- On the GPUs, turbo was faster. Only the integrated GPU (Vulkan) came out a tie in total on this version. As I noted at the end of the previous article, the spread across the 4 runs overlaps; excluding loading, turbo is 188ms faster.
- The transcription matched the correct text in every condition.
The previous comparison was within one PC, so I couldn't tell whether the explanation "where the encoder is heavy, the model with the smaller encoder wins" was specific to that PC. The same direction showed up on a CPU with a different instruction set and on a different family of GPU, so the explanation now covers a little more ground.
The setup
Both machines used the same recording (Japanese speech recorded with the watch's microphone, 6.635 s, 16kHz, mono, 16-bit) and the same arguments:
whisper-cli -m <model> -l ja -t <threads> -bs 5 -bo 5 -f sample.wav -otxt -nt
| x86 PC | Arm PC | |
|---|---|---|
| Machine | Laptop with Intel Core Ultra 7 258V | Surface Laptop Ultra (Windows 11 Arm64) |
| CPU | 8 cores, 8 threads (AVX2) | Arm cores of the NVIDIA RTX Spark N1X (20 threads) |
| GPU | Integrated Intel Arc 140V (Vulkan build) | RTX Spark N1X GPU (6144 cores, Blackwell, CUDA 13.4 build) |
| CPU build | MSVC, AVX2. No BLAS | clang 19.1.5 (arm64-windows-llvm toolchain), NEON. No BLAS |
| Threads | -t 8 |
-t 8 (20-thread results later) |
- I built whisper.cpp v1.9.5 myself on both machines, from the same commit. The x86 numbers in the previous article were 1.8.4, and the CPU side there was the official prebuilt package with OpenBLAS. This time the x86 CPU build has no OpenBLAS, so the previous CPU column differs in two ways: version and BLAS. Please read the previous table (1.8.4 vs. 1.8.4) and this 4-column table as separate comparisons.
- Without BLAS, the encoder is clearly slower. On the same PC, turbo's encode went from 7,700ms with 1.8.4 (OpenBLAS) to 11,235ms with v1.9.5 (no BLAS). The Arm CPU build has no BLAS either, so the two CPU columns are matched on "no BLAS".
- The Arm build needed clang's OpenMP runtime (
libomp140.aarch64.dll) at run time. I used the debug, non-redistributable copy bundled with Visual Studio, so this is for testing only and can't be redistributed. I'm not distributing the built executables or DLLs.
On Windows on Arm, MSVC doesn't use NEON
At first I built the Arm CPU version with MSVC. These are its numbers:
| Model | MSVC build | clang build | Ratio |
|---|---|---|---|
| large-v3-turbo (fp16) | 84,531ms | 13,873ms | 6.1 |
| medium | 53,894ms | 9,698ms | 5.6 |
| large-v3-turbo (q5_0) | 72,145ms | 7,108ms | 10.2 |
6.6 seconds of audio took 84 seconds. One line of system_info shows why:
# MSVC build (no NEON)
system_info: n_threads = 8 / 20 | ... | CPU : ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |
# clang build (NEON = 1 appears)
system_info: n_threads = 8 / 20 | ... | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |
With the MSVC I used, the list of predefined macros (cl /PD) had no __ARM_NEON. ggml's CPU backend switches code paths on whether __ARM_NEON is defined, so an MSVC build runs matrix multiplication and the like in generic C code. When I tried to work around it by passing /D__ARM_NEON=1 from outside, compilation failed in the NEON helper functions. (At least in v1.9.5, that means ggml's NEON path can't be used with MSVC.)
The whisper.cpp repository has a clang toolchain file for Windows on Arm, and specifying it gave NEON = 1. Other things I tried on the MSVC build, such as turning off flash attention or enabling llamafile's sgemm, made no difference.
Look at system_info before measuring speed. That's all it is, but if I had only looked at the totals, I would have stopped at "the Arm CPU is slow." As far as I found, when building this kind of library on Windows on Arm, the choice of compiler seems to make an order-of-magnitude difference.
By phase, all four had the same shape
From the phase timings whisper.cpp prints on exit, I took the encoder (encode) and the decoder side (batchd) and lined up turbo minus medium. Positive means turbo loses; negative means turbo gains.
| Environment | encode diff | batchd diff | Total diff |
|---|---|---|---|
| x86 CPU | +4,604ms | −1,653ms | +2,822ms (turbo slower) |
| Arm CPU | +5,267ms | −1,067ms | +4,176ms (turbo slower) |
| x86 iGPU | +312ms | −455ms | −15ms (tie) |
| Arm CUDA | +24ms | −391ms | −257ms (turbo faster) |
turbo is large-v3 with the decoder cut from 32 layers to 4. The encoder stays the same as large-v3: 32 layers, 1280 dimensions (medium has 24 layers, 1024 dimensions). So turbo gains on the decoder side and loses in the encoder. Which of the two is larger flips with the environment.
On the CPUs the encoder loss is large. On the GPUs the encoder gap shrinks, so the decoder gain wins. On CUDA the encoder gap was down to 24ms, and the cost of turbo's larger encoder was barely visible.
The raw breakdown (-t 8, average of runs 2–5, ms):
| Environment | Model | load | encode | batchd | Total |
|---|---|---|---|---|---|
| x86 CPU | turbo fp16 | 1,268 | 11,235 | 519 | 13,192 |
| x86 CPU | medium | 1,223 | 6,631 | 2,171 | 10,370 |
| Arm CPU | turbo fp16 | 1,186 | 12,005 | 564 | 13,873 |
| Arm CPU | medium | 1,146 | 6,738 | 1,631 | 9,698 |
| x86 iGPU | turbo fp16 | 1,230 | 720 | 309 | 2,393 |
| x86 iGPU | medium | 1,057 | 408 | 763 | 2,408 |
| Arm CUDA | turbo fp16 | 1,118 | 130 | 147 | 1,537 |
| Arm CUDA | medium | 974 | 106 | 538 | 1,794 |
The faster the GPU, the more the total is about model loading
On CUDA, 1,118ms of turbo's 1,537ms total is loading; only 419ms is computation. Re-ranked with loading excluded:
| Environment | turbo fp16 | medium | Difference |
|---|---|---|---|
| x86 CPU | 11,924ms | 9,147ms | medium faster by 2,778ms |
| Arm CPU | 12,687ms | 8,551ms | medium faster by 4,136ms |
| x86 iGPU | 1,163ms | 1,352ms | turbo faster by 188ms |
| Arm CUDA | 419ms | 820ms | turbo faster by 401ms |
Both the integrated GPU, which looked like a tie in total, and CUDA, where loading is most of the time, show a clear order once loading is taken out. It also means the same models can rank differently depending on how you run them: starting a process every time (CLI) versus keeping one running and sending requests to it (resident server).
I also measured the round trip of sending the same recording over HTTP to a resident server. Loading then happens only once, at startup.
| Server setting | Model | x86 iGPU | Arm CUDA |
|---|---|---|---|
| Default (greedy) | large-v3-turbo | 867ms | 202ms |
| Default (greedy) | medium | 921ms | 473ms |
-bs 5 (same width as the CLI) |
large-v3-turbo | 1,269ms | 340ms |
-bs 5 (same width as the CLI) |
medium | 1,501ms | 724ms |
As a resident server, turbo was faster on both GPUs. Compared with the integrated GPU, CUDA returned in roughly a quarter of the time for turbo and roughly half for medium. Note that whisper-server defaults to greedy decoding when nothing is specified, while whisper-cli defaults to beam search with width 5: the defaults are opposite. (Last time, this left my conditions mismatched.)
The benefit of quantization (q5_0) shrank as the hardware got faster
As a third model, I measured turbo's q5_0 (quantized) version under the same conditions.
| Environment | q5_0 total | Excluding loading | vs. fp16 (excluding loading) |
|---|---|---|---|
| x86 CPU | 12,021ms | 11,525ms | q5_0 faster by 400ms |
| Arm CPU | 7,108ms | 6,690ms | q5_0 faster by 5,997ms |
| x86 iGPU | 1,371ms | 944ms | q5_0 faster by 219ms |
| Arm CUDA | 795ms | 404ms | q5_0 faster by 15ms |
-
On the Arm CPU, q5_0 was the fastest of the three models, and faster than the x86 CPU. This clang build has no fp16 vector arithmetic (
HAVE_FP16_VECTOR_ARITHMETICisn't set), while the integer instructions are available. My guess is that this is why the two fp16 models were slow, but that's a guess; I didn't measure which operations were used or how much. - On CUDA, q5_0's advantage was almost entirely lighter loading (a smaller file). The compute time differed from fp16 by only 15ms.
- Quantization tends to get summed up as "faster," but as far as I found, the compute benefit seems to shrink on faster hardware, leaving mostly the lighter load.
More threads shrink the encoder, not the decoder side
whisper.cpp sees 20 threads on the Arm PC (n_threads = 8 / 20 in the output), so I measured both -t 8 and -t 20.
| Model |
-t 8 total |
-t 20 total |
encode | batchd |
|---|---|---|---|---|
| large-v3-turbo (fp16) | 13,873ms | 11,015ms | 12,005 → 9,004ms | 564 → 579ms |
| medium | 9,698ms | 7,478ms | 6,738 → 4,162ms | 1,631 → 2,069ms |
| large-v3-turbo (q5_0) | 7,108ms | 4,448ms | 6,326 → 3,714ms | 276 → 226ms |
Every total went down, but not evenly inside. The encoder is one call that passes the whole audio through at once, so it shrinks cleanly. The decoder side is more than 160 small calls, so it shrinks less, and for medium it actually grew. More threads don't make everything uniformly faster; that's what this table supports.
On variance: turbo fp16 at -t 20 ranged from 10,294 to 11,674ms (13%) even across runs 2–5, wider than the other conditions (a few %). If you measure only the total once and compare, any difference smaller than that spread disappears.
Where this stops
- I didn't compare GPUs on the same backend. I didn't build a Vulkan version on the Arm side, so the Arc 140V (Vulkan) and the RTX Spark N1X (CUDA) differ in both backend and generation. This is not a measurement that can say which GPU is faster.
-
The two CPU columns are matched on "no BLAS", but not on compiler and flags (x86: MSVC, AVX2; Arm: clang,
-march=armv8.7-aand others). - These are measurements on one machine each, so I can't claim more than the same direction showed up on these two machines. It could reverse on other CPUs or GPUs.
- The measurement uses only the same single sample as last time (6.6 s, one sentence, one speaker). The ratios could change with long audio, noisy audio or multiple speakers; I haven't checked.
- The transcription matched the correct text in every condition. I didn't compare accuracy (this clip shows no difference, so it can't support any accuracy ranking).
- The clang OpenMP runtime used for the Arm build is the debug (non-redistributable) copy bundled with the development tools. I'm not distributing the built executables or DLLs.
Takeaways
- The previous result, medium on the CPU and turbo on the GPU, showed the same direction on a CPU with a different instruction set and on a different GPU family. (On the integrated GPU, v1.9.5 gave a tie in total; excluding loading, turbo is faster.)
- The reason was the same too: turbo cuts the decoder but keeps the large encoder. Where the encoder is heavy, the loss wins; on the GPU, the gain wins.
- The faster the GPU, the larger the share of model loading in the total. Look at the phases, not the total, or keep the model resident and measure without loading.
- On Windows on Arm, the compiler used for the build changed the result by an order of magnitude. Check the
system_infoline first.
whisper.cpp is licensed under MIT (ggml-org/whisper.cpp). The phase names come from the comments on the counters in the source.
About this series
This is part of my series on a DIY ESP32-S3 smartwatch. The watch sends recorded speech to a PC, which transcribes it; this article and the previous one compare models for that step. The Arm PC here is the same Surface Laptop Ultra I've written about in English before. Related articles:
- whisper の turbo と medium、速いほうが CPU と内蔵 GPU で入れ替わった。区間に分けて測る (Japanese): the previous article, on one PC
- 電子工作素人の ESP32-S3 スマートウォッチ自作奮闘記 (Japanese): the series entry point
- Moving ComfyUI from Intel Arc to Arm WSL2: The Same requirements.txt Didn't Install the Same Versions (English): same Surface
- Ollama vs. llama.cpp on the Same GGUF: The 15% Speed Gap Was the Defaults, Not the Engine (English): same Surface
The original of this article: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった.
Top comments (0)