DEV Community

Cover image for Whisper turbo vs. medium on Arm and CUDA: The Faster Model Still Flipped Between CPU and GPU
maruhana
maruhana

Posted on Edited on Originally published at zenn.dev AI-assisted

Whisper turbo vs. medium on Arm and CUDA: The Faster Model Still Flipped Between CPU and GPU

whisper.cpp v1.9.5, one recording, four builds (x86 CPU, Intel iGPU, Arm CPU, RTX Spark CUDA): medium was faster on both CPUs, turbo was faster or tied on both GPUs, and the per-phase timings had the same shape on all four.

This is an English adaptation of an article I originally published in Japanese on Zenn: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.

Short version: on an x86 CPU and an Arm CPU, medium was faster than large-v3-turbo; on two different GPUs, turbo was faster (or tied in total), and the per-phase timings explained the flip the same way on all four.

A 67-second video summary (narration is synthetic, English captions available):

TL;DR

  • Same recording, same arguments, whisper.cpp v1.9.5, four builds: x86 CPU (AVX2), x86 integrated GPU (Intel Arc 140V, Vulkan), Arm CPU (NEON), Arm GPU (RTX Spark N1X, CUDA).
  • medium was faster on both CPUs (1.27× and 1.43×). turbo was faster on CUDA (by 257ms). On the integrated GPU the totals were a tie (15ms apart), but turbo was faster once model loading is excluded.
  • The breakdown had the same shape on all four: turbo gains in the decoder and loses in the encoder, and which is larger depends on the hardware.
  • Building whisper.cpp with MSVC on Windows on Arm didn't use NEON and was 5.6–10.2× slower than a clang build. One line of system_info shows it.
  • The faster the GPU, the more of the total is model loading, which hides the compute difference. On CUDA, 1,118ms of turbo's 1,537ms was loading.

Scope

  • What: transcribing one short Japanese clip with whisper.cpp, comparing large-v3-turbo (fp16), medium, and turbo's q5_0 quantization.
  • On what: an x86 laptop (Intel Core Ultra 7 258V with the Arc 140V integrated GPU) and a Surface Laptop Ultra (Windows 11 Arm64, NVIDIA RTX Spark N1X). One CPU build and one GPU build on each.
  • Measured: CLI total time and per-phase time (load, encode, batchd), averaged over runs 2–5; round-trip time to a resident server; -t 8 vs. -t 20 on the Arm CPU.
  • Not measured: see "Where this stops" near the end.
  • Versions are as recorded in October 2026, when I wrote the original.

This continues an earlier article (in Japanese): whisper の turbo と medium、速いほうが CPU と内蔵 GPU で入れ替わった. That one covers the method (same recording, same arguments, each condition run 5 times in a row with the first run discarded) and how to read the phase timings. There, on one PC, medium was faster on the CPU and turbo was faster on the integrated GPU. This time I added a second PC with a different CPU architecture and a different GPU family.

The result first

whisper.cpp at v1.9.5 on both machines, same recording, same arguments. Average of runs 2–5, in ms.

Model x86 CPU x86 iGPU Arm CPU Arm CUDA
large-v3-turbo (fp16) 13,192 2,393 13,873 1,537
medium 10,370 2,408 9,698 1,794
large-v3-turbo (q5_0) 12,021 1,371 7,108 795
Faster of turbo and medium medium (1.27×) tie (15ms apart) medium (1.43×) turbo (by 257ms)
  • On both CPUs, medium was faster. x86 uses AVX2 and Arm uses NEON: different instruction sets.
  • On the GPUs, turbo was faster. Only the integrated GPU (Vulkan) came out a tie in total on this version. As I noted at the end of the previous article, the spread across the 4 runs overlaps; excluding loading, turbo is 188ms faster.
  • The transcription matched the correct text in every condition.

The previous comparison was within one PC, so I couldn't tell whether the explanation "where the encoder is heavy, the model with the smaller encoder wins" was specific to that PC. The same direction showed up on a CPU with a different instruction set and on a different family of GPU, so the explanation now covers a little more ground.

The setup

Both machines used the same recording (Japanese speech recorded with the watch's microphone, 6.635 s, 16kHz, mono, 16-bit) and the same arguments:

whisper-cli -m <model> -l ja -t <threads> -bs 5 -bo 5 -f sample.wav -otxt -nt
Enter fullscreen mode Exit fullscreen mode
x86 PC Arm PC
Machine Laptop with Intel Core Ultra 7 258V Surface Laptop Ultra (Windows 11 Arm64)
CPU 8 cores, 8 threads (AVX2) Arm cores of the NVIDIA RTX Spark N1X (20 threads)
GPU Integrated Intel Arc 140V (Vulkan build) RTX Spark N1X GPU (6144 cores, Blackwell, CUDA 13.4 build)
CPU build MSVC, AVX2. No BLAS clang 19.1.5 (arm64-windows-llvm toolchain), NEON. No BLAS
Threads -t 8 -t 8 (20-thread results later)
  • I built whisper.cpp v1.9.5 myself on both machines, from the same commit. The x86 numbers in the previous article were 1.8.4, and the CPU side there was the official prebuilt package with OpenBLAS. This time the x86 CPU build has no OpenBLAS, so the previous CPU column differs in two ways: version and BLAS. Please read the previous table (1.8.4 vs. 1.8.4) and this 4-column table as separate comparisons.
  • Without BLAS, the encoder is clearly slower. On the same PC, turbo's encode went from 7,700ms with 1.8.4 (OpenBLAS) to 11,235ms with v1.9.5 (no BLAS). The Arm CPU build has no BLAS either, so the two CPU columns are matched on "no BLAS".
  • The Arm build needed clang's OpenMP runtime (libomp140.aarch64.dll) at run time. I used the debug, non-redistributable copy bundled with Visual Studio, so this is for testing only and can't be redistributed. I'm not distributing the built executables or DLLs.

On Windows on Arm, MSVC doesn't use NEON

At first I built the Arm CPU version with MSVC. These are its numbers:

Model MSVC build clang build Ratio
large-v3-turbo (fp16) 84,531ms 13,873ms 6.1
medium 53,894ms 9,698ms 5.6
large-v3-turbo (q5_0) 72,145ms 7,108ms 10.2

6.6 seconds of audio took 84 seconds. One line of system_info shows why:

# MSVC build (no NEON)
system_info: n_threads = 8 / 20 | ... | CPU : ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |

# clang build (NEON = 1 appears)
system_info: n_threads = 8 / 20 | ... | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |
Enter fullscreen mode Exit fullscreen mode

With the MSVC I used, the list of predefined macros (cl /PD) had no __ARM_NEON. ggml's CPU backend switches code paths on whether __ARM_NEON is defined, so an MSVC build runs matrix multiplication and the like in generic C code. When I tried to work around it by passing /D__ARM_NEON=1 from outside, compilation failed in the NEON helper functions. (At least in v1.9.5, that means ggml's NEON path can't be used with MSVC.)

The whisper.cpp repository has a clang toolchain file for Windows on Arm, and specifying it gave NEON = 1. Other things I tried on the MSVC build, such as turning off flash attention or enabling llamafile's sgemm, made no difference.

Look at system_info before measuring speed. That's all it is, but if I had only looked at the totals, I would have stopped at "the Arm CPU is slow." As far as I found, when building this kind of library on Windows on Arm, the choice of compiler seems to make an order-of-magnitude difference.

By phase, all four had the same shape

From the phase timings whisper.cpp prints on exit, I took the encoder (encode) and the decoder side (batchd) and lined up turbo minus medium. Positive means turbo loses; negative means turbo gains.

Environment encode diff batchd diff Total diff
x86 CPU +4,604ms −1,653ms +2,822ms (turbo slower)
Arm CPU +5,267ms −1,067ms +4,176ms (turbo slower)
x86 iGPU +312ms −455ms −15ms (tie)
Arm CUDA +24ms −391ms −257ms (turbo faster)

turbo is large-v3 with the decoder cut from 32 layers to 4. The encoder stays the same as large-v3: 32 layers, 1280 dimensions (medium has 24 layers, 1024 dimensions). So turbo gains on the decoder side and loses in the encoder. Which of the two is larger flips with the environment.

On the CPUs the encoder loss is large. On the GPUs the encoder gap shrinks, so the decoder gain wins. On CUDA the encoder gap was down to 24ms, and the cost of turbo's larger encoder was barely visible.

The raw breakdown (-t 8, average of runs 2–5, ms):

Environment Model load encode batchd Total
x86 CPU turbo fp16 1,268 11,235 519 13,192
x86 CPU medium 1,223 6,631 2,171 10,370
Arm CPU turbo fp16 1,186 12,005 564 13,873
Arm CPU medium 1,146 6,738 1,631 9,698
x86 iGPU turbo fp16 1,230 720 309 2,393
x86 iGPU medium 1,057 408 763 2,408
Arm CUDA turbo fp16 1,118 130 147 1,537
Arm CUDA medium 974 106 538 1,794

The faster the GPU, the more the total is about model loading

On CUDA, 1,118ms of turbo's 1,537ms total is loading; only 419ms is computation. Re-ranked with loading excluded:

Environment turbo fp16 medium Difference
x86 CPU 11,924ms 9,147ms medium faster by 2,778ms
Arm CPU 12,687ms 8,551ms medium faster by 4,136ms
x86 iGPU 1,163ms 1,352ms turbo faster by 188ms
Arm CUDA 419ms 820ms turbo faster by 401ms

Both the integrated GPU, which looked like a tie in total, and CUDA, where loading is most of the time, show a clear order once loading is taken out. It also means the same models can rank differently depending on how you run them: starting a process every time (CLI) versus keeping one running and sending requests to it (resident server).

I also measured the round trip of sending the same recording over HTTP to a resident server. Loading then happens only once, at startup.

Server setting Model x86 iGPU Arm CUDA
Default (greedy) large-v3-turbo 867ms 202ms
Default (greedy) medium 921ms 473ms
-bs 5 (same width as the CLI) large-v3-turbo 1,269ms 340ms
-bs 5 (same width as the CLI) medium 1,501ms 724ms

As a resident server, turbo was faster on both GPUs. Compared with the integrated GPU, CUDA returned in roughly a quarter of the time for turbo and roughly half for medium. Note that whisper-server defaults to greedy decoding when nothing is specified, while whisper-cli defaults to beam search with width 5: the defaults are opposite. (Last time, this left my conditions mismatched.)

The benefit of quantization (q5_0) shrank as the hardware got faster

As a third model, I measured turbo's q5_0 (quantized) version under the same conditions.

Environment q5_0 total Excluding loading vs. fp16 (excluding loading)
x86 CPU 12,021ms 11,525ms q5_0 faster by 400ms
Arm CPU 7,108ms 6,690ms q5_0 faster by 5,997ms
x86 iGPU 1,371ms 944ms q5_0 faster by 219ms
Arm CUDA 795ms 404ms q5_0 faster by 15ms
  • On the Arm CPU, q5_0 was the fastest of the three models, and faster than the x86 CPU. This clang build has no fp16 vector arithmetic (HAVE_FP16_VECTOR_ARITHMETIC isn't set), while the integer instructions are available. My guess is that this is why the two fp16 models were slow, but that's a guess; I didn't measure which operations were used or how much.
  • On CUDA, q5_0's advantage was almost entirely lighter loading (a smaller file). The compute time differed from fp16 by only 15ms.
  • Quantization tends to get summed up as "faster," but as far as I found, the compute benefit seems to shrink on faster hardware, leaving mostly the lighter load.

More threads shrink the encoder, not the decoder side

whisper.cpp sees 20 threads on the Arm PC (n_threads = 8 / 20 in the output), so I measured both -t 8 and -t 20.

Model -t 8 total -t 20 total encode batchd
large-v3-turbo (fp16) 13,873ms 11,015ms 12,005 → 9,004ms 564 → 579ms
medium 9,698ms 7,478ms 6,738 → 4,162ms 1,631 → 2,069ms
large-v3-turbo (q5_0) 7,108ms 4,448ms 6,326 → 3,714ms 276 → 226ms

Every total went down, but not evenly inside. The encoder is one call that passes the whole audio through at once, so it shrinks cleanly. The decoder side is more than 160 small calls, so it shrinks less, and for medium it actually grew. More threads don't make everything uniformly faster; that's what this table supports.

On variance: turbo fp16 at -t 20 ranged from 10,294 to 11,674ms (13%) even across runs 2–5, wider than the other conditions (a few %). If you measure only the total once and compare, any difference smaller than that spread disappears.

Where this stops

  • I didn't compare GPUs on the same backend. I didn't build a Vulkan version on the Arm side, so the Arc 140V (Vulkan) and the RTX Spark N1X (CUDA) differ in both backend and generation. This is not a measurement that can say which GPU is faster.
  • The two CPU columns are matched on "no BLAS", but not on compiler and flags (x86: MSVC, AVX2; Arm: clang, -march=armv8.7-a and others).
  • These are measurements on one machine each, so I can't claim more than the same direction showed up on these two machines. It could reverse on other CPUs or GPUs.
  • The measurement uses only the same single sample as last time (6.6 s, one sentence, one speaker). The ratios could change with long audio, noisy audio or multiple speakers; I haven't checked.
  • The transcription matched the correct text in every condition. I didn't compare accuracy (this clip shows no difference, so it can't support any accuracy ranking).
  • The clang OpenMP runtime used for the Arm build is the debug (non-redistributable) copy bundled with the development tools. I'm not distributing the built executables or DLLs.

Takeaways

  1. The previous result, medium on the CPU and turbo on the GPU, showed the same direction on a CPU with a different instruction set and on a different GPU family. (On the integrated GPU, v1.9.5 gave a tie in total; excluding loading, turbo is faster.)
  2. The reason was the same too: turbo cuts the decoder but keeps the large encoder. Where the encoder is heavy, the loss wins; on the GPU, the gain wins.
  3. The faster the GPU, the larger the share of model loading in the total. Look at the phases, not the total, or keep the model resident and measure without loading.
  4. On Windows on Arm, the compiler used for the build changed the result by an order of magnitude. Check the system_info line first.

whisper.cpp is licensed under MIT (ggml-org/whisper.cpp). The phase names come from the comments on the counters in the source.

About this series

This is part of my series on a DIY ESP32-S3 smartwatch. The watch sends recorded speech to a PC, which transcribes it; this article and the previous one compare models for that step. The Arm PC here is the same Surface Laptop Ultra I've written about in English before. Related articles:

The original of this article: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった.

Top comments (0)