Whisper on the RK3588 NPU: the encoder offload works — everything around it is the real lesson

Status: encoder offload merged into a working whisper.cpp path on an RK3588 board, transcription byte-identical to the CPU reference, PR filed upstream (ggml-org/whisper.cpp#4159). This post is the full map — the win, the two dead ends, and the tooling traps — because the dead ends are the part nobody publishes.

The win

Whisper's encoder is a fixed-shape graph, which makes it the natural candidate for whole-graph NPU offload. With a native 30-second window export, the geometry lines up exactly with whisper.cpp: ONNX [1,80,3000] → [1,1500,512].

Same board, same input:

• RKNN encoder, single run: 703.0 ms (of which rknn_run 676.8 ms)
• CPU encoder: 4,729.4 ms
• End-to-end transcription: 13.76 s → 11.15 s, output byte-identical with the offload on and off

The glue code is 143 lines. It is small because the measurements came first (see the last section).

Reality check 1: NPU time is the small part

Before building anything, I ran Rockchip's official Model Zoo whisper build on the same board. Its RKNN inference totals 2.47 s (×realtime 0.225 — matches the official 0.215 claim, good). But the end-to-end wall clock is 16.39 s: the Python host side eats roughly 13.9 s. Against the CPU build's 28.26 s wall clock, the "NPU version" is only 1.72× faster end to end.

NPU fast ≠ product fast. If your harness is Python and your pre/post-processing is heavy, the accelerator is not your bottleneck.

Reality check 2: the decoder graph is structurally wrong

The Model Zoo decoder runs at 87 ms/step, 24 steps ≈ 2.1 s — it runs, but it re-runs the whole graph per token because there is no KV-cache structure. That's why this round offloads the encoder only, and why "just convert the whole model" is a trap (below).

The dead ends (with receipts)

1. Whole-model .rknn conversion was slower than CPU. Earlier attempts in upstream issue #1557 measured exactly that; the decoder graph shape is the reason. Skipping this saved me a week.
2. Per-operator matmul offload doesn't pay. RK3588 GEMM is weak (~FP16 10 GFLOPS per unit), and rknpu2 matmul demands a specific weight repacking (A→(K/8,N,8), B→(M/16,K/32,16,32), C→(N/4,M,4)). Someone built it, measured it, and advised against it — in public, in the same issue thread. Whole-graph or nothing.

Tooling traps worth knowing

• A converted .rknn file can lie in its header. My first decoder conversion claimed 159,697,664 bytes in the header while actually being 64,778,240 bytes; the runtime rejected it with RKNN_ERR_MODEL_INVALID. Verify header size against actual file size after every conversion.
• rknn-toolkit-lite2 import pollutes logging — it mutates logging._nameToLevel, after which torch crashes. Lazy-import it.
• Window geometry is a silent mismatch. The Model Zoo build changed CHUNK_LENGTH to 20 s; its output frame count doesn't match whisper.cpp's native 30 s / 1500 frames. The parts don't compose until you re-export at 30 s.
• My own test harness was wrong once: an expected-transcript file carried a leading space from test code and nearly failed a correct OFF transcription. Fixed the expectation, not the model output — and kept the correction script as evidence.

Measurement honesty

The RK3588 board I test on carries production load (load average 16–20 on 8 cores) and no idle window appeared in a week of watching. So every CPU wall-clock number above is a loaded-board number and is labeled as such wherever I cite it. The per-call NPU figures (703 ms etc.) are single-run timings inside one process and are not load-sensitive in the same way, but the comparison context matters and I'd rather state it than hide it.

The order that made this cheap

1. CPU baseline first → the encoder was the single biggest item (9.57 s median encode on an 11 s clip, loaded board).
2. Model Zoo run → proved the whole-graph NPU ceiling and exposed the host-overhead problem before I wrote any code.
3. Only then, 143 lines of glue — after a reuse inventory (the OpenVINO backend precedent is 108+31 lines of registration code) instead of hand-rolling a ggml operator backend.

Measure first, write code last. The PR is ggml-org/whisper.cpp#4159 if you want to follow or review.