DEV Community

JC
JC

Posted on Originally published at jac-76.github.io

Running Whisper + LLMs on an AMD NPU under Linux

#amd #ryzenai #npu #linux

TL;DR — On a MSI Stealth A16 AI+ (Ryzen AI 9 365, XDNA2 NPU) running Arch,
I got OpenAI's whisper-large-v3-turbo transcribing on the NPU — not the
CPU, not the GPU — at RTF ≈ 0.18 (a 30 s clip in ~5.2 s) for roughly a
tenth of the energy the same job costs on the CPU, plus an LLM answering
on the same NPU through an OpenAI-compatible API. The
whole path is local and offline. This is the write-up of the driver stack,
the one real gotcha (memlock), and the runtime that made it a 20-minute job
instead of a weekend.

Why this is worth writing down

AMD's "Ryzen AI" NPU (the XDNA / XDNA2 block in Phoenix / Hawk Point / Strix
Point laptops) is marketed almost entirely around Windows: the Ryzen AI SDK,
the ONNX Runtime VitisAI execution provider, Lemonade, and the demos all
assume you're on Windows with the official stack. On Linux the picture in early
2026 is better than most people think — the NPU driver has been in the
mainline kernel as amdxdna since 6.14 — but the "load a real model and run
it" story still isn't well documented.

Here's what actually worked, end to end.

The hardware

Part Detail
Laptop MSI Stealth A16 AI+ A3HVGG
APU AMD Ryzen AI 9 365 (Strix Point)
NPU XDNA2, 8 columns, exposed as /dev/accel/accel0
NPU firmware 1.1.2.64
Kernel 7.1.9-arch1 (amdxdna in-tree)
OS Omarchy (Arch Linux)

AMD quotes the Strix Point NPU at up to 50 TOPS, INT8.

1. The driver stack

Three pieces have to be in place before any runtime can touch the NPU:

  1. amdxdna — the kernel driver. In-tree from Linux 6.14; it's what creates /dev/accel/accel0. Check it's bound: 
   $ ls /dev/accel/
   accel0
   $ dmesg | grep -i amdxdna
Enter fullscreen mode Exit fullscreen mode
  1. XRT (Xilinx/AMD Runtime) + the xrt-plugin-amdxdna shim. XRT is the userspace API; the plugin teaches it about the XDNA device. On Arch both are in extra: 
   $ sudo pacman -S xrt xrt-plugin-amdxdna
   $ xrt-smi examine
   ...
   XRT
     Version              : 2.21.75
     NPU Firmware Version  : 1.1.2.64

   Device(s) Present
   |BDF             |Name          |
   |----------------|--------------|
   |[0000:66:00.1]  |RyzenAI-npu4  |
Enter fullscreen mode Exit fullscreen mode

You want a Device(s) Present line with a RyzenAI-npu* name. If XRT is
installed but the plugin isn't, xrt-smi runs but that table is empty.
xrt-smi examine --report platform then shows Total Columns : 8 — the
XDNA2 array this SoC exposes.

  1. Versions that worked here: xrt 2.21.75, xrt-plugin-amdxdna (same release), NPU firmware 1.1.2.64, and flm validate reporting the amdxdna driver interface as 0.8.

The one real gotcha: memlock

The NPU runtime pins model weights into physical RAM, so the calling user needs
an unlimited memlock rlimit. The default (usually 8 MiB or 64 MiB) is nowhere
near enough and the failure mode is an unhelpful allocation error deep in the
runtime.

$ sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited"
$ sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited"
# log out and back in
Enter fullscreen mode Exit fullscreen mode

You want to see this afterwards:

$ ulimit -l
unlimited
Enter fullscreen mode Exit fullscreen mode

2. The shortcut: FastFlowLM instead of the Ryzen AI SDK

The "official" Linux route is: build ONNX Runtime with the VitisAI EP, install
the Ryzen AI SDK bits, quantize your model to the NPU's format, wrangle a Python
venv full of onnxruntime-vitisai and Vitis tooling. It's a lot, and much of it
is Windows-first.

FastFlowLM (flm) skips all of that. It's a
NPU-first runtime (Rust/C++) that ships prebuilt xclbins (the NPU binary
kernels) and libwhisper_npu.so in the package itself:

$ ls /usr/share/flm/xclbins/
encoder_attn  encoder_dequant  encoder_mm  whisper_head  ...
$ flm --version
FLM v1.0.2
Enter fullscreen mode Exit fullscreen mode

Because the kernels are bundled, there is no onnxruntime-vitisai / Ryzen AI SDK
venv to build. (Arch's stock python-onnxruntime-cpu only has
CPUExecutionProvider anyway — irrelevant here.) On Arch: sudo pacman -S
fastflowlm.

Validate the whole stack in one shot:

$ flm validate
[Linux]  Kernel: 7.1.9-arch1-2
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.8
[Linux]  Memlock Limit: infinity
Enter fullscreen mode Exit fullscreen mode

All green = ready. If Memlock Limit says anything other than infinity, go
back to the limits.conf step.

3. Whisper on the NPU

Pull the model — whisper-v3:turbo is large-v3-turbo quantized for XDNA2:

$ flm pull whisper-v3:turbo
# ~650 MB: model.q4nx + tokenizers -> ~/.config/flm/models/Whisper-V3-Turbo-NPU2/
Enter fullscreen mode Exit fullscreen mode

Serve it. On FLM 1.0.2+ Whisper loads standalone — older docs claimed you had
to co-load an LLM, but you don't:

$ flm serve --asr 1          # OpenAI-compatible server on :52625
Enter fullscreen mode Exit fullscreen mode

Transcribe over the HTTP API (anything ffmpeg can decode — wav/mp3/ogg/m4a/flac):

$ curl http://127.0.0.1:52625/v1/audio/transcriptions \
    -F "file=@audio.ogg" \
    -F "model=whisper-v3"
Enter fullscreen mode Exit fullscreen mode

There's also a CLI path: flm run <model> --asr 1, then /input "clip.mp3" in
the chat prompt.

Results

Benchmarked with the bundled bench.py — 10 runs, first 2 discarded as warm-up,
audio length read from the file via ffprobe so the RTF is honest and
reproducible:

Metric Value
Audio length 30.0 s (JFK, Rice University speech excerpt)
Transcription wall time (warm) 5.2 s (σ 0.04 s within a run; 5.17–5.6 s across sessions)
Real-time factor (RTF) ≈ 0.17–0.19
Transcript accuracy correct, verbatim

Roughly 5–6× faster than real time. Within a single benchmark the spread is
under 1%; between sessions the mean drifts a few hundred ms with machine
temperature and background load.

Is it actually on the NPU?

Two checks. First, the FLM log prints [NPU Locked!] when a job starts and
[NPU Lock Released!] when it finishes. Second — and more convincing — sample
system load while the benchmark runs and see that nothing else is doing the
work:

Device Idle baseline During 10 transcriptions
CPU (20 threads, system-wide) 2.2 % 4.3 % mean, 14.4 % peak
iGPU (Radeon 890M) 7 % 10 % mean, 15 % peak
dGPU (RTX 4070) 0 % 0 %

The CPU rises about two points over idle — that's the curl/harness overhead and
the server's I/O thread, not inference. The iGPU delta is desktop compositing
(Hyprland renders on the 890M), and the discrete GPU is never touched at all.
The 30 seconds of audio is being processed somewhere that doesn't show up in any
of these three counters, which is exactly the point: the CPU and both GPUs stay
free while the NPU works.

What the NPU actually buys you: energy

Speed alone isn't the story — whisper-large-v3-turbo has a tiny decoder and
runs fine on CPU. So I built whisper.cpp from source (the Arch package's ggml
backend is currently broken) and ran the same 30 s clip through the same
model on the CPU, tuned to 16 threads, reading the RAPL energy counters
(/sys/class/powercap/intel-rapl:0) around every run.

NPU (FastFlowLM) CPU (whisper.cpp, -t 16)
Wall time (30 s clip) ~5.3 s ~6.5 s
RTF 0.18 0.22
CPU-package power while running ~20 W ~73 W
CPU-core power while running ~0.8 W ~10 W
Energy per transcription, over idle ~45 J ~410 J
Energy per transcription, total package ~105 J ~478 J

The wall-clock win is modest — about 25%. The energy difference is the
point: transcribing that clip on the NPU costs roughly an order of magnitude
less energy than doing it on the CPU (~45 J vs ~410 J above idle). Package
power rises ~10 W instead of ~60 W, the CPU cores never leave idle, and the fans
stay quiet. Per hundred transcriptions that's about 1 W·h versus 11 W·h — and 20
CPU threads left free the whole time.

(Measured on a live desktop, so absolute wattages drift a few watts between
runs with background activity — "energy over idle" is the stable figure and what
the comparison rests on. whisper-cli also reloads the 1.6 GB model each run,
which pads its wall time slightly but not its energy. Both harnesses are in the
npu-whisper repo:
bench.py --power for the NPU column, bench_cpu.py for the CPU column.)

4. An LLM on the same NPU

FLM serves LLMs on the NPU through the same OpenAI-compatible surface. Its model
catalogue covers the usual small-to-mid open weights:

$ flm list
  gemma3:1b        ✅
  qwen3:1.7b       ⏬
  llama3.2:3b      ⏬
  phi4-mini-it:4b  ⏬
  deepseek-r1:8b   ⏬
  gpt-oss:20b      ⏬
  whisper-v3:turbo ✅
  ...
Enter fullscreen mode Exit fullscreen mode

One server can expose both ASR and chat:

$ flm serve gemma3:1b --asr 1
$ curl http://127.0.0.1:52625/v1/chat/completions \
    -H 'content-type: application/json' \
    -d '{"model":"gemma3:1b","messages":[{"role":"user","content":"hello"}]}'
Enter fullscreen mode Exit fullscreen mode

So /v1/audio/transcriptions and /v1/chat/completions are both live on
:52625 from a single process on the NPU.

5. Tying it together — two small tools

With the endpoint working, the rest is glue:

  • npu-whisper — a zero-dependency
    Bash wrapper: run it on an audio file and it auto-starts flm serve --asr 1
    if it's down, waits for readiness, curls the transcript, and leaves the
    server warm. --json, --status, --stop.

  • local-ai-assistant — a
    fully offline voice assistant, stdlib-only Python:
    pw-record → Whisper (NPU) → LLM (NPU) → piper TTS → speaker. One FLM server
    backs the whole chain. chat keeps conversation history across runs.

Both are deliberately small — the interesting work was getting the NPU to do the
inference, not the plumbing on top.

Gotchas & rough edges

  • memlock is the thing that bites everyone first. flm validate catches it.
  • FLM version: 1.0.2 here; 1.0.3 exists but the update notice points at a Windows .msi — 1.0.2 is fine to stay on for Linux.
  • Model format is runtime-specific. These .q4nx weights + bundled xclbins are FastFlowLM's; you can't point llama.cpp or vanilla ONNX Runtime at them.
  • No GPU/CPU fallback worth using. If the NPU path fails, you're better off fixing it than limping along on CPU — whisper.cpp on CPU is far slower for large-v3-turbo.
  • Discoverability. Almost every AMD NPU tutorial is Windows. The Linux amdxdna + XRT + FLM combination works well but you have to assemble it yourself. That's the gap this post is trying to close.

Reproduce it 

# 1. driver stack
sudo pacman -S xrt xrt-plugin-amdxdna fastflowlm
sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited"
sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited"
# log out / back in
flm validate            # want: all green, Memlock Limit: infinity

# 2. models
flm pull whisper-v3:turbo
flm pull gemma3:1b

# 3. run
flm serve gemma3:1b --asr 1
curl http://127.0.0.1:52625/v1/audio/transcriptions -F file=@clip.wav -F model=whisper-v3
Enter fullscreen mode Exit fullscreen mode

Requirements: a Ryzen AI (XDNA / XDNA2) laptop, kernel ≥ 6.14 with amdxdna,
and the memlock bump.

Top comments (0)