Converting a model to RKNN and watching it "run" on a Rockchip board proves exactly one thing: it runs. It says nothing about where it runs. RKNN will happily execute unsupported or badly-quantized layers on the CPU, silently, while your benchmarks just look "a bit slower than expected." In my own testing I have seen a SAM-variant conversion whose decoder carried a dozen CPU ops — the model worked, the demos worked, and a chunk of it was never touching the NPU.
Until now, checking meant spelunking through runtime perf dumps by hand. So I added a model command to RK Edge Doctor (the board health checker I released earlier this month) that answers it in one run:
git clone https://github.com/anguoyang/rk-edge-doctor.git
cd rk-edge-doctor
PYTHONPATH=src python3 -m rkdoctor model model.rknn --json report.json --html report.html
Real output, from a public YOLOv8n (FP16) on my RK3588 board:
Model: yolov8n_det.rknn | Verdict: NPU_ONLY
Latency us: p50=40933 p95=41523 mean=41096.3 (runs=20)
CPU fallback operators: none
108 layers measured. Total layer time 38.2 ms; CPU time 203 µs — and every microsecond of that is the input/output staging operators, which live on the CPU by design. The report lists them separately so the tool doesn't cry wolf: staging is structural, everything else on CPU is fallback, and the verdict comes from the fallback share of time, not from op counts:
-
NPU_ONLY— fallback under 1% of layer time, and no single fallback op ≥ 100 µs -
MIXED— fallback between 1% and 20% -
CPU_HEAVY— fallback over 20% (threshold adjustable, and recorded in the report) -
UNKNOWN— the tool couldn't run the model, and says why instead of guessing
How it works (and the trap that cost me an hour)
No rknnlite, no toolkit install, no pip anything. The tool binds librknnrt directly through ctypes — the runtime library is the one thing every RKNN board already has — runs warmup plus measured inferences with a zero input, and reads the runtime's own accounting: PERF_RUN for per-run duration and PERF_DETAIL for the per-layer table with each layer's target (NPU/CPU) and time.
Two details worth knowing if you ever do this yourself:
-
Latency is hardware time, not wall clock. The board I measured on runs production services at load average ~16.
PERF_RUNcomes from the runtime's counters, so p50/p95 stay meaningful on a busy board. The report still records the host load average, because honesty is kind of the theme here. -
PERF_DETAILis empty unless you ask at init time. You must pass the collect-performance flag (RKNN_FLAG_COLLECT_PERF_MASK) torknn_init. Forget it and the query succeeds, returns zero bytes, and tells you nothing — no error, just silence. Ask me how I know.
The honest part
The official model-zoo conversions I tested (YOLOv8n, YOLOv8s-seg) come out clean: NPU_ONLY. That's expected — they are the reference conversions. The fallback problem shows up with your model: a custom architecture, an exotic op, a quantization choice the compiler didn't like. That's exactly the case where you currently have no easy way to look.
So the most valuable thing you can do with this release is not run it on a zoo model — it's run it on the weird one. If you get a MIXED or CPU_HEAVY verdict, the JSON report (which operators, how many microseconds each) is the whole story in one file; sharing it in an issue helps everyone, and reports record the model file name and hash only, never its path.
v0.2 also keeps everything from v0.1: rkdoctor board runs 16 board health checks (SoC, rknpu driver, runtime version, thermals…) and emits the same style of JSON/HTML report. Reports from boards other than RK3588 — RK3588S (Orange Pi 5, ROCK 5B), RK3566/3568, RK3576 — are still the contribution I need most.
Repo (Apache-2.0, Python stdlib only): https://github.com/anguoyang/rk-edge-doctor
Next up: v0.3 turns the same reporting machinery to RTSP camera acceptance — real frame rate, frame-gap jitter, freeze detection — the "attach it to the delivery document" test for cameras.
Top comments (0)