10 RKNN models benchmarked on RK3588 — and the fastest one is cheating onto the CPU

Picking a model for a Rockchip board usually means trusting a vendor table
or timing one run yourself and hoping. I wanted a leaderboard I could
reproduce, so I built rk-bench and measured 10 public models on an
RK3588 (driver 0.9.8, librknnrt 2.3.2) under one frozen protocol.

Three rules make the numbers worth trusting:

1. Latency comes from the NPU runtime's own PERF_RUN hardware counters, not wall-clock time. The board was busy (load ≈ 16) the whole time; hardware counters don't care.
2. Every model gets a CPU-fallback verdict from PERF_DETAIL per-layer accounting — the same method as RK Edge Doctor, with the same thresholds. Input/output staging never counts as fallback.
3. No number comes from a single run. More on why below; it turned out to be the most interesting part.

The leaderboard

Model	Task	p50 (ms)	p95 (ms)	FPS	Verdict	RSS Δ (MB)
RetinaFace Mobile320	face-detect	6.43	6.68	155.6	MIXED	12.9
MobileNetV2	classify	8.63	8.89	115.9	NPU_ONLY	23.9
ResNet50-v2	classify	25.40	29.34	39.4	NPU_ONLY	110.1
YOLOv8n	detect	37.50	38.58	26.7	NPU_ONLY	47.4
YOLOv8n-obb	detect-obb	37.65	38.26	26.6	NPU_ONLY	44.8
YOLOv5s (ReLU)	detect	37.80	38.58	26.5	NPU_ONLY	52.8
YOLO11n	detect	43.67	49.79	22.9	NPU_ONLY	46.1
YOLOv8n-pose	pose	49.73	51.27	20.1	NPU_ONLY	47.6
YOLOv10s	detect	88.09	90.49	11.4	NPU_ONLY	78.3
YOLOv8s-seg	segment	104.53	113.08	9.6	NPU_ONLY	112.1

All FP, official RKNN Model Zoo conversions (seven pre-converted zoo
packages; MobileNetV2 / ResNet50-v2 / RetinaFace converted from the zoo
ONNX with Toolkit2 2.3.2 using the zoo's own parameters). Full table with
model sizes and per-frame memory traffic:
docs/LEADERBOARD.md.
Raw data is one JSON file — the single source of truth the tables are
generated from.

Finding 1: the fastest model is the only one on the CPU

RetinaFace Mobile320 is the quickest model on the board — and the only
one that isn't fully on the NPU. Five Reshape operators around its
detection-head concats run on the CPU: 196 µs, 4.05% of frame time.
Cheap in absolute terms, and you'd never see it from end-to-end timing.
Verdict: MIXED. The other nine are clean NPU_ONLY, including all the
YOLO variants.

Finding 2: your first benchmark run is ~15% too slow

While validating the protocol I hit something I haven't seen documented:
the first RKNN process after an idle gap measures ~15% slower across
every single run — YOLO11n at 51.4 ms instead of 44.0 ms, 50 runs in a
row, all slow. Warmup inside that process does not fix it. Start a
second process and the speed is back.

The board uses the rknpu_ondemand governor, and the effect smells like
first-context state at the driver level; I can't prove the mechanism
from outside, but I can prove the cure, because rk-bench now does it by
default:

• a priming cycle (one sacrificial load + 5 runs in a throwaway process) before measuring, and
• three independent sessions (separate processes), publishing the median p50/p95 and the worst verdict. The three session p50s are printed in every entry's notes, so the spread is auditable.

This also explained a nagging discrepancy: an earlier single-process
health table (RK Edge Doctor's) had YOLO11n at 52.9 ms. It wasn't wrong —
it was measuring the first-context state. Single-process benchmarks of
RK3588 should be treated with suspicion, including, now, my own older
ones; that table has moved into rk-bench and links here.

A related trap: even primed, a single session can sit off the pace
(YOLOv8s-seg produced sessions of 104.4 / 114.3 / 104.5 ms). Median of
three is what keeps one bad minute on a busy board out of the table.

Reproduce it

Zero dependencies — Python 3.9+ standard library and ctypes against the
board's own librknnrt.so:

git clone https://github.com/anguoyang/rk-bench
cd rk-bench
PYTHONPATH=src python3 -m rkbench run yolov8n.rknn --name YOLOv8n \
  --task detect --soc rk3588 --json report.json

Got a model or a board that isn't on the table? Run it and open a
bench-report issue with the JSON — reports whose runtime/driver differ
from a board's table start a new table instead of mixing in (RK3576 is
the seeded next board). INT8 twins of the YOLO rows are the other big
open item.

If you benchmark RKNN models, I'd genuinely like to know: have you seen
the first-process effect on your board, and does your methodology defend
against it?

Disclosure: I maintain rk-bench and RK Edge Doctor; both are Apache-2.0
and exist to drive traffic to my RK integration work — the data,
however, stands on its own and every number above is reproducible from
the repository.