Benchmarking RKNN on RK3576 and RK3588

RKNN-Toolkit2 needs a target platform when it converts an ONNX model. For an RK3576-versus-RK3588 test, that means building one .rknn file for rk3576 and another for rk3588 from the same source model.

The useful comparison starts there. Same ONNX graph, same input shape, same quantization mode, same calibration images, and the same test frames on both boards.

Build for Both Targets

Don't copy an RKNN file built for one target to the other board and call the result a platform comparison. The conversion step can apply target-specific graph optimization, so the compiled model belongs in the test record along with the toolkit and runtime versions.

The conversion commands can be as simple as this when using the official Rockchip RKNN Model Zoo:

python3 convert.py model.onnx rk3576 i8 model-rk3576.rknn
python3 convert.py model.onnx rk3588 i8 model-rk3588.rknn

For INT8, keep the calibration list identical. Changing those images between builds adds another variable, and a faster model isn't useful if quantization quietly damages the output you care about.

So check correctness before timing anything. Run a fixed validation set, save the decoded outputs, and compare detections or task-specific accuracy against the original model. If the two RKNN builds don't produce acceptably similar results, the latency table can wait.

Time the Runtime Call

Once the outputs look right, time the runtime call without camera capture or drawing code around it. This gives you the closest useful measurement of model execution from the application side.

Discard the first runs. Model initialization, memory allocation, caches, and frequency scaling can make the first few samples unlike the next few hundred, which is exactly why a screenshot of one fast run doesn't tell you much.

Assuming the model is already loaded and the RKNNLite runtime has been initialized, a small Python harness is enough for the first pass:

from statistics import mean
from time import perf_counter


def benchmark_inference(rknn_lite, input_tensor, warmup=20, runs=200):
    for _ in range(warmup):
        rknn_lite.inference(inputs=[input_tensor])

    samples_ms = []
    for _ in range(runs):
        started = perf_counter()
        rknn_lite.inference(inputs=[input_tensor])
        samples_ms.append((perf_counter() - started) * 1000)

    samples_ms.sort()
    average = mean(samples_ms)

    return {
        "mean_ms": average,
        "p50_ms": samples_ms[len(samples_ms) // 2],
        "p95_ms": samples_ms[int(len(samples_ms) * 0.95) - 1],
        "fps_from_mean": 1000 / average,
    }

This measures the wall time of inference() as seen by Python. It does not measure camera capture, resize and color conversion, output decoding, non-maximum suppression, tracking, display, storage, or network work.

Measure the Complete Pipeline

And those stages can change the board choice. A camera application that spends 7 ms in inference and 18 ms preparing and decoding a frame is not a 7 ms application, no matter which number gets printed by the demo.

Measure the complete loop with separate timestamps rather than one timer around everything:

t0 = perf_counter()
frame = read_frame()
t1 = perf_counter()
input_tensor = preprocess(frame)
t2 = perf_counter()
outputs = rknn_lite.inference(inputs=[input_tensor])
t3 = perf_counter()
result = postprocess(outputs)
t4 = perf_counter()

sample = {
    "capture_ms": (t1 - t0) * 1000,
    "preprocess_ms": (t2 - t1) * 1000,
    "inference_ms": (t3 - t2) * 1000,
    "postprocess_ms": (t4 - t3) * 1000,
    "total_ms": (t4 - t0) * 1000,
}

This is where the rest of each SoC starts showing up. RK3576 uses Cortex-A72/A53 CPU cores and a 32-bit memory interface; RK3588 moves to Cortex-A76/A55 and a 64-bit interface. The RK3576 and RK3588 hardware comparison covers those differences in more detail. Preprocessing and post-processing still run somewhere, and extra memory bandwidth can matter when frames and tensors move between blocks.

The board matters too. Cooling changes sustained clocks, RAM capacity decides whether the full process fits comfortably, and the available camera, storage, and network interfaces decide whether the test setup resembles the product you're building. The RK3588 single-board computer overview shows how two boards can expose the same SoC differently. A bare NPU benchmark avoids all of that—useful for one question, misleading for several others.

Keep the Test Reproducible

For each run, record:

• Model file hash
• Target platform
• RKNN-Toolkit2 version
• Runtime and driver versions
• Input dimensions and data type
• Quantization mode and calibration-set revision
• Board image and kernel version
• CPU governor and cooling setup
• Start and sustained temperatures
• Dropped or delayed frames

An impressive inference average can coexist with a capture queue falling further behind.

If both boards return similar NPU-call latency, that isn't a broken test. Their published rating is similar. The developer question is whether the rest of the pipeline also stays similar once the CPU, memory system, and I/O are doing real work.

If RK3588 pulls ahead only in preprocessing or post-processing, write that down rather than attributing the whole result to its NPU. And if RK3576 meets the end-to-end latency target with stable temperatures, the larger chip has not solved a problem your application currently has.

Keep the raw samples and the exact commands with the result. Six months later, a table containing only FPS: 42 is nearly useless; a repeatable test with two target builds tells you what changed when the model, runtime, or board image moves on.