Most "Raspberry Pi AI benchmark" numbers on the internet are one-off runs with unknown settings. If you want numbers you can trust - and compare - you need a method. This is the one I use.
1. Install llama.cpp
sudo apt update && sudo apt install build-essential cmake git -y
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j4
Build from source. Prebuilt binaries rarely match your kernel and flags, and a mismatched build can cost you 20% throughput.
2. Pick models that actually fit
A Pi 5 has 8 GB of RAM shared with the OS. Stay under ~5 GB for model + context or you'll swap and your numbers are garbage. Reliable picks in Q4: TinyLlama 1.1B, Qwen 1.5B/3B, Phi-3-mini (tight but works), Gemma 2B. Skip 7B+ unless you enjoy watching swap thrash.
3. Measure honestly
Use llama-bench with a fixed prompt and generation length. Log three numbers per run: prompt processing tok/s, generation tok/s, and SoC temperature at start and end. A run without a temperature is not a benchmark - thermal throttling on an uncooled Pi 5 shows up around 85°C and can cut throughput by a third mid-run.
./build/bin/llama-bench -m models/qwen-3b-q4.gguf -p 128 -n 256 -r 3
4. Keep runs repeatable
- Fix the CPU governor (
performance) for the run and note it down. - Same prompt set every time. Different prompts = different numbers.
- Cool the board between runs, or say so in the log.
- Record ambient temperature when comparing across days.
Going further
I run this protocol often enough that I packaged it: a Benchmark Workbook with the log sheets and test matrix pre-built, and a Report Generator that turns raw logs into a shareable report.
Full guide with more detail: Local AI on Raspberry Pi - guides hub
What are you running on your Pi? Curious what tok/s people are seeing on 3B-class models.
Top comments (0)