The training is the part everyone writes about. Epochs, batch size, mAP50 on the validation set — there are tutorials for all of it. I covered the dataset sourcing and training setup in an earlier piece, and the model hit mAP50 0.844 on a held-out test set of 277 images. The interesting problem starts after the .pt file exists.
My target hardware is a Raspberry Pi 3 Model B Rev 1.2, 906 MB RAM, Raspberry Pi OS Bookworm arm64 Lite, running headless in a room. Not a laptop, not a GPU server. The model needs to run inference there. NCNN is how I got there. Here is what I measured and what I'd do differently.
Why NCNN instead of ONNX Runtime or TFLite
Three reasons made NCNN the right choice for this specific setup.
No Python runtime overhead at inference time. NCNN is a C++ library. On a Pi 3 with 906 MB RAM, not loading the full PyTorch or ONNX Runtime Python stack matters. The Ultralytics export path produces .ncnn.param and .ncnn.bin files that the inference script reads through a thin Python wrapper on top of the C++ library — far lighter than loading torch and a full model graph.
ARM-native optimization. NCNN was built for mobile and embedded ARM from the start. It uses NEON intrinsics for vectorized arithmetic on ARM CPUs. A Pi 3 runs a Cortex-A53 which supports NEON. TFLite also targets mobile ARM, but the Ultralytics export path to NCNN is more direct for a YOLO model and better documented for this hardware class.
No driver dependency. There is no CUDA on a Pi 3. The choices narrow to CPU-only inference frameworks. NCNN's weights are self-contained files that the inference script reads without a runtime installation step on the Pi itself — the weights are exported on the training machine (my Mac) and copied over.
The export command:
yolo export model=best.pt format=ncnn imgsz=416
That imgsz=416 matters more than it looks. The training used imgsz=640, which is where the mAP50 0.844 number came from. The export resolution is different, and the reason for the difference is the subject of the next section.
The result: model.ncnn.bin at 36 MB and a companion model.ncnn.param. The PyTorch best.pt it was exported from is 18 MB. The NCNN binary is larger because it includes precomputed operator metadata and weights in a form the runtime reads without Python graph execution overhead.
The resolution tradeoff: 640 training, 416 inference
Training used imgsz=640. That is the resolution at which the mAP50 0.844 on the held-out test set was measured.
Running inference at 416 px costs accuracy. I don't have a measured quantification of how much — I didn't run the held-out evaluation pipeline at 416 px separately — so I won't claim a number I don't have. What I can say is why the tradeoff was worth making despite that unknown.
The Pi 3 has 906 MB RAM. At inference time, the NCNN runtime, the Python wrapper, the camera capture buffer, and the OS all share that memory. Running at 640 px puts pressure on available memory that 416 px avoids. I also measured the actual inference time at 416 px — median 8.5 seconds across 19 real scans on the device. I did not measure at 640 px, so I can't give a number for the comparison.
The deeper justification: this detector's job is empty-space detection on a fixed shelf. The single class is empty_space. The post-processing stages — ROI masking, baseline subtraction, temporal majority vote — handle the hard reliability problems. What the model needs to provide is a useful detection surface, not pixel-perfect localization of gaps. 416 px delivers that for a shelf region that occupies a significant fraction of the frame.
The tradeoff looks different for a model that needs to read fine product labels or distinguish nearly-identical items by shape. This detector doesn't — it's designed around not solving the hard SKU identification problem. That design constraint is what makes the lower inference resolution acceptable.
What 8.5 seconds means when you scan hourly
The measured inference time on the Pi 3 is a median of 8.5 seconds across 19 real scans, with a range of 8.4–11.8 seconds.
That is slow. For a production system requiring sub-second response, a Pi 3 with NCNN would not be the right hardware. But this system scans hourly. The cron fires once an hour, takes one capture, runs inference, writes the result, and waits 60 minutes. At that cadence, whether inference takes 8.5 or 11.8 seconds is irrelevant to the system's usefulness.
The headless failure mode I care about isn't inference speed — it's scan collisions when the previous inference run is still running when the cron fires again. At an 11.8-second maximum and a one-hour cadence, that collision cannot happen under normal conditions. An inference run would need to take over 3,599 seconds, and the SD card would have failed first.
This is a latency vs. cadence decision, not a hardware problem. If the cadence were every 30 seconds, 8.5 seconds median would be dangerous. If the cadence were daily, 15-second inference at 640 px on a Pi 3 would be fine. The number that matters isn't the raw inference time — it's whether the inference time is small relative to your detection cadence.
For retail empty-shelf detection, hourly is often fine. Shelf replenishment happens on a cadence measured in hours. A gap that appears at 9:00am and gets detected at 9:00am is functionally the same as one detected at 9:08am. The 8.5-second inference is the wrong number to worry about; the detection cadence is the design constraint.
YOLO11n vs YOLO11s: the model selection decision
I trained both variants. On the validation set at epoch 60:
| Model | Precision | Recall | mAP50 |
|---|---|---|---|
| YOLO11n | 0.782 | 0.734 | 0.792 |
| YOLO11s | 0.833 | 0.736 | 0.820 |
YOLO11s is meaningfully better. Precision is 5 points higher; mAP50 is 2.8 points higher. The held-out test set evaluation — 277 images, 1,255 instances — used YOLO11n and hit precision 0.82, recall 0.786, mAP50 0.844. I don't have a separate test-set evaluation for YOLO11s.
I deployed YOLO11n on the Pi. The reason is that YOLO11s is heavier, and I did not measure its inference time on the Pi 3. My assumption is that it would be slower — but that assumption was never tested. Choosing YOLO11n without measuring YOLO11s's actual inference cost on the device is the decision I'd revisit if I were building this more carefully.
Given that the post-processing stages carry most of the reliability burden, and YOLO11n's mAP50 of 0.792 on validation was already sufficient for the post-processing to work with, the accuracy gap from YOLO11s felt acceptable. But "felt acceptable" is not the same as "measured acceptable."
How fswebcam's warm-up frames changed capture reliability
Capture uses fswebcam at 1280x720, discarding 10 warm-up frames before taking the final shot.
The warm-up frames solve an auto-exposure problem. A USB webcam connected to a Pi without a continuously running camera application initializes from a cold state each time the cron fires fswebcam. The camera's auto-exposure algorithm adjusts over the first several frames. In a dimly lit room, those early frames are frequently underexposed — dark enough that detection against a well-lit reference image produces false positives throughout the frame.
Discarding 10 frames is a heuristic, not a measured optimum. I don't have a precise measurement of how many frames this specific webcam needs for exposure to settle; 10 is conservative. What I can say is that adding the warm-up discard made capture quality consistent enough to feed into inference without a brightness-based pre-filter.
The alternative — a fixed exposure setting — would also work, but requires knowing the right value for the room's lighting and updating it when ambient light changes. The warm-up approach adapts automatically. For a headless device I interact with infrequently, adaptation beats configuration.
This is a pattern worth generalizing: in embedded vision pipelines, the gap between "inference is accurate" and "capture is reliable" is larger than ML tutorials suggest. Getting a well-exposed, correctly-framed image to the inference stage is where most of the unexplained detection variance shows up. The operational failure modes I documented were almost all in the capture and infrastructure layer, not in the model.
What I'd do differently
Measure the accuracy cost of the 416px resolution drop. I chose 416 px because the Pi could handle it and it worked. But I don't have the accuracy comparison against 640 px. If I were taking this beyond a PoC, I'd run the held-out test at both resolutions and quantify the tradeoff I made implicitly.
Test YOLO11s inference speed before dismissing it. The accuracy gap between YOLO11n and YOLO11s is real. I assumed the Pi 3 couldn't run the larger model at a useful speed — but I never measured it. That's a decision made on an assumption, not a measurement. It might be wrong.
Add structured inference logging from the start. The 8.5-second median and 8.4–11.8 second range come from 19 scans over the device's operating history. I got that sample by retrofitting timing instrumentation to a PoC that initially had none. Starting with structured scan logs — timestamp, inference duration, detection count, confirmation state — would have given a richer picture earlier, and would have flagged the scan collision risk before I had to reason about it after the fact. The output inspection principle I'd apply here: the pipeline that produces well-formed output but wrong content is the silent failure mode.
The post-processing design — ROI mask, baseline subtraction, temporal majority vote — is the part I'm most confident about. None of it depended on the deployment format or the inference resolution. A future version running on better hardware can keep all three stages and change nothing else. That's the right architecture property: the reliability mechanism is decoupled from the inference stack.
NCNN itself is also worth treating as a first-class dependency choice, not just a convenient export format. I wrote about the repos I keep returning to — NCNN is on that list specifically because the C++ runtime stays current and the ARM support is not an afterthought.
FAQ
Why not TFLite?
TFLite is a reasonable choice and Google publishes pre-quantized models in .tflite format for many architectures. For a YOLO model trained with Ultralytics and running headless on Linux without a dedicated ML accelerator (no Coral Edge TPU), NCNN's export path was more direct and the inference performance on CPU-only ARM was comparable. Without a TPU, both frameworks run on the same Cortex-A53 cores, so the differentiator is the export pipeline and the runtime library's ARM optimization depth.
Does the 36 MB NCNN binary fit comfortably on a Pi 3 SD card?
Yes. The weights, the inference script, the OS image, and several weeks of scan history occupy well under 1 GB combined. SD card exhaustion is a real failure mode to handle explicitly — not because of model weight size, but because accumulated scan output, system logs, and OS writes fill cards faster than you expect on a headless device with no manual cleanup.
Is NCNN faster than pure PyTorch CPU inference?
I don't have a benchmark comparing the two on this device. PyTorch CPU inference on a Pi 3 includes the overhead of the full torch runtime loading and the Python graph execution path. NCNN's C++ runtime eliminates that overhead. I expect NCNN is faster for this reason, but I won't claim a speedup ratio I haven't measured.
What accuracy does the system actually need?
For a PoC detecting whether any gap exists in a shelf region, the bar is: correctly flag the presence of an empty-space region. False negatives (missed gaps) matter more than false positives (flagged gaps that don't exist), because a missed gap means a product isn't restocked. The temporal majority vote — 2 of 3 consecutive scans confirming — reduces false positives at the cost of confirmation latency. At hourly cadence, a two-scan confirmation takes at most two hours, which is acceptable.
What would make the Pi 3 no longer the right hardware?
If the cadence needed to drop below 30 seconds, or if multi-shelf coverage required running several inference passes per cron cycle, the 8.5-second median inference would become a bottleneck. The Pi 3 is appropriate for a single shelf at hourly cadence. It is not the right hardware for anything requiring faster detection cycles or parallel coverage.
Related articles:
- The Three Layers That Made My 0.844 mAP Shelf Detector Useful on a Pi 3 — post-processing stages: ROI mask, baseline subtraction, temporal vote
- Three Headless Pi 3 Failure Modes I Had to Handle Before My Shelf Scanner Stayed Running — operational failure modes in the capture and infrastructure layer
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)