Machine learning tutorials stop at model accuracy. The model trains, reaches a reasonable mAP50, exports to a deployment format, and the tutorial ends. The gap between "inference script works on my laptop" and "this thing runs unattended on a Pi 3 in a room for a week" is where the interesting problems live.
I ran an empty-shelf detector on a Raspberry Pi 3 Model B Rev 1.2 — 906 MB RAM, headless, USB webcam, Raspberry Pi OS Bookworm arm64 Lite — and did an audit pass specifically to catch the failure modes the model work didn't surface. Three of them mattered enough to explicitly handle before I trusted the system to produce useful results. The Pi has since produced 19 real scans and correctly detected a gap.
Failure 1: Scan collisions from overlapping cron runs
A cron-based scanner fires on a schedule. On a Pi 3, an inference run takes a median 8.5 seconds (n=19 real scans, range 8.4–11.8 s). That number is slow enough that if a cron job fires while the previous run is still completing — because the Pi was busy, the SD card was slow, or the camera stalled — you get two concurrent scan processes competing for the same hardware.
The symptom is subtle: scans succeed individually, but the temporal majority vote that determines whether a detection is confirmed (2 of 3 consecutive scans) starts reading from a corrupt sequence. A collision mid-history produces a stale detection that looks confirmed because the timestamp ordering is wrong.
The fix is the standard lock-file pattern, but the specific lesson is: on constrained hardware with a fixed cadence, "won't happen in practice" is wrong. An 8.5 s median with an 11.8 s maximum in the same 19-run sample is wide enough to cause overlap under any realistic cron interval shorter than 30 seconds.
Failure 2: SD card exhaustion
The Pi writes scan output — bounding boxes, timestamps, detection history — to the SD card. It also writes system logs, fswebcam temporary files, and whatever the OS accumulates during normal operation. An SD card that fills up does not raise a clear error; cron jobs silently fail to write, scan history stops updating, and the system looks like it is running when it is producing nothing.
The operational check I added after finding this: a pre-scan assertion on available disk space before writing anything. If the threshold is not met, the scan exits without writing and logs the failure clearly. Without that check, the system was "green" at the process level — cron fired, the binary ran, no crash — while producing no useful output.
This is the same pattern as five silent failures in a content pipeline: a well-formed job that fails to write is not the same as a job that crashes. Process-level monitoring catches the latter; only output inspection catches the former.
Failure 3: Stale history and atomic writes
The temporal majority vote reads the last three scan results from a history file. That file is written after every scan. If the write is not atomic — if a crash or power interruption occurs between the old file being removed and the new file being written — the history becomes incomplete or corrupt.
The vote that follows reads a truncated history and either skips a detection that should be confirmed, or confirms one that shouldn't be. On a Pi 3 without a UPS, this is not theoretical: power interruptions happen, and the SD card is not designed for write-durability under sudden loss.
The fix is the standard write-to-temp-then-rename pattern. A completed file rename is atomic on any POSIX filesystem; a partially written temp file is discarded, and the last complete history file survives intact. The lesson is that it needs to be explicit — the first version of the script wrote directly, and the audit caught it.
The Wi-Fi self-repair problem
The Pi sits in a room without a keyboard or monitor. If Wi-Fi drops and doesn't reconnect, there is no remote access and no way to restart without physically carrying the Pi to a keyboard. The SD card provisioning script I wrote on the Mac includes a firstrun hook with explicit Wi-Fi self-repair: if the connection drops, the hook attempts reconnection on a schedule. This means the Pi comes back without human intervention after a router restart or transient dropout.
It is not a sophisticated solution, and it is not something I would have written without experiencing the problem first.
What the model work didn't teach me
The mAP50 of 0.844 on the held-out test set, the post-processing layers documented in the three-layers post, the NCNN export — all of that determines what the model does when it runs. None of it determines whether it runs reliably for a week in a room.
The operational layer — lock files, disk assertions, atomic writes, network self-repair — is invisible to any model benchmark. It is also what determines whether the system is actually a system or just a demo that works when you're watching it.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)