The On-Device People Counter Only Worked After I Made It Boring

I built a tiny line-crossing people counter for a Seeed XIAO ESP32S3 Sense this week. Camera on the board, model on the board, counts on the board. No laptop in the loop unless I want the more accurate host-mode version.

The first useful constraint was admitting what the hardware could actually do.

The host detector runs at 416×416 and is about 1.1 GFLOPs. That is fine on a laptop. On an ESP32-S3 it is fantasy. The on-device detector is roughly 15× smaller and still takes about 118 ms per frame.

At QVGA the budget ended up like this:

JPEG decode      ~30 ms
preprocess        9 ms
model           118 ms
postprocess       2 ms
---------------------
total          ~160 ms

So the real target is not “real time” in the desktop-CV sense. It is about 6 fps, which is enough for someone walking through a doorway if the rest of the pipeline does not get clever.

The counter is deliberately plain:

1. capture the newest frame
2. detect people
3. use the bottom-center of each box as the foot point
4. match points with a nearest-centroid tracker
5. count when a track crosses a configured line
6. log the event to serial, optionally expose /counts over WiFi

The foot point mattered more than I expected. Box centroids move around with posture and crop shape. If the counting line is on the floor, the bottom-center of the box is a better approximation of the thing that actually crosses the line.

The tracker is also intentionally boring:

// ponytail: no motion model, no re-id. Fine for a doorway at ~6 fps;
// swap in ByteTrack if people cross paths in frame.
class Tracker {
  // id -> centroid, max distance, max age
};

No Kalman filter. No re-identification. No cloud fallback. Just keep a track alive for a few missed frames and drop it when it gets too stale.

That is not because those techniques are bad. It is because this is a doorway counter on a microcontroller. People mostly move in one direction, the camera points down, and the useful failure mode is obvious: if people cross paths in frame, this tracker is too small and should be replaced.

That shape is easier to operate than a “smart” pipeline pretending it has more signal than it does.

The board can serve its own tiny monitoring page when WiFi is configured:

GET /counts
{"in":14,"out":11,"occupancy":3}

But it also works serial-only with empty WiFi settings. That was another good constraint. The core counting path should not depend on a dashboard, a backend, or a phone app being alive.

The bit I like most is that the counting logic has a host self-check separate from the ESP-IDF build:

make -C firmware/test
# ok: 6 counter tests

For edge AI, that split is worth doing. The model and camera are the annoying parts to test. The line-crossing rules do not need to be. Pull the state machine into boring C++, beat it up on the host, then let the board worry about frames and inference.

The lesson was the same one on-device AI keeps teaching me: the model is only one budget. Latency, memory, camera format, mounting angle, testability, and failure honesty matter just as much.

A smaller model plus a boring counter beats an impressive pipeline that cannot keep up with someone walking through a door.