I built on-device workout rep counting in Flutter — here's what actually worked

Taka — Wed, 08 Jul 2026 18:49:31 +0000

I'm building TrainWiz, a Flutter app that turns real exercise into a pet-raising game: you do squats or push-ups, your phone counts the reps, and a little creature levels up and evolves. The core technical problem sounds trivial and absolutely is not: count reps from the camera, on-device, without uploading a single frame.

Here's what broke along the way, and what finally worked.

Why on-device

Two reasons: privacy and latency. A fitness camera that streams your body to a server is a non-starter for most people, and rep feedback has to feel instant or the whole "game" loop dies. So everything runs locally with tflite_flutter + an on-device pose model — no footage ever leaves the phone.

Naive attempt #1: joint-angle thresholds

The obvious approach: track the knee angle, count a rep when it dips below X° and comes back up.

// looks fine in a demo, dies in the real world
final kneeAngle = angleBetween(hip, knee, ankle);
if (!_down && kneeAngle < 100) _down = true;
if (_down && kneeAngle > 160) { reps++; _down = false; }

It demos beautifully. Then real users prop the phone on the floor, stand at an angle, and it falls apart.

The trap: a phone camera gives you 2D pose. A "120° knee angle" flattens completely depending on where the camera sits — the same squat reads as 90° or 150° purely from perspective. Lifting to 3D via the model's z doesn't save you either; monocular z is noisy enough that the angle jitters across your threshold and double-counts.

Naive attempt #2: a "body-line" gate

Next idea: figure out which exercise you're doing so I can pick the right signal. Standing (squat) vs. horizontal (push-up) should be easy — just check if shoulder, hip and heel form a straight line, right?

Wrong, again for the 2D reason. In a real push-up shot from the front-corner, shoulder–hip–heel are not collinear on the image plane — perspective bends them. I gated push-up counting on "body is a straight line" and it would just... stop counting mid-set. Nothing is more rage-inducing than an app that silently decides your perfectly good push-up doesn't count.

What actually worked

Two changes:

Stop trying to classify posture from geometry that perspective destroys. Instead of "is the body a straight line", use the orientation of the torso axis (vector from hips to shoulders). Vertical-ish → upright exercises, horizontal-ish → floor exercises. That single vector is far more robust to camera angle than any multi-joint colinearity check.
Make the rep signal swappable per exercise, and A/B it on the device. There is no single "rep signal" that works for everything, and you cannot pick the right one from a spreadsheet — you have to watch it run on a real body in a real room. I put every candidate signal (torso displacement, normalized limb travel, angle-with-smoothing) behind a debug switch so I could flip between them live and see which one stayed honest.

The meta-lesson: treat rep counting as a signals problem, not a "compute the angle" problem. 2D pose lies to you constantly; the robust features are the ones that survive an arbitrary camera position.

The stack

Flutter (iOS + Android from one codebase)
tflite_flutter + on-device pose estimation — everything local, nothing uploaded
Riverpod for state, drift (SQLite) as the local-first source of truth
Cloud sync is fire-and-forget on top of local writes, so a rep counts instantly even offline

Takeaways if you're doing camera ML on mobile

Assume 2D perspective will break any feature that depends on absolute angles or colinearity. Prefer relative orientation and displacement.
Monocular z from a pose model is a hint, not a measurement. Don't threshold on it.
Build a debug switch to hot-swap your signal on-device. The winning approach was not the one that looked best on paper — it was the one I could only find by flipping between them on a real rep.

If you want to see where it ended up, it's live here: TrainWiz (iOS + Android, free). Happy to answer anything about the pose pipeline in the comments.