I'm building TrainWiz, a Flutter app that turns real exercise into a pet-raising game: you do squats or push-ups, your phone counts the reps, and a little creature levels up and evolves. The core technical problem sounds trivial and absolutely is not: count reps from the camera, on-device, without uploading a single frame.
Here's what broke along the way, and what finally worked.
Why on-device
Two reasons: privacy and latency. A fitness camera that streams your body to a server is a non-starter for most people, and rep feedback has to feel instant or the whole "game" loop dies. So everything runs locally with tflite_flutter + an on-device pose model — no footage ever leaves the phone.
Naive attempt #1: joint-angle thresholds
The obvious approach: track the knee angle, count a rep when it dips below X° and comes back up.
// looks fine in a demo, dies in the real world
final kneeAngle = angleBetween(hip, knee, ankle);
if (!_down && kneeAngle < 100) _down = true;
if (_down && kneeAngle > 160) { reps++; _down = false; }
It demos beautifully. Then real users prop the phone on the floor, stand at an angle, and it falls apart.
The trap: a phone camera gives you 2D pose. A "120° knee angle" flattens completely depending on where the camera sits — the same squat reads as 90° or 150° purely from perspective. Lifting to 3D via the model's z doesn't save you either; monocular z is noisy enough that the angle jitters across your threshold and double-counts.
Naive attempt #2: a "body-line" gate
Next idea: figure out which exercise you're doing so I can pick the right signal. Standing (squat) vs. horizontal (push-up) should be easy — just check if shoulder, hip and heel form a straight line, right?
Wrong, again for the 2D reason. In a real push-up shot from the front-corner, shoulder–hip–heel are not collinear on the image plane — perspective bends them. I gated push-up counting on "body is a straight line" and it would just... stop counting mid-set. Nothing is more rage-inducing than an app that silently decides your perfectly good push-up doesn't count.
What actually worked
Two changes:
Stop trying to classify posture from geometry that perspective destroys. Instead of "is the body a straight line", use the orientation of the torso axis (vector from hips to shoulders). Vertical-ish → upright exercises, horizontal-ish → floor exercises. That single vector is far more robust to camera angle than any multi-joint colinearity check.
Make the rep signal swappable per exercise, and A/B it on the device. There is no single "rep signal" that works for everything, and you cannot pick the right one from a spreadsheet — you have to watch it run on a real body in a real room. I put every candidate signal (torso displacement, normalized limb travel, angle-with-smoothing) behind a debug switch so I could flip between them live and see which one stayed honest.
The meta-lesson: treat rep counting as a signals problem, not a "compute the angle" problem. 2D pose lies to you constantly; the robust features are the ones that survive an arbitrary camera position.
The stack
- Flutter (iOS + Android from one codebase)
- tflite_flutter + on-device pose estimation — everything local, nothing uploaded
- Riverpod for state, drift (SQLite) as the local-first source of truth
- Cloud sync is fire-and-forget on top of local writes, so a rep counts instantly even offline
Takeaways if you're doing camera ML on mobile
- Assume 2D perspective will break any feature that depends on absolute angles or colinearity. Prefer relative orientation and displacement.
- Monocular
zfrom a pose model is a hint, not a measurement. Don't threshold on it. - Build a debug switch to hot-swap your signal on-device. The winning approach was not the one that looked best on paper — it was the one I could only find by flipping between them on a real rep.
If you want to see where it ended up, it's live here: TrainWiz (iOS + Android, free). Happy to answer anything about the pose pipeline in the comments.
Top comments (0)