Tracking the right player on a crowded pickleball court with Apple Vision

I'm building Pickle Sensei, an iPhone app that coaches your pickleball technique. You prop the phone up beside the court, hit one stroke, and the app scores your form and gives you one thing to fix. Everything runs on the device. Video and pose data never leave the phone.

The stack is React Native 0.87 and TypeScript for the app, with a native Swift module that runs Apple's VNDetectHumanBodyPoseRequest on camera frames. Supabase handles accounts, RevenueCat handles purchases.

Getting a body skeleton out of Vision is the easy part. The hard part is everything a real court throws at you. Here are three problems that broke the first versions and how they got fixed.

1. Which person is "the player"?

A pickleball court is never empty. There's your partner, your opponents, people walking behind the fence. Vision happily returns a pose for all of them.

The first approach was simple, pick the person with the largest torso, measured from the shoulder midpoint to the hip midpoint in normalized coordinates. People missing those joints get a tiny score, so a full-body detection always beats a fragment.

That works on frame one and falls apart on frame forty, when someone walks closer to the camera. So the score is weighted by distance to where the selected player was last frame.

func score(_ observation: VNHumanBodyPoseObservation) -> Double {
  let span = torsoSpan(observation)
  guard let anchor, let mid = torsoMid(observation) else { return span }
  let distance = Double(hypot(mid.x - anchor.x, mid.y - anchor.y))
  return span / (1.0 + 3.0 * distance)
}

Distance decay alone still lost the athlete to a decisively larger newcomer. The fix was incumbent hysteresis. The candidate nearest the previous anchor (within 0.12 image units) keeps the identity unless a challenger beats its score by about 1.43x. On 36 replayed test cases, the share of frames locked on the right person after the initial lock went from 0.54 to 0.61.

There's also a manual override. Tapping a person seeds the anchor directly, which taught one more lesson. Vision uses a bottom-left origin and UIKit uses top-left, and an untested y-flip on that seed silently regressed once. It has a test now.

2. Walking is not a forehand

Stroke detection is a small state machine over wrist and paddle speed, with a minimum-confidence trigger and a refractory period so paddle twirls and picking up a ball don't count.

The early version measured absolute motion in the image. A player walking to the baseline moves their whole body, plus their arm swings, and that crossed the trigger threshold. A camera bump moved every pixel and looked like a swing too.

Two changes fixed it.

• All speeds are measured in body-heights per second, the displacement divided by the observed body scale, so thresholds describe the athlete instead of the camera distance.
• Motion is measured relative to the body. The hip-midpoint displacement over the same interval is subtracted from the wrist displacement. Walking now contributes only the arm swing, and a camera bump reads close to zero. If the hips aren't visible in a frame, that frame produces no sample at all rather than falling back to absolute speed.

3. A readiness window that was never ready at 30 fps

Before capturing, the app waits for the player to stand still in the frame for a moment. The readiness evaluator kept a rolling window of recent "stable" samples and pruned everything older than the window length.

The bug was subtle. Pruning everything strictly older than the cutoff meant the oldest remaining sample was almost always younger than the window, unless a frame landed exactly on the cutoff. At 30 fps a perfectly still player was never "ready", and at 60 fps with dropped frames readiness flickered.

The fix keeps the newest sample at or before the cutoff as the window's anchor. The same change made the PoseFrame boundary drop landmarks with non-finite coordinates and treat a non-finite confidence as zero, because one corrupt sample was enough to poison the body-scale moving average that everything downstream depends on.

What I'd tell anyone doing on-device pose work

• Test at real camera cadence, not at idealized timestamps. Half these bugs only exist at 30 fps with jitter.
• Normalize by body size early. It makes every threshold portable across camera placements.
• When the input is bad, say so. The app refuses to score a clip it can't trust and explains why, instead of guessing.

If you play pickleball, the app is free to try with one rating: Pickle Sensei on the App Store. Happy to answer questions about the pose pipeline in the comments.

Disclosure, I used AI tools to help build the app and to draft this post from the project's code and commit notes. The details above come from the actual source.