The Tracking Bug Wasn't the Model. It Was the Dot Lying.

I spent part of this week debugging an on-device livestock counter in a field-video mode. The model was fine. The count was not.

The symptom from the real device run was simple: a tracking dot would leave one animal, drift onto its neighbour for a few frames, then snap back. In a dense mob that is exactly the kind of tiny lie that ruins the user's trust. If the UI says that sheep is being tracked, it had better mean it.

The first version of the counter was deliberately boring:

• detect animals with a small CoreML model
• maintain short-lived tracks between frames
• count when a track crosses a tripwire line
• let missed tracks coast for a few frames so one bad detection does not drop the animal

That worked in still-ish tests. It failed once the phone and the sheep were both moving.

Prediction is not presentation

The lazy assumption was that the predicted track position could also be the drawn dot position.

Bad assumption.

A tracker is allowed to guess. A UI marker is a claim. Those are different contracts.

A track can go unseen for up to 15 frames, because dropping it too quickly loses animals behind brief occlusion. During that time prediction uses the last observed heading plus camera-motion compensation. Good for matching. Bad for display. On a slow pan, a neighbouring sheep is about one body length away, so a few frames of coasting was enough to draw the dot onto the wrong animal.

The fix was smaller than the bug report made it sound:

let visibleCoastLimit = 2
let isDisplayPrediction = track.missedFrames > 0

if track.missedFrames <= visibleCoastLimit {
    drawDot(at: track.predictedCenter,
            opacity: isDisplayPrediction ? 0.45 : 1.0,
            scale: isDisplayPrediction ? 0.7 : 1.0)
}

Matching can keep the longer memory. Display gets the honest one.

Centroid distance breaks in a flock

The second bug was association cost.

I had already moved past purely greedy matching, but the cost was still mostly centroid distance. That sounds reasonable until you put the phone in front of a packed group of sheep. Every animal has another animal a body length away. Centroids are not enough signal.

The better signal is overlap.

Two boxes on the same animal overlap heavily between adjacent frames. Two boxes on neighbouring animals usually do not. So the association cost now prefers overlap first and only falls back to distance when there is no meaningful overlap:

if iou > 0 {
    cost = 1 - iou
} else {
    cost = 1 + normalisedDistance
}

That one change makes the tracker much less willing to jump to a neighbour just because its centre is nearby. It also means tracks need to keep the whole observed box, not just a centre point and longest side. Slightly more state, much better truth.

The useful lesson

Most ML product bugs are not model bugs.

The detector can be good enough and the product can still feel broken because the tracking layer lies, the UI overstates certainty, or the recovery path optimizes for the wrong thing.

In this case the fix was not a new model, a heavier tracker, or a dependency. It was separating prediction from presentation, then using the geometry the video already had.

Boring, local, testable. My favourite kind of AI fix.