DEV Community

Todd Sullivan
Todd Sullivan

Posted on

The CoreML Model Was Fine. The Photo Was Too Big.

#ai #coreml #computervision #ios

I hit a very ordinary computer-vision failure this week: the model was doing what I asked, but the image pipeline was making the task impossible.

The app counts livestock on-device with a small YOLO/CoreML model. No network, no account, no server-side fallback. That constraint is useful, but it means the phone pipeline has to be honest about what the model actually sees.

The first photo path looked sensible:

let request = VNCoreMLRequest(model: model)
request.imageCropAndScaleOption = .scaleFit
handler.perform([request])
Enter fullscreen mode Exit fullscreen mode

One image in, detections out.

Then I tested it on a real paddock photo with roughly 250 sheep in it. The count came back as 14.

That was not a model mystery. A 12MP photo was being squeezed into the model input. By the time the flock reached the network, distant sheep were just a few pixels each. There was nothing clever for the detector to recover.

The fix was boring and much better than changing the model:

  1. run the full image once
  2. run overlapping tiles over the same image
  3. map tile-relative boxes back into image coordinates
  4. drop boxes cut by an inner tile edge
  5. merge the remaining detections with class-agnostic NMS

The important bit is that tiling changes the object size the model sees without changing the model.

let regions = [CGRect(x: 0, y: 0, width: 1, height: 1)]
    + Self.tiles(aspect: aspect, across: 3, overlap: 0.25)

for region in regions {
    request.regionOfInterest = region
}
Enter fullscreen mode Exit fullscreen mode

On a 4:3 photo that becomes 12 tiles plus the full-frame pass. More inference, but still on-device and still only a fraction of a second for a still photo.

The same paddock photo went from 14 to 173 detections. Not perfect, but suddenly useful. Close-up hen photos held steady, which mattered just as much: the fix improved recall where scale was the problem without inflating easy counts.

The non-obvious part was the merge step.

Default IoU-only NMS is not enough when tiling. A partial head box can sit inside a whole-animal box and have low IoU, so I also merge on intersection-over-smaller. I also make the merge class-agnostic because the same animal can come back under two labels from different passes. Count the animal, not the label argument.

The lesson was simple: before blaming the model, check what pixels you actually gave it.

For on-device AI, the model is often the least flexible part of the system. The image pipeline around it is where the cheap wins hide.

Top comments (0)