DEV Community

SoftwareDevs mvpfactory.io
SoftwareDevs mvpfactory.io

Posted on Originally published at mvpfactory.io

Wiring Core ML to a Real-Time Object Detection Pipeline with Vision Framework

---
title: "Wiring Core ML to a Real-Time Object Detection Pipeline with Vision Framework"
published: true
description: "Integrate quantized YOLOv8n into a live iOS camera pipeline — CVPixelBuffer formatting, ANE dispatch, NMS post-processing, and the 150MB memory budget that keeps your app alive."
tags: ios, swift, mobile, architecture
canonical_url: https://mvpfactory.co/blog/coreml-vision-realtime-object-detection
---
Enter fullscreen mode Exit fullscreen mode

What We Are Building

A production-grade real-time object detection pipeline on iOS using a quantized YOLOv8n Core ML model and Apple's Vision framework. By the end of this tutorial you will have CVPixelBuffer preprocessing that does not waste memory, explicit ANE dispatch that removes thermal unpredictability, and an on-device NMS implementation that keeps you inside the 150MB memory ceiling the OS enforces without warning.

Requirements: iOS 15+, Swift 5.9, Core ML 7.


Prerequisites

  • Xcode 15+
  • A YOLOv8n INT8-quantized .mlpackage model file
  • Familiarity with AVCaptureSession and AVCaptureVideoDataOutput

Step 1 — Pick the Right Model Variant

Let me show you a pattern I use in every project: start with YOLOv8n INT8, not FP32.

Model Variant Disk Size mAP50 (COCO) Latency (A15, ANE)
YOLOv8n FP32 ~24MB 37.3 ~18ms
YOLOv8n INT8 ~6MB 36.1 ~9ms
YOLOv8s INT8 ~11MB 44.9 ~14ms

Measured on iPhone 13 (A15), iOS 17.4, single warm inference, 640×640 input.

A 1.2 mAP drop is a worthwhile trade. You halve latency and reduce memory pressure — the two numbers that determine whether you sustain 30fps or stutter.


Step 2 — Preprocess CVPixelBuffer Correctly

Here is the gotcha that will save you hours: Vision's VNImageRequestHandler expects BGRA natively from AVCaptureVideoDataOutput. Use kCVPixelFormatType_32BGRA. Mismatching to RGBA forces an internal copy — your first hidden allocation.

func pixelBuffer(from ciImage: CIImage, size: CGSize, context: CIContext) -> CVPixelBuffer? {
    let attrs: [String: Any] = [
        kCVPixelBufferCGImageCompatibilityKey as String: true,
        kCVPixelBufferCGBitmapContextCompatibilityKey as String: true,
        kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA
    ]
    var pixelBuffer: CVPixelBuffer?
    guard CVPixelBufferCreate(
        kCFAllocatorDefault,
        Int(size.width), Int(size.height),
        kCVPixelFormatType_32BGRA,
        attrs as CFDictionary,
        &pixelBuffer
    ) == kCVReturnSuccess, let buffer = pixelBuffer else { return nil }

    CVPixelBufferLockBaseAddress(buffer, [])
    context.render(
        ciImage.transformed(by: CGAffineTransform(
            scaleX: size.width / ciImage.extent.width,
            y: size.height / ciImage.extent.height
        )),
        to: buffer,
        bounds: CGRect(origin: .zero, size: size),
        colorSpace: CGColorSpaceCreateDeviceRGB()
    )
    CVPixelBufferUnlockBaseAddress(buffer, [])
    return buffer
}
Enter fullscreen mode Exit fullscreen mode

Reuse a single CIContext across frames. Allocating one per frame is a common and expensive mistake.


Step 3 — Set Compute Units Explicitly

The docs do not mention this, but .automatic is not production-grade for a sustained camera pipeline.

let config = MLModelConfiguration()
config.computeUnits = .cpuAndNeuralEngine  // Not .all
Enter fullscreen mode Exit fullscreen mode

On iOS 15 and earlier, .automatic frequently fell back to GPU under thermal load. On iOS 16+, ANE dispatch is more predictable — but it still responds to device state you cannot observe at runtime. Using .all enables GPU fallback, which burns significantly more memory and generates thermal pressure. Forcing .cpuAndNeuralEngine reduced peak RAM by roughly 40MB compared to .all on the same device in production systems I have shipped.


Step 4 — Implement NMS On-Device

YOLO outputs raw bounding boxes with class probabilities. Vision does not run NMS for you with a custom Core ML model. Write your own.

func applyNMS(boxes: [Detection], iouThreshold: Float = 0.45) -> [Detection] {
    let sorted = boxes.sorted { $0.confidence > $1.confidence }
    var kept: [Detection] = []
    for box in sorted {
        if kept.allSatisfy({ iou($0.rect, box.rect) < iouThreshold }) {
            kept.append(box)
        }
    }
    return kept
}
Enter fullscreen mode Exit fullscreen mode

This is O(n²) against candidate count. For typical YOLO output after confidence filtering — usually under 100 candidates — it is fast enough. Above 100, replace the inner loop with a vectorized IoU matrix using Accelerate. Set your confidence threshold between 0.25 and 0.35. Below 0.25, false positives degrade UX. Above 0.40, you miss valid detections in occluded scenes.


The 150MB Memory Budget

Work backwards from the ceiling:

Component Approx. Memory
Core ML model (INT8) ~6MB
ANE working memory ~30MB
CVPixelBuffer pool (3 frames) ~15MB
App baseline + UI ~50MB
Headroom remaining ~49MB

That headroom disappears the moment you add a second model, increase buffer pool depth, or enable GPU fallback. Profile with Xcode's Memory Graph Debugger during active camera capture on the oldest device you support. Steady-state memory under inference load is the only number that matters.


Gotchas

.automatic is not .cpuAndNeuralEngine. The ambiguity costs you ~40MB under thermal load. Remove it.

NMS is your responsibility. Vision's built-in object recognition APIs abstract it away. A custom Core ML model does not. Benchmark your IoU threshold against your target domain and version it alongside the model.

didReceiveMemoryWarning is too late. The OS terminates background processes without ceremony. Measure during sustained inference, not at launch.


Conclusion

Here is the minimal setup to get this working in production: INT8 model, BGRA pixel buffers, explicit .cpuAndNeuralEngine, and your own NMS with a 0.45 IoU baseline. Everything else is tuning against your specific domain and device matrix.

Profile on your oldest supported hardware. That is the number that ships.

Further reading: Core ML documentation · Vision framework · Accelerate framework

Top comments (0)