---
title: "Wiring Core ML to a Real-Time Object Detection Pipeline with Vision Framework"
published: true
description: "Integrate quantized YOLOv8n into a live iOS camera pipeline — CVPixelBuffer formatting, ANE dispatch, NMS post-processing, and the 150MB memory budget that keeps your app alive."
tags: ios, swift, mobile, architecture
canonical_url: https://mvpfactory.co/blog/coreml-vision-realtime-object-detection
---
What We Are Building
A production-grade real-time object detection pipeline on iOS using a quantized YOLOv8n Core ML model and Apple's Vision framework. By the end of this tutorial you will have CVPixelBuffer preprocessing that does not waste memory, explicit ANE dispatch that removes thermal unpredictability, and an on-device NMS implementation that keeps you inside the 150MB memory ceiling the OS enforces without warning.
Requirements: iOS 15+, Swift 5.9, Core ML 7.
Prerequisites
- Xcode 15+
- A YOLOv8n INT8-quantized
.mlpackagemodel file - Familiarity with
AVCaptureSessionandAVCaptureVideoDataOutput
Step 1 — Pick the Right Model Variant
Let me show you a pattern I use in every project: start with YOLOv8n INT8, not FP32.
| Model Variant | Disk Size | mAP50 (COCO) | Latency (A15, ANE) |
|---|---|---|---|
| YOLOv8n FP32 | ~24MB | 37.3 | ~18ms |
| YOLOv8n INT8 | ~6MB | 36.1 | ~9ms |
| YOLOv8s INT8 | ~11MB | 44.9 | ~14ms |
Measured on iPhone 13 (A15), iOS 17.4, single warm inference, 640×640 input.
A 1.2 mAP drop is a worthwhile trade. You halve latency and reduce memory pressure — the two numbers that determine whether you sustain 30fps or stutter.
Step 2 — Preprocess CVPixelBuffer Correctly
Here is the gotcha that will save you hours: Vision's VNImageRequestHandler expects BGRA natively from AVCaptureVideoDataOutput. Use kCVPixelFormatType_32BGRA. Mismatching to RGBA forces an internal copy — your first hidden allocation.
func pixelBuffer(from ciImage: CIImage, size: CGSize, context: CIContext) -> CVPixelBuffer? {
let attrs: [String: Any] = [
kCVPixelBufferCGImageCompatibilityKey as String: true,
kCVPixelBufferCGBitmapContextCompatibilityKey as String: true,
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA
]
var pixelBuffer: CVPixelBuffer?
guard CVPixelBufferCreate(
kCFAllocatorDefault,
Int(size.width), Int(size.height),
kCVPixelFormatType_32BGRA,
attrs as CFDictionary,
&pixelBuffer
) == kCVReturnSuccess, let buffer = pixelBuffer else { return nil }
CVPixelBufferLockBaseAddress(buffer, [])
context.render(
ciImage.transformed(by: CGAffineTransform(
scaleX: size.width / ciImage.extent.width,
y: size.height / ciImage.extent.height
)),
to: buffer,
bounds: CGRect(origin: .zero, size: size),
colorSpace: CGColorSpaceCreateDeviceRGB()
)
CVPixelBufferUnlockBaseAddress(buffer, [])
return buffer
}
Reuse a single CIContext across frames. Allocating one per frame is a common and expensive mistake.
Step 3 — Set Compute Units Explicitly
The docs do not mention this, but .automatic is not production-grade for a sustained camera pipeline.
let config = MLModelConfiguration()
config.computeUnits = .cpuAndNeuralEngine // Not .all
On iOS 15 and earlier, .automatic frequently fell back to GPU under thermal load. On iOS 16+, ANE dispatch is more predictable — but it still responds to device state you cannot observe at runtime. Using .all enables GPU fallback, which burns significantly more memory and generates thermal pressure. Forcing .cpuAndNeuralEngine reduced peak RAM by roughly 40MB compared to .all on the same device in production systems I have shipped.
Step 4 — Implement NMS On-Device
YOLO outputs raw bounding boxes with class probabilities. Vision does not run NMS for you with a custom Core ML model. Write your own.
func applyNMS(boxes: [Detection], iouThreshold: Float = 0.45) -> [Detection] {
let sorted = boxes.sorted { $0.confidence > $1.confidence }
var kept: [Detection] = []
for box in sorted {
if kept.allSatisfy({ iou($0.rect, box.rect) < iouThreshold }) {
kept.append(box)
}
}
return kept
}
This is O(n²) against candidate count. For typical YOLO output after confidence filtering — usually under 100 candidates — it is fast enough. Above 100, replace the inner loop with a vectorized IoU matrix using Accelerate. Set your confidence threshold between 0.25 and 0.35. Below 0.25, false positives degrade UX. Above 0.40, you miss valid detections in occluded scenes.
The 150MB Memory Budget
Work backwards from the ceiling:
| Component | Approx. Memory |
|---|---|
| Core ML model (INT8) | ~6MB |
| ANE working memory | ~30MB |
| CVPixelBuffer pool (3 frames) | ~15MB |
| App baseline + UI | ~50MB |
| Headroom remaining | ~49MB |
That headroom disappears the moment you add a second model, increase buffer pool depth, or enable GPU fallback. Profile with Xcode's Memory Graph Debugger during active camera capture on the oldest device you support. Steady-state memory under inference load is the only number that matters.
Gotchas
.automatic is not .cpuAndNeuralEngine. The ambiguity costs you ~40MB under thermal load. Remove it.
NMS is your responsibility. Vision's built-in object recognition APIs abstract it away. A custom Core ML model does not. Benchmark your IoU threshold against your target domain and version it alongside the model.
didReceiveMemoryWarning is too late. The OS terminates background processes without ceremony. Measure during sustained inference, not at launch.
Conclusion
Here is the minimal setup to get this working in production: INT8 model, BGRA pixel buffers, explicit .cpuAndNeuralEngine, and your own NMS with a 0.45 IoU baseline. Everything else is tuning against your specific domain and device matrix.
Profile on your oldest supported hardware. That is the number that ships.
Further reading: Core ML documentation · Vision framework · Accelerate framework
Top comments (0)