---
title: "Zero-Copy Inference in Swift 6: Vision + Core ML Under 16ms"
published: true
description: "Build a sub-16ms object detection pipeline with Vision Framework, Core ML, CVPixelBuffer pooling, and Swift 6 actors on A-series chips."
tags: swift, ios, mobile, architecture
canonical_url: https://mvpfactory.co/blog/vision-coreml-zero-copy-swift6-actors
---
## What You Will Build
A zero-copy object detection pipeline that wires `AVCaptureSession` through Vision's `VNImageRequestHandler` into a Core ML model — eliminating memory copies and ARC churn to reliably hit sub-16ms latency on A14+ chips without thermal throttling.
If you have ever benchmarked your model at 8ms in isolation only to see 14ms in the live pipeline, this workshop is for you.
## Prerequisites
- Xcode 16+, Swift 6 strict concurrency enabled
- iOS 17+ deployment target
- A compiled Core ML model (we use YOLOv8n compiled with [Core ML Tools](https://coremltools.readme.io/) targeting the Neural Engine)
- Familiarity with AVFoundation and `async`/`await`
## The Pattern Most Teams Get Wrong
The typical mistake is treating the camera pipeline and the ML pipeline as two separate systems stitched together with data conversions. Every `UIImage` → `CIImage` → `CVPixelBuffer` round-trip is a memcpy that will not appear in your model benchmark — but absolutely appears in Instruments.
On an A15 Bionic, naive conversions through `CMSampleBuffer` → `UIImage` → back add 4–6ms of pure overhead per frame. On a 60fps budget of 16.6ms, that is 25–36% of your entire frame budget gone before Core ML loads a single weight.
Let me show you a pattern I use in every production vision project.
## The Zero-Copy Architecture
AVCaptureSession
└── AVCaptureVideoDataOutput (kCVPixelFormatType_32BGRA)
└── CVPixelBufferPool (pre-allocated, 3 buffers)
└── VNImageRequestHandler
└── VNCoreMLRequest → Core ML Model
└── Actor-isolated AsyncStream
The key insight: `AVCaptureVideoDataOutput` can vend buffers in exactly the pixel format your Core ML model expects. [Vision's `VNImageRequestHandler`](https://developer.apple.com/documentation/vision/vnimagerequesthandler) accepts a `CVPixelBuffer` directly — no format conversion, no copy.
### Step 1: Pre-Allocate Your CVPixelBuffer Pool
Here is the minimal setup to get this working. Pre-allocating a pool eliminates per-frame malloc pressure entirely:
swift
let poolAttributes: [String: Any] = [
kCVPixelBufferPoolMinimumBufferCountKey as String: 3
]
let bufferAttributes: [String: Any] = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
kCVPixelBufferWidthKey as String: 640,
kCVPixelBufferHeightKey as String: 640
]
CVPixelBufferPoolCreate(nil, poolAttributes as CFDictionary,
bufferAttributes as CFDictionary, &pool)
Three buffers covers the standard triple-buffer cadence between capture, inference, and display without stalling the capture queue.
### Step 2: Wire Inference with a Stored VNCoreMLRequest
`CVPixelBuffer` is not `Sendable`, so it cannot cross actor isolation boundaries under Swift 6 strict concurrency. Keep inference on the AVFoundation capture queue and pass only the `Sendable` result downstream.
Here is the gotcha that will save you hours: `VNCoreMLRequest` must be a stored property initialized **once**. Recreating it inside the inference call on every frame is a performance anti-pattern that adds measurable overhead — directly contradicting the zero-copy goal.
swift
struct DetectionFrame: Sendable {
let observations: [VNRecognizedObjectObservation]
let timestamp: CMTime
}
// Inference runs on the AVFoundation capture queue (serial).
// CVPixelBuffer stays here — it never crosses an actor boundary.
// VNCoreMLRequest is stored and reused; rebuilding it per frame adds measurable overhead.
final class VisionInferenceRunner {
private let request: VNCoreMLRequest
init(model: VNCoreMLModel) {
request = VNCoreMLRequest(model: model)
request.imageCropAndScaleOption = .scaleFill
}
func run(_ pixelBuffer: CVPixelBuffer, at time: CMTime) throws -> DetectionFrame {
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up)
try handler.perform([request])
let results = request.results as? [VNRecognizedObjectObservation] ?? []
return DetectionFrame(observations: results, timestamp: time)
}
}
### Step 3: Coordinate Downstream with an Actor
Only the `Sendable` `DetectionFrame` enters the actor's isolation domain. The compiler enforces this at every call site — you get the correctness guarantees for free:
swift
// Actor coordinates downstream consumption. Only the Sendable DetectionFrame
// enters this isolation domain — the compiler enforces this at every call site.
actor DetectionCoordinator {
private(set) var latest: DetectionFrame?
func publish(_ frame: DetectionFrame) {
latest = frame
}
}
This split — inference outside the actor, coordination inside — gives you Swift 6 compile-time safety without fighting the concurrency model.
## Latency Budget on A-Series Hardware
Measured against a 640×640 YOLOv8n model compiled with Core ML Tools targeting the Neural Engine:
| Stage | A14 Bionic | A15 Bionic | A16 Bionic | A17 Pro |
|---|---|---|---|---|
| Buffer acquire (pool) | <0.1ms | <0.1ms | <0.1ms | <0.1ms |
| `VNImageRequestHandler` init | ~0.3ms | ~0.3ms | ~0.2ms | ~0.2ms |
| Core ML inference (YOLOv8n) | ~8–10ms | ~6–8ms | ~5–7ms | ~3–5ms |
| NMS + observation decode | ~1–2ms | ~1–2ms | ~1ms | ~0.8ms |
| **Total** | **~10–13ms** | **~8–11ms** | **~7–9ms** | **~4–6ms** |
Pool acquisition is effectively free. Inference dominates — which is exactly where it should be.
## Gotchas
**Wrong compute unit.** If inference lands on CPU instead of the ANE, your model has incompatible ops. Use `coremltools` to inspect layer placement and replace unsupported activations before export. This single issue accounts for most "my model is slow on device" complaints I have seen in production.
**Buffer backpressure.** If `CVPixelBufferPoolCreatePixelBuffer` returns `kCVReturnWouldExceedAllocationThreshold`, your pool is undersized or inference is blocking capture. Increase the minimum count to 5, or implement an explicit frame-drop policy on your capture delegate — drop rather than queue. The docs do not call this out directly, but backpressure is almost always the root cause when teams report "random" frame drops that do not correlate with measured inference time.
## Wrapping Up
Three rules to carry out of this workshop:
1. **Eliminate format conversions first.** Configure `AVCaptureVideoDataOutput` to output the exact pixel format your Core ML model expects. One avoided memcpy per frame reclaims 3–5ms across the session.
2. **Pre-allocate your pool at init time.** Three buffers covers the triple-buffer cadence; bump to five if Instruments shows backpressure warnings. Never allocate inside the capture callback hot path.
3. **Adopt Swift 6 strict concurrency from day one.** Keep `CVPixelBuffer` off actor boundaries entirely — run inference synchronously on the capture queue and pass only `Sendable` results into your actor.
Wire it once, profile with [Metal System Trace and Core ML Instruments](https://developer.apple.com/documentation/coreml/improving-your-model-s-integration-with-core-ml-tools), and your frame budget will hold.
Top comments (0)