---
title: "CoreML in iOS 18: Detecting Silent CPU Fallback Before It Breaks Your Real-Time Pipeline"
published: true
description: "CoreML's compute unit scheduler silently demotes ops to CPU with no warning. Learn to detect CPU fallback at runtime with MLComputePlan, instrument ANE dispatch with os_signpost, and control delegation at compile time."
tags: ios, swift, mobile, architecture
canonical_url: https://mvpfactory.co/blog/coreml-compute-units-ios18-ane-gpu-cpu-fallback
---
## What You Will Build
By the end of this tutorial you will have a CI-ready validation pass that enumerates per-operation compute device assignments using `MLComputePlan`, an `os_signpost` instrumentation wrapper to isolate ANE vs CPU dispatch in Instruments, and a clear mental model of why `.all` is not the same as "always ANE."
Let me show you a pattern I use in every project that ships on-device inference.
## Prerequisites
- Xcode 16, iOS 18 deployment target
- A compiled `.mlmodelc` or `.mlpackage`
- Basic familiarity with CoreML model loading
- Python + `coremltools` for the conversion step
---
## Step 1: Understand the Compute Unit Hierarchy
CoreML exposes four policies via `MLComputeUnits`:
| Policy | What it actually does |
|---|---|
| `.all` | ANE preferred, GPU fallback, CPU last resort |
| `.cpuAndNeuralEngine` | ANE + CPU only, no GPU |
| `.cpuAndGPU` | GPU + CPU, ANE explicitly excluded |
| `.cpuOnly` | Deterministic, slowest, no delegation |
The default `.all` sounds optimal. In practice it is a black box that hands your op graph to a scheduler you cannot observe without deliberate instrumentation.
Here is the gotcha that will save you hours: **a single unsupported op forces an entire subgraph onto the CPU**, not just that op. No exception. No warning in the logs. It happens silently at model load time.
---
## Step 2: Detect CPU Fallback with MLComputePlan
`MLComputePlan`, available since iOS 17, lets you enumerate per-operation compute device assignments before you run a single inference pass.
swift
import CoreML
let config = MLModelConfiguration()
config.computeUnits = .all
let modelURL = Bundle.main.url(forResource: "MyModel", withExtension: "mlmodelc")!
MLComputePlan.load(contentsOf: modelURL, configuration: config) { result in
guard case .success(let plan) = result,
case .program(let program) = plan.modelStructure else { return }
for (functionName, function) in program.functions {
for block in function.blocks {
for op in block.operations {
guard let usage = plan.computeDeviceUsage(for: op) else { continue }
switch usage.preferredComputeDevice {
case .cpu:
print("[CPU FALLBACK] \(op.operator.name) in '\(functionName)'")
case .gpu:
print("[GPU] \(op.operator.name)")
case .neuralEngine:
break
@unknown default:
break
}
}
}
}
}
Run this in your CI validation pass, not just locally. A model that delegates cleanly on the simulator may fall back on physical hardware due to ANE microarchitecture differences between A-series generations.
---
## Step 3: Instrument Inference with os_signpost
Pair the `MLComputePlan` analysis with signpost instrumentation to isolate dispatch overhead from compute time in Instruments:
swift
import os.signpost
let log = OSLog(subsystem: "com.yourapp.inference", category: .pointsOfInterest)
func runInference(input: MLFeatureProvider) throws -> MLFeatureProvider {
let id = OSSignpostID(log: log)
os_signpost(.begin, log: log, name: "CoreML Inference", signpostID: id)
defer { os_signpost(.end, log: log, name: "CoreML Inference", signpostID: id) }
return try model.prediction(from: input)
}
Open Instruments → Core ML Instrument. Any CPU segments in a model configured as `.all` confirm fallback — and you now have the `MLComputePlan` output to pinpoint the offending op.
---
## Step 4: Set the Right Deployment Target at Conversion Time
python
import coremltools as ct
model = ct.convert(
source_model,
compute_units=ct.ComputeUnit.ALL,
minimum_deployment_target=ct.target.iOS18
)
model.save("MyModel.mlpackage")
The docs do not mention this clearly, but `minimum_deployment_target` unlocks ANE op support unavailable when targeting earlier OS versions — even on iOS 18 hardware running a model compiled for iOS 16. If you left this at `iOS15` and cannot explain unexpected latency, that is almost certainly why.
---
## Gotchas
**Common ANE-incompatible patterns that trigger silent fallback:**
- Dynamic shapes without a fixed upper bound
- Custom `MLCustomLayer` implementations
- Certain gather/scatter ops with non-static indices
- `bfloat16` activations on older ANE generations
**The latency gap is not theoretical.** On an iPhone 15 Pro (A17 Pro) with a MobileNetV3-style vision model:
| Compute path | Latency (approx.) |
|---|---|
| ANE | 8–12ms |
| GPU | 15–25ms |
| CPU | 60–120ms |
A real-time pose estimation pipeline at 30fps has roughly 33ms per frame — the kind of tight budget you would see in a guided exercise app like [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk), where frame deadline misses translate directly to broken movement tracking. One subgraph on CPU blows past that ceiling before you catch it in testing.
---
## Three Actionable Takeaways
1. **Assert zero CPU fallback in CI.** Run the `MLComputePlan` enumeration against every model update and fail the build if any op on your critical path resolves to `.cpu` with `.all` configured.
2. **Set `minimum_deployment_target` to your actual floor.** Mismatched targets silently suppress ANE ops your hardware fully supports. The fix is one argument in your conversion script.
3. **Isolate known-incompatible ops into a separate `.cpuOnly` model.** If your graph contains a custom layer or unsupported op you cannot remove, partition it out explicitly so it cannot contaminate subgraph scheduling for the rest of your ANE-bound pipeline.
CoreML's automatic compute unit selection is not magic — it is a graph-partitioning algorithm with well-defined failure modes you can now observe and control. Instrument first, ship with confidence.
Top comments (0)