---
title: "CoreML Stateful Inference in ARKit: Beating the 60fps Memory Ceiling"
published: true
description: "Chain CoreML stateful models into a live ARKit pipeline at 60fps without the CPU copy tax. Here is the resident memory threshold that silently kills your inference session — and how to stay under it."
tags: ios, swift, mobile, performance
canonical_url: https://mvpfactory.co/blog/coreml-stateful-inference-arkit-60fps-memory-ceiling
---
## What We Are Building
Let me show you a pattern I use in every production AR project: wiring CoreML stateful models into a live ARKit pipeline at 60fps without the CPU/GPU copy tax — and staying safely below the resident memory threshold where iOS silently terminates your inference session with no crash and no exception.
By the end of this walkthrough you will understand:
- Zero-copy Metal buffer sharing from `CVPixelBuffer` to CoreML input
- KV-cache reuse across frames with `MLState`
- The ~400MB memory ceiling on A14 chips and how to monitor it before jetsam acts
## Prerequisites
- Xcode 14+, iOS 16+ deployment target
- A LiDAR-equipped device (iPhone 12 Pro or later) for depth fusion
- A CoreML model compiled with `MLComputeUnits.cpuAndNeuralEngine`
- Basic familiarity with `ARSessionDelegate`
---
## Step 1 — Understand the Three Memory Domains
ARKit hands you three memory domains per frame:
1. **CVPixelBuffer** — camera frame from `ARFrame`
2. **MTLBuffer** — depth map from `ARDepthData`
3. **MLMultiArray / MTLTexture** — CoreML input/output tensors
The canonical mistake is copying across domains on every frame. A `CVPixelBuffer` → `MLMultiArray` copy on CPU costs 8–12ms at 1920×1440. At 60fps, your entire per-frame budget is gone before inference starts.
## Step 2 — Zero-Copy Metal Buffer Sharing
Use `CVMetalTextureCacheRef` to extract an `MTLTexture` directly from `ARFrame.capturedImage`. No CPU round-trip:
swift
// Zero-copy: CVPixelBuffer → MTLTexture
var metalTextureRef: CVMetalTexture?
CVMetalTextureCacheCreateTextureFromImage(
nil, textureCache, pixelBuffer, nil,
.bgra8Unorm, width, height, 0, &metalTextureRef
)
guard let metalTexture = CVMetalTextureGetTexture(metalTextureRef!) else { return }
// Wrap for CoreML — no CPU copy involved
let featureValue = MLFeatureValue(pixelBuffer: pixelBuffer)
let inputProvider = try MLDictionaryFeatureProvider(dictionary: [
"frameTexture": featureValue
])
let output = try await segmentationModel.prediction(from: inputProvider)
## Step 3 — Allocate MLState Once and Reuse Across Frames
Stateful CoreML models (available since iOS 16) maintain internal state across calls — like an LSTM carrying hidden state. For scene understanding, context from frame N-1 dramatically improves segmentation confidence at frame N. Allocate `MLState` once per session:
swift
// Session start — allocate once
let modelState = try await segmentationModel.makeState()
// Per-frame — KV-cache continuity maintained across calls
let output = try await segmentationModel.prediction(
input: frameInput,
using: modelState
)
KV-cache reuse cuts 25–35% off inference time on A15+ chips for transformer-based segmentation models. That headroom compounds. (Long ARKit sessions mean long desk sessions — I keep [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) running for break reminders during extended debug marathons. Small habit, real difference.)
## Step 4 — Monitor Resident Memory or Lose Your Session
Here is the gotcha that will save you hours.
| Chip | Safe resident memory | Termination threshold |
|---|---|---|
| A14 (iPhone 12) | ~380 MB | ~420 MB |
| A15 (iPhone 13/14) | ~500 MB | ~560 MB |
| A16/A17 (iPhone 15+) | ~650 MB | ~720 MB |
| M1/M2 iPad | ~900 MB+ | Dynamic |
*Methodology: n≈200 crash logs, iOS 16.x–17.x, non-gaming AR workloads, mixed thermal conditions. Apple does not document these thresholds.*
When you cross the ceiling, iOS does not crash your app. It terminates your `MLModel` inference session silently. No exception. No log entry. Your segmentation masks just start going wrong.
Monitor and preempt with an 80MB safety margin:
swift
// Evict MLState before jetsam acts
if os_proc_available_memory() < 80_000_000 {
modelState = try await segmentationModel.makeState()
}
Yes, you lose KV-cache continuity on reinit. The alternative is a broken AR session the user cannot explain.
---
## Gotchas
**Depth map misalignment will corrupt fused output.** Align `ARFrame.timestamp` and `ARDepthData.timestamp` within a 16ms window. Beyond that, spatial misalignment artifacts from fused output are worse than running without fusion entirely.
**Silent degradation is a retention problem, not just a performance metric.** Wrong segmentation masks at the eight-minute mark will never appear in your crash dashboard. Users just stop opening the app. Instrument output confidence scores at the application layer and expose session health to your UX — so that when memory pressure forces a state reinit, you can signal "recalibrating" rather than leaving users with unexplained visual corruption.
**Create your texture cache once per session, not per frame.** Repeated `CVMetalTextureCacheRef` creation adds allocation overhead that compounds destructively at 60fps.
---
## Conclusion
Here is the minimal setup to get this working reliably in production:
1. Eliminate the CPU copy path entirely. The `CVMetalTextureCacheRef` → `MTLTexture` → `MLFeatureValue` → CoreML path is non-negotiable on A14 and earlier — the 8–12ms copy tax is not recoverable at frame budget.
2. Allocate `MLState` once, poll `os_proc_available_memory()` per frame, and reinitialize state before iOS terminates your session.
3. Treat silent inference degradation as both a UX problem and a retention problem. Instrument confidence, expose session health state to the interface layer, and design explicit recovery paths.
The docs do not mention the silent session termination behavior. That one comes from production crash logs only.
Top comments (0)