---
title: "Streaming Core ML LLM Tokens to SwiftUI with Swift 6 Actors"
published: true
description: "Stream Core ML LLM tokens token-by-token into SwiftUI using Swift 6 actors, back-pressured AsyncStream, and ANE dispatch tuning to sustain 15+ tok/s on device without memory termination."
tags: ios, swift, architecture, mobile
canonical_url: https://mvpfactory.co/blog/streaming-core-ml-llm-tokens-swiftui-actors
---
Running a local LLM on-device is not the hard part. Streaming its output token-by-token into a responsive SwiftUI view without blocking the main thread, blowing the memory budget, or stalling on prefill — that is where most implementations fall apart.
Let me show you a pattern I use in every production on-device inference project.
## What We Are Building
A four-layer pipeline: Core ML model → Swift 6 actor → back-pressured `AsyncStream` → `@MainActor` SwiftUI view. Each layer has a strict ownership contract. Violate any boundary and you get either a data race under Swift 6's strict concurrency checker, or a dropped frame.
CoreMLEngine (Actor)
└── AsyncStream
└── TokenStreamViewModel (@MainActor)
└── SwiftUI Text view
## Prerequisites
- Xcode 16+, Swift 6 strict concurrency enabled
- A Core ML compiled LLM that exposes `input_ids` and `past_key_values` as input/output pairs
- Familiarity with `async`/`await` and `@MainActor`
---
## Step 1 — The Actor That Owns Inference State
The actor boundary is load-bearing in Swift 6. `MLModel`, `inputArray`, and `kvCache` all live here and nowhere else.
swift
actor CoreMLEngine {
private let model: MLModel
private var inputArray: MLMultiArray
private var kvCache: MLMultiArray
init(model: MLModel) throws {
self.model = model
self.inputArray = try MLMultiArray(shape: [1, 1], dataType: .int32)
self.kvCache = try MLMultiArray(shape: kvCacheShape, dataType: .float16)
}
func generateTokens(prompt: String) -> AsyncStream<String> {
AsyncStream(bufferingPolicy: .bufferingNewest(16)) { continuation in
Task {
var tokenIds = tokenize(prompt)
while !shouldStop(tokenIds) {
guard let next = runSingleStep(context: tokenIds) else { break }
tokenIds.append(next.id)
continuation.yield(next.text)
}
continuation.finish()
}
}
}
private func runSingleStep(context: [Int]) -> (id: Int, text: String)? {
inputArray[0] = context.last.map(NSNumber.init) ?? 0
let features = try? MLDictionaryFeatureProvider(dictionary: [
"input_ids": MLFeatureValue(multiArray: inputArray),
"past_key_values": MLFeatureValue(multiArray: kvCache)
])
guard let features,
let prediction = try? model.prediction(from: features),
let logits = prediction.featureValue(for: "logits")?.multiArrayValue,
let updatedCache = prediction.featureValue(for: "present_key_values")?.multiArrayValue
else { return nil }
kvCache = updatedCache
let id = argmax(logits)
return (id: id, text: detokenize(id))
}
}
## Step 2 — Back-Pressure With AsyncStream
`AsyncStream` does not apply back-pressure by default. On an A17 Pro, the ANE can outrun SwiftUI's text layout engine by 3–4x during burst decode. Setting `.bufferingNewest(16)` caps unbounded queue growth — sixteen tokens is enough headroom for rendering jitter without letting the queue balloon.
## Step 3 — SwiftUI Without Main-Thread Blocking
swift
@MainActor
class TokenStreamViewModel: ObservableObject {
@Published var output = ""
func stream(from engine: CoreMLEngine, prompt: String) async {
for await token in await engine.generateTokens(prompt: prompt) {
output += token
}
}
}
Each `await` in the `for await` loop is a suspension point. SwiftUI's render loop gets CPU time between tokens. No `DispatchQueue.main.async` wrangling required.
## Step 4 — Memory Pressure Callbacks
swift
nonisolated func observeMemoryPressure() {
NotificationCenter.default.addObserver(
forName: UIApplication.didReceiveMemoryWarningNotification,
object: nil,
queue: nil
) { [weak self] _ in
guard let self else { return }
Task { await self.flushKVCache() }
}
}
The `nonisolated` keyword lets this be called from any context. The closure re-enters actor isolation via `Task { await }` — Swift 6's strict concurrency checker correctly rejects calling actor-isolated methods directly from a non-isolated closure.
---
## ANE vs. CPU: The Dispatch Tradeoff
| Target | Throughput | Prefill Latency | Memory |
|---|---|---|---|
| ANE only | 15–22 tok/s | 180ms | Low |
| CPU only | 4–7 tok/s | 90ms | High |
| ANE + CPU | 12–18 tok/s | 120ms | Medium |
Compile with `computeUnits = .cpuAndNeuralEngine` as the default. For models above 2B parameters, force `.neuralEngine` explicitly for attention layers — the automatic scheduler makes conservative choices under memory pressure and leaves throughput on the table.
For a 1B INT4 quantized model: ~600MB weights, ~120MB KV cache at 512 context, ~40MB runtime overhead. That lands around 760MB peak — under the ~1.2GB soft jetsam limit on an iPhone 15. Profile with Instruments' Memory Graph, not Xcode's summary, which underreports ANE allocations.
---
## Gotchas
**Missing KV cache threading is the silent killer.** If you do not pass `past_key_values` back in on every decode step, the model recomputes full attention history from scratch each token. You will pass a 50-token smoke test and get a jetsam kill at 300 tokens. Every time.
swift
// Wrong: forces full-context attention recompute every step
let input = try! MLMultiArray(shape: [1, 1], dataType: .int32)
// no past_key_values passed — guaranteed OOM beyond ~128 tokens
// Correct: pre-allocated input, explicit cache threading (see actor init)
**Per-step `MLMultiArray` allocation adds up.** Across a 512-token run, that is 512 short-lived heap objects. Measurable allocator pressure that compounds under thermal throttling. Pre-allocate once in `init`, mutate in place.
**No `bufferingPolicy` means no back-pressure.** This masks latency issues in development and surfaces them under thermal conditions in production. Set `.bufferingNewest(16)` and move on.
The docs do not mention this, but the automatic `computeUnits` selection is intentionally conservative. Manually targeting `.neuralEngine` for attention layers can recover 30–40% throughput on supported hardware — worth profiling with Instruments' Core ML template before shipping.
---
## Wrapping Up
The full pipeline — actor-owned inference state, back-pressured `AsyncStream`, `@MainActor` view model — is the minimal setup that survives real thermal and memory conditions on device. Each layer earns its place.
Pre-allocate your `MLMultiArray`. Thread your KV cache. Cap your stream buffer. Profile ANE dispatch before you ship.
The architecture is strict by design. Swift 6's concurrency checker will surface every violation at compile time, which is exactly the right time to find them.
**Resources:**
- [Core ML documentation — MLModel prediction](https://developer.apple.com/documentation/coreml/mlmodel)
- [Swift concurrency — AsyncStream](https://developer.apple.com/documentation/swift/asyncstream)
- [Instruments — Core ML template](https://developer.apple.com/documentation/xcode/instruments)
---
*If you spend long hours building on-device inference pipelines, [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) keeps desk fatigue in check with break reminders and guided exercises — worth having running in the background during a long profiling session.*
Top comments (0)