DEV Community: NagaYu

Seven ways Core ML's state APIs crash a stateful VLM

NagaYu — Mon, 27 Jul 2026 08:58:01 +0000

Environment for everything below: macOS 26.5 (Darwin 25.5.0), Apple Silicon (arm64), coremltools 9.0, Python 3.13. Every crash described here reproduces deterministically with the scripts in the repo. A later OS may fix some of these — or quietly change the numerics; I measured both happening. So read this as a dated field report, not eternal truth.

I converted SmolVLM-256M — a small multimodal vision-language model — to Core ML and wrote a zero-dependency Swift CLI that captions an image in about 1.3 seconds, entirely on device: no Python, no third-party packages at runtime, and preprocessing that is bit-exact with the HuggingFace reference (down to a libjpeg-compatible JPEG decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified against golden data on every run).

Converting the model was the easy part. It always is. The part worth writing down is what it took to make generation run reliably — because almost none of it is documented anywhere, and most of it is the Core ML runtime crashing with SIGSEGV in ways the API gives you no reason to expect.

This is that catalog, plus the architecture I ended up with that sidesteps all of it.

The obvious design is a trap

Autoregressive decoding wants a KV cache: you run the prompt once ("prefill"), keep every layer's attention keys and values, then generate tokens one at a time ("decode"), reusing that cache so each step is cheap. On Core ML the sanctioned way to carry a cache between calls is MLState — stateful models, make_state(), and (in the WWDC-era examples) a multifunction package where a prefill function and a decode function share the same state.

That shared-state prefill/decode pattern is the design everyone reaches for first. On this stack, in every form I tried it, it crashes the process. Here's what I found.

Seven ways it broke

1–3: anything that moves state between models corrupts memory.

Writing state buffers from Python with MLState.write_state succeeds and round-trips cleanly through read_state — and then a decode loop using that state dies with SIGSEGV roughly 30–40 predict calls later. The delay is the tell: it looks like corruption of the state's backing store, not a caller-side lifetime bug. I reproduced it with both raw fp16 arrays and contiguous float32 copies while keeping the source buffers alive, so it isn't Python garbage-collecting the buffers out from under me.
Passing one model's state into a different model's predict — two separately loaded models declaring byte-identical state names, shapes, and dtypes — crashes immediately. If cross-model state simply isn't supported, that should be a catchable error, not a process kill.
A single multifunction .mlpackage where prefill and decode declare the same states — i.e. the documented KV-sharing pattern — gets one successful cross-function predict, then SIGSEGVs inside the decode loop, under every compute-unit combination I tested (including CPU_ONLY × CPU_ONLY). Loading that package with MLModel(path, function_name:) also fails outright with "functionName must be nil unless the model type is ML Program" — even though both functions are ML Programs. Only CompiledMLModel accepts the function name at all.

The throughline: the moment state has to cross a boundary — between processes' expectations, between models, between functions — the runtime stops giving you errors and starts giving you memory corruption.

4: flexible shapes and state don't mix. Combine a RangeDim or EnumeratedShapes input with states= and the runtime crashes (SIGSEGV/SIGTRAP) on the first stateful predict. Fully static shapes work. So a stateful graph has to be completely static — which quietly removes dynamic sequence lengths from the table.

5: INT8 symmetric quantization fails on the GPU. linear_quantize_weights with mode="linear_symmetric", int8, per-tensor or per-channel, produces a model that crashes at GPU load with MPSGraphExecutable.mm: failed assertion 'Error: MLIR pass manager failed'. The asymmetric path (mode="linear") and the per-block variants compile fine. So the bug is specific to how the symmetric int8 constexpr_affine_dequantize lowers — and the fix is just to use asymmetric per-channel and move on.

6: the ANE is not an option for this graph — in three different ways.

A fully static S=128 stateful prefill graph fails ANE compilation (ANECCompile() FAILED, an E5RT STL exception) under CPU_AND_NE/ALL, while CPU and GPU compile it fine.
A decode graph mixing large fp16 inputs ([1,90,1024,64]), value-dependent masks, and state crashes with SIGSEGV — not a graceful -14 — when loaded with ALL. Worse, that load happens inside ct.convert's post-conversion validation, so it takes the converting process down with it.
Compile the same decode graph for CPU_ONLY and it loads — but produces numerically wrong output; caption quality just collapses, while the GPU output is correct.

Net effect: for this pipeline, .cpuAndGPU isn't a preference, it's the only compute-unit setting that both loads and is correct. That's a measured conclusion, not a default I liked the sound of.

7 (informational): greedy output drifts across OS sessions. With bit-identical inputs and artifacts — verified at every stage — greedy decode produced different captions across OS sessions days apart. Within a session it's fully deterministic (I checked across processes). The likely cause is last-ulp changes in GPU kernels between system updates. Probably not a bug. But it means: never use a day-old caption string as a regression baseline. Re-derive goldens against the current OS.

The architecture is the workaround

The fix for all of 1–4 isn't a flag. It's refusing to ever cross a state-transfer API.

Prefill is stateless. It returns all 30 layers' K/V as ordinary model outputs — plain [1, 90, P, 64] tensors, not hidden state.
Decode takes them back as ordinary inputs, one step at a time, and keeps MLState only for the tokens it generates. State never leaves the model that owns it.

No write_state, no cross-model state, no shared-function state — none of the three landmines are ever stepped on. And the payoff showed up when I ported the whole thing to Swift: the identical design ran without a single crash. The Swift Core ML pipeline produces captions that match the Python Core ML pipeline character for character, deterministically, across repeated runs — and the non-split path matches HuggingFace transformers greedy output token-for-token too.

The lesson generalizes past this one model: on today's Core ML, treat MLState as something you write once, inside one model, for one purpose. Anything fancier is a memory-corruption bug waiting on a timer.

The constraints that are measured, not chosen

A few more things that only surface once it runs:

computeUnits is pinned to .cpuAndGPU — .cpuOnly breaks decode numerically, .all (ANE) crashes at model load. (See #6.)
INT8 costs ~12 s of GPU shader compilation per process — Core ML doesn't cache it across launches. You get roughly half the model size for a cold-start tax on every run, so the CLI defaults to fp16 and only uses INT8 for long-lived processes.
Preprocessing parity is achievable but unforgiving. Getting the Swift path bit-exact meant matching Pillow's LANCZOS down to its fixed-point arithmetic (PRECISION_BITS = 22, two-pass separable convolution, identical rounding) and libjpeg's baseline decode to 0/255. A --check mode re-proves all of it on every run — tokenizer IDs (81/81 and 876/876), JPEG decode vs. raw PIL pixels, resampler vs. Pillow, normalized tensors to fp16 — so a regression can't hide.
One honest gap: on the high-resolution split path, the last few tokens can differ from a HuggingFace run whose vision tower is torch fp32. That's inherent to running vision in Core ML fp16, not a porting bug — the Python Core ML pipeline diverges from torch-fp32 in exactly the same place.

I verified Apple's Private Cloud Compute from my own device export

NagaYu — Mon, 27 Jul 2026 08:55:40 +0000

Apple's pitch for Private Cloud Compute (PCC) is unusually strong for a cloud service: every production software build that can handle your Apple Intelligence requests is recorded in a public, append-only, cryptographically tamper-evident transparency log, and your device is supposed to refuse to send data to any build that isn't in it. Apple also lets you export a record of your own requests — Settings → Privacy & Security → Apple Intelligence Report → Export Activity.

Two halves of a promise. The catch is that, until now, nothing connected them for a normal person. "PCC is verifiable" was a claim you couldn't actually act on unless you stood up Apple's full Virtual Research Environment — macOS-bound, and gated behind a restrictive source license. The third-party tools that existed were report viewers: they pretty-print the export, but they don't check any of the cryptography.

So I wrote pcc-verify: an open-source, pip-installable tool that takes your exported report and checks each cloud request's attestation against Apple's live transparency log — the same researcher endpoints Apple points auditors at, reachable from an ordinary machine with no Apple entitlements. This post is about what that check actually proves, and — just as importantly for anything wearing the word "verify" — where it stops.

What's actually in the export

The interesting discovery is that the export is genuinely cross-referenceable. For each request that went to the cloud, the report carries, per PCC node, an attestation bundle containing two things that matter:

the node's software measurements — the AP boot ticket plus the per-slot cryptex image measurements (a "sealed hash ledger"), i.e. a fingerprint of exactly which OS and components were running; and
a full transparency-log inclusion proof — the leaf bytes, the Merkle path to the root, and a signed log head.

In other words, the device didn't just record "I talked to a cloud node." It recorded a fingerprint of that node's software and a cryptographic proof that the fingerprint's release sits in Apple's public log. That pairing is the whole reason a post-hoc verifier can exist. (Apple documents this deliberately, so researchers can inspect the attestations returned to their devices.)

What pcc-verify checks

For every cloud request, for each node that carries a non-empty bundle, the tool runs five checks:

Inclusion proof. Recompute the Merkle root from the leaf and the sibling hashes along its path (the log uses the RFC 9162 hashing scheme — SHA-256(0x00 ‖ leaf) for leaves, SHA-256(0x01 ‖ left ‖ right) for interior nodes), and require it to equal the root in the signed log head the device trusted.
Signature. Verify that log head's ECDSA P-256 / SHA-256 signature over the raw log-head bytes, against Apple's published log key — fetched live from the researcher endpoint, and cross-checked so the key's own hash matches the key identifier embedded in the bundle.
Measurement binding — the crux. Recompute the release digest from the bundle's own tickets, following Apple's documented canonicalization (null out the per-device personalized fields, drop the per-node signature and certificates, keep only the manifest body — because two nodes running identical software differ in exactly those dropped fields), and require that digest to equal the one committed in the log leaf. This is the step that makes the proof mean something: without it you've shown "some release is in the log," not "this node's actual software is the release in the log."
Freshness. Check the release leaf hadn't expired at the time of your request (in-use releases are periodically republished with new expiry, so the same digest legitimately recurs).
Log consistency (online). Fetch the current log head and confirm the log has only grown consistently since your device's view — an independent vantage point that feeds exactly the "split-view" detection the design is meant to enable.

Run it and you get per-request verdicts, not a single green light:

$ uvx pcc-verify Apple_Intelligence_Report.json
Requests routed to Private Cloud Compute: 12 (plus 34 handled on-device)
9 verified — attestation chains cryptographically to a release in Apple's public log
3 not verifiable — the export carries no attestation data for these entries

verified, proof-invalid (a real red flag — should never happen for genuine data), incomplete, not-verifiable, and on-device are all distinct. Requests handled on-device are reported as exactly that: nothing left the device, so there's nothing to check against a cloud log — the tool says so rather than implying a pass. Empty bundles are reported not-verifiable, never silently waved through. Honesty is the point of the exercise, so it's the point of the output.

What a green check does not mean

This is the part I care most about getting right, because a security tool that overstates is worse than none. A verified result proves one specific thing: the PCC software your device recorded talking to was a build Apple publicly committed to in its transparency log, and the proof binds to that build's actual measurements. Here is what it does not establish:

The report is unsigned JSON, produced by the very device stack being examined. pcc-verify checks Apple's log commitment against what the device recorded; it cannot prove the device recorded truthfully or completely. Verifying the device stack itself is what Apple's VRE and security bounty exist for.
Runtime properties leave no trace in the export. Whether the OHTTP relay actually concealed your IP, whether your data was statelessly deleted after processing, how encryption keys were released — none of that is in the file, so no post-hoc tool can speak to it. Those remain Apple claims, checkable only through source review and the research environment.
Full SEP-attestation semantic validation is out of scope in this version. Pinning the node's certificate chain to Apple's data-center CA and evaluating the Secure Enclave register policy isn't done yet — the fields are surfaced, not judged. That's stated plainly rather than blurred into the pass.

So the honest one-line version is: this narrows "trust Apple's word" down to "trust that your own device reported honestly," and makes everything after that point cryptographically checkable by anyone. That's a meaningful reduction in what you have to take on faith — and it's a bounded one. Both facts belong in the same sentence.

Two things worth flagging for the security-minded

It's a clean-room implementation. Apple's own security-pcc repository ships under an internal-use license that forbids redistribution and covers not just code but the data, fixtures, and .proto files. pcc-verify contains none of it — every protocol fact (field numbers, endpoints, algorithms, the canonicalization rules) is reimplemented from Apple's public PCC Security Guide and cited to it in the source; the test fixtures are synthesized to match the documented schema, not copied. Apple's pccvre proves the workflow is real and supported; what didn't exist was a portable, non-macOS, license-unencumbered verifier anyone can run. That's the niche.

It independently verifies something your own device skips. For requests routed through a Trusted Proxy, Apple's on-device daemon does not fully verify the proxied compute node's attestation (there's a comment to that effect in Apple's own client). When such bundles show up in the export, pcc-verify checks them anyway — and labels a pass on those as its own independent verification, not a re-confirmation of something the device already did. That's a genuine value-add, and it's marked as one rather than quietly folded into the same green check.

Why bother

The reason "verifiable" matters as a word is that it's supposed to mean you don't have to trust me — check it yourself. For PCC that promise was technically real but practically out of reach: the tooling assumed you were an Apple-entitled researcher on a Mac. Collapsing that to uvx pcc-verify Apple_Intelligence_Report.json — no account, no entitlement, your report never leaving your machine (the only network calls are to Apple's own transparency-log endpoints, allowlisted in the code; --offline disables even those) — is the point. Transparency logs are only as good as the number of independent parties actually looking at them.

It's v0.1.0, and it will need maintenance: Apple already changed the bundle encoding once (JSON before OS 26.4, Base64 protobuf after), which is exactly why the parser versions its assumptions and cites the doc revision it implements. SEP semantic validation is the obvious next piece. But the core — inclusion proof, signature, and the measurement-binding digest — verifies fully offline today, against real exports.

Try it
sh
uvx pcc-verify Apple_Intelligence_Report.json # or: pip install pcc-verify

To get a report: Settings (iOS/iPadOS) or System Settings (macOS) → Privacy & Security → Apple Intelligence Report → set the duration to 7 days (the 15-minute default captures almost nothing) → use Apple Intelligence for a while → Export Activity. Flags worth knowing: --verbose for per-check detail, --json / --markdown for reports, --offline with --keys to pin Apple's log keys once and verify with no network at all, and a non-zero exit on proof-invalid so it can gate automation.

Source, and the full feasibility write-up documenting what the export contains, how the log is accessed, and every boundary above: github.com/NagaYu/pcc-verify. Every load-bearing claim in that write-up was verified firsthand — Apple's published fixtures parsed directly, the license read in full, the production transparency-log endpoints probed live from a non-entitled machine. Anything that couldn't be established is listed as such rather than assumed. Same standard I'd ask of anyone else's "verify" tool.