Environment for everything below: macOS 26.5 (Darwin 25.5.0), Apple Silicon (arm64), coremltools 9.0, Python 3.13. Every crash described here reproduces deterministically with the scripts in the repo. A later OS may fix some of these — or quietly change the numerics; I measured both happening. So read this as a dated field report, not eternal truth.
I converted SmolVLM-256M — a small multimodal vision-language model — to Core ML and wrote a zero-dependency Swift CLI that captions an image in about 1.3 seconds, entirely on device: no Python, no third-party packages at runtime, and preprocessing that is bit-exact with the HuggingFace reference (down to a libjpeg-compatible JPEG decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified against golden data on every run).
Converting the model was the easy part. It always is. The part worth writing down is what it took to make generation run reliably — because almost none of it is documented anywhere, and most of it is the Core ML runtime crashing with SIGSEGV in ways the API gives you no reason to expect.
This is that catalog, plus the architecture I ended up with that sidesteps all of it.
The obvious design is a trap
Autoregressive decoding wants a KV cache: you run the prompt once ("prefill"), keep every layer's attention keys and values, then generate tokens one at a time ("decode"), reusing that cache so each step is cheap. On Core ML the sanctioned way to carry a cache between calls is MLState — stateful models, make_state(), and (in the WWDC-era examples) a multifunction package where a prefill function and a decode function share the same state.
That shared-state prefill/decode pattern is the design everyone reaches for first. On this stack, in every form I tried it, it crashes the process. Here's what I found.
Seven ways it broke
1–3: anything that moves state between models corrupts memory.
Writing state buffers from Python with MLState.write_state succeeds and round-trips cleanly through read_state — and then a decode loop using that state dies with SIGSEGV roughly 30–40 predict calls later. The delay is the tell: it looks like corruption of the state's backing store, not a caller-side lifetime bug. I reproduced it with both raw fp16 arrays and contiguous float32 copies while keeping the source buffers alive, so it isn't Python garbage-collecting the buffers out from under me.
Passing one model's state into a different model's predict — two separately loaded models declaring byte-identical state names, shapes, and dtypes — crashes immediately. If cross-model state simply isn't supported, that should be a catchable error, not a process kill.
A single multifunction .mlpackage where prefill and decode declare the same states — i.e. the documented KV-sharing pattern — gets one successful cross-function predict, then SIGSEGVs inside the decode loop, under every compute-unit combination I tested (including CPU_ONLY × CPU_ONLY). Loading that package with MLModel(path, function_name:) also fails outright with "functionName must be nil unless the model type is ML Program" — even though both functions are ML Programs. Only CompiledMLModel accepts the function name at all.
The throughline: the moment state has to cross a boundary — between processes' expectations, between models, between functions — the runtime stops giving you errors and starts giving you memory corruption.
4: flexible shapes and state don't mix. Combine a RangeDim or EnumeratedShapes input with states= and the runtime crashes (SIGSEGV/SIGTRAP) on the first stateful predict. Fully static shapes work. So a stateful graph has to be completely static — which quietly removes dynamic sequence lengths from the table.
5: INT8 symmetric quantization fails on the GPU. linear_quantize_weights with mode="linear_symmetric", int8, per-tensor or per-channel, produces a model that crashes at GPU load with MPSGraphExecutable.mm: failed assertion 'Error: MLIR pass manager failed'. The asymmetric path (mode="linear") and the per-block variants compile fine. So the bug is specific to how the symmetric int8 constexpr_affine_dequantize lowers — and the fix is just to use asymmetric per-channel and move on.
6: the ANE is not an option for this graph — in three different ways.
A fully static S=128 stateful prefill graph fails ANE compilation (ANECCompile() FAILED, an E5RT STL exception) under CPU_AND_NE/ALL, while CPU and GPU compile it fine.
A decode graph mixing large fp16 inputs ([1,90,1024,64]), value-dependent masks, and state crashes with SIGSEGV — not a graceful -14 — when loaded with ALL. Worse, that load happens inside ct.convert's post-conversion validation, so it takes the converting process down with it.
Compile the same decode graph for CPU_ONLY and it loads — but produces numerically wrong output; caption quality just collapses, while the GPU output is correct.
Net effect: for this pipeline, .cpuAndGPU isn't a preference, it's the only compute-unit setting that both loads and is correct. That's a measured conclusion, not a default I liked the sound of.
7 (informational): greedy output drifts across OS sessions. With bit-identical inputs and artifacts — verified at every stage — greedy decode produced different captions across OS sessions days apart. Within a session it's fully deterministic (I checked across processes). The likely cause is last-ulp changes in GPU kernels between system updates. Probably not a bug. But it means: never use a day-old caption string as a regression baseline. Re-derive goldens against the current OS.
The architecture is the workaround
The fix for all of 1–4 isn't a flag. It's refusing to ever cross a state-transfer API.
Prefill is stateless. It returns all 30 layers' K/V as ordinary model outputs — plain [1, 90, P, 64] tensors, not hidden state.
Decode takes them back as ordinary inputs, one step at a time, and keeps MLState only for the tokens it generates. State never leaves the model that owns it.
No write_state, no cross-model state, no shared-function state — none of the three landmines are ever stepped on. And the payoff showed up when I ported the whole thing to Swift: the identical design ran without a single crash. The Swift Core ML pipeline produces captions that match the Python Core ML pipeline character for character, deterministically, across repeated runs — and the non-split path matches HuggingFace transformers greedy output token-for-token too.
The lesson generalizes past this one model: on today's Core ML, treat MLState as something you write once, inside one model, for one purpose. Anything fancier is a memory-corruption bug waiting on a timer.
The constraints that are measured, not chosen
A few more things that only surface once it runs:
computeUnits is pinned to .cpuAndGPU — .cpuOnly breaks decode numerically, .all (ANE) crashes at model load. (See #6.)
INT8 costs ~12 s of GPU shader compilation per process — Core ML doesn't cache it across launches. You get roughly half the model size for a cold-start tax on every run, so the CLI defaults to fp16 and only uses INT8 for long-lived processes.
Preprocessing parity is achievable but unforgiving. Getting the Swift path bit-exact meant matching Pillow's LANCZOS down to its fixed-point arithmetic (PRECISION_BITS = 22, two-pass separable convolution, identical rounding) and libjpeg's baseline decode to 0/255. A --check mode re-proves all of it on every run — tokenizer IDs (81/81 and 876/876), JPEG decode vs. raw PIL pixels, resampler vs. Pillow, normalized tensors to fp16 — so a regression can't hide.
One honest gap: on the high-resolution split path, the last few tokens can differ from a HuggingFace run whose vision tower is torch fp32. That's inherent to running vision in Core ML fp16, not a porting bug — the Python Core ML pipeline diverges from torch-fp32 in exactly the same place.
Top comments (0)