A 2.6B model on an iPhone that always answers in valid JSON

#ios #machinelearning #llm #swift

Small language models are good at one thing that apps need all the time: read this text and fill these fields. A letter becomes sender, deadline and next steps; a voice note becomes a task list; a receipt becomes an expense entry. This post shows how we run that fully on an iPhone with llama.cpp and LFM2-2.6B, what it costs in seconds, and the five problems we had to solve on the way.

Everything here runs offline on an iPhone 15 (A16, 6 GB RAM). That phone has no Apple Intelligence, which is exactly the case a local model is for.

The setup

Model: LiquidAI/LFM2-2.6B-GGUF, Q4_K_M, 1.6 GB
Runtime: llama.cpp b11887 as an xcframework, Metal for all layers, n_ctx = 4096
Sampling: greedy, constrained by a GBNF grammar
Inputs: Apple Vision for OCR, Apple's on-device speech recognition for audio

Measured on the phone:

Task	Time
First model load	~20 s (then it stays in memory)
Short text → 6-field JSON	4–6 s
Voice note → nested task list	7–9 s
Scanned bill (~1,500 chars) → 7 fields	10–14 s
OCR of a page (Vision)	~1 s

1. Make invalid JSON impossible

Asking a 2B model to "answer in JSON" works most of the time. Every few hundred calls a bracket goes missing or a key is invented, and your parser throws. A GBNF grammar fixes this at the sampling step: tokens that would break the shape are never allowed. Writing grammars by hand is tedious, so we describe the shape in Swift and generate it:

let schema = Schema("Letter") {
    Property("sender", .string, "Who sent it, short")
    Property("action", .choice(["none", "pay", "reply", "sign"]))
    Property("deadline", .date, "Only if the letter names one")
    Property("steps", .list(.string, max: 4))
}
let grammar = schema.gbnf   // pass to llama_sampler_init_grammar(vocab, grammar, "root")

Keys come out in the declared order, which also helps the model: put the fields it should "think about" first and the conclusion last. The generator is MIT licensed: swift-gbnf.

2. Long prompts abort the app

Short tests worked, a real letter killed the app with GGML_ASSERT(n_tokens_all <= cparams.n_batch). The default batch is 512 tokens and we decoded the whole prompt in one call. Setting n_batch = n_ctx (and leaving n_ubatch at 512) fixed it without noticeable memory cost. Also check prompt tokens + max new tokens <= n_ctx before decoding, and fail gracefully if it doesn't fit.

3. Don't let the model do numbers

LFM2 reads a bill well, but on a Japanese electricity bill it picked a line item (8,120 yen) instead of the total (10,056 yen). We stopped asking the model for numbers we can read with rules: the amount next to Total / Summe / ご請求金額 / 合計, dates, IBANs. The model writes the summary and the steps, the rules provide the facts, and the facts override the model.

4. Give it a calendar

"Your check-up is on Tuesday" turned into random dates. Small models can't count weekdays. Adding one line to the prompt fixed it almost completely:

Today is Friday 2026-10-30. Next days: Saturday 2026-10-31, Sunday 2026-11-01, Monday 2026-11-02, Tuesday 2026-11-03, ...

5. Language: say it twice

Given a German receipt and the instruction "answer in English" in the system prompt, the model often answered in German. Repeating the wish right after the input text, (Write every text value in English.), made it reliable. Reading a Japanese letter and explaining it in English works well once that's in place.

Bonus: the Simulator

The official llama.cpp xcframework has no iOS Simulator slice, so apps that link it don't build for the Simulator. We built the same release with llama.cpp's own build-xcframework.sh and published it unchanged: llama-xcframework. In the Simulator, load with n_gpu_layers = 0: its Metal lacks most kernels and is slower than the CPU.

Takeaway

A 2.6B model on a phone is a very good reader: summaries, classification, extraction, in many languages, private by design. Pair it with a grammar for the shape and plain rules for the numbers, and it behaves like a reliable component instead of a demo.

Screen recordings of all of this on the iPhone: theyplay.games/ondevice. We packaged our full setup (Apple Intelligence first, the local model as fallback, background model download, OCR and speech helpers) as a Swift package there, but everything in this post works on its own.

DEV Community