---
title: "Structured Output Streaming from On-Device LLMs: Enforcing JSON Schemas with Constrained Decoding"
published: true
description: "Wire grammar-based constrained decoding (GBNF in llama.cpp, EBNF in MLX-LM) to enforce valid JSON from quantized on-device models — covering token masking, incremental validation, and real backend overhead numbers."
tags: android, ios, mobile, architecture
canonical_url: https://mvpfactory.co/blog/structured-output-on-device-llm-constrained-decoding
---
## What We Are Building
By the end of this workshop, you will know how to enforce valid JSON schema output from a quantized on-device LLM — without relying on prompt engineering. We will cover how token masking works at the logit layer, how to layer incremental semantic validation on top of the structural guarantee, and what the performance cost actually looks like across NNAPI, ANE, and CPU backends.
This is the pattern I use in every on-device inference pipeline that needs structured output.
---
## Prerequisites
- Familiarity with llama.cpp (Android/JNI) or MLX-LM (iOS/Swift)
- A quantized model — Q4_K_M on a 7B-class architecture is a reasonable starting point
- Basic understanding of LLM sampling (logits → softmax → token)
---
## Step 1 — Understand What Token Masking Actually Does
Standard sampling picks the next token from a probability distribution over the full vocabulary. Constrained decoding intercepts this *before* sampling and zeroes out the logits of every token that would violate the current parse state of your grammar.
raw logits (vocab × 1)
↓
[Grammar State Machine] → valid token set
↓
masked logits (invalid tokens → -∞)
↓
softmax → sample
The grammar state machine advances *incrementally* as tokens are emitted. You are not validating the completed output — you are validating the prefix at every step. Miss this distinction and your mental model of how to debug failures will be wrong.
---
## Step 2 — Wire GBNF on Android via llama.cpp
In llama.cpp, grammar constraints are defined as GBNF (GGML BNF) rules, passed to `llama_grammar_init`, and applied by the sampler on every forward pass.
kotlin
// Android / llama.cpp via JNI — simplified
val grammar = LlamaGrammar.fromGBNF("""
root ::= object
object ::= "{" ws members ws "}"
members ::= member ("," ws member)*
member ::= string ws ":" ws value
value ::= string | number | object | array | "true" | "false" | "null"
""")
llamaContext.setSamplerGrammar(grammar)
// Token stream now structurally guaranteed to match the grammar
The token stream is now structurally guaranteed to match the grammar. Nothing about this enforces semantic correctness yet — that comes in Step 3.
---
## Step 3 — Layer Incremental Schema Validation on iOS
In MLX-LM on Apple platforms, the same concept applies via EBNF-style constraint objects passed to the generation loop. Grammar enforcement guarantees syntax. A lightweight incremental validator enforces semantics alongside the stream.
swift
// iOS / MLX-LM — incremental validation sketch
var partialBuffer = ""
for await token in mlxSession.generateStream(grammar: jsonGrammar) {
partialBuffer += token
if let completed = partialBuffer.lastCompletedJSONValue() {
try schemaValidator.validate(completed, against: actionItemSchema)
}
yield token
}
If semantic validation fails mid-stream, you have two recovery paths: abort and retry with a tighter grammar, or surface the partial output with an error flag. In production, abort-and-retry at the token level is prohibitively expensive — design your grammar to encode value constraints where possible from the start.
---
## Step 4 — Benchmark Your Backend Before Committing to a UX Contract
Grammar enforcement is essentially free on CPU. The masking operation is O(vocab_size), cheap relative to the transformer forward pass. On accelerated backends, the picture changes significantly.
| Backend | Observed Overhead | Root Cause |
|---|---|---|
| CPU (ARM NEON) | ~1–3% token latency | Masking runs on-thread, minimal impact |
| NNAPI (Android) | ~8–15% token latency | GPU/DSP sync required per token; masks applied CPU-side |
| ANE (Apple Neural Engine) | ~10–20% token latency | ANE handles matrix ops; logit masking pulled back to CPU |
| Metal (iOS GPU) | ~3–8% token latency | Logit tensor more accessible; masking more efficient than ANE path |
> Numbers from internal testing on a Pixel 8 Pro (NNAPI), iPhone 15 Pro (ANE/Metal), and Snapdragon 8 Gen 2 device (CPU), using a Q4_K_M quantized 7B-class model with a 32K vocabulary. Latency measured as per-token wall-clock time averaged over 200-token sequences.
The overhead on NNAPI and ANE comes from a fundamental architectural mismatch: the accelerator handles matrix multiplications, but logit masking must happen CPU-side, requiring a device-to-host transfer of the logit tensor on every token. For a 32K vocabulary model streaming 40 tokens per second, that is 40 round-trips per second between accelerator memory and CPU. Budget for this before you design your streaming UX.
---
## Gotchas
**Hand-authoring GBNF grammars will drift from your schema.** This is the mistake that hurts most teams in production. Your schema evolves, your grammar does not, and nobody notices until users report malformed output. The docs do not mention this strongly enough, but you should treat grammar generation as a compile-time step, not a one-time handoff. The most mature option today is [lm-format-enforcer](https://github.com/noamgat/lm-format-enforcer), which generates token masks directly from Pydantic models or JSON Schema objects and supports both llama.cpp and Hugging Face backends.
**Skipping the semantic validation layer.** A token sequence can be valid GBNF and still produce `{"action_items": 42}` when you expected an array. Grammar enforces structure; you still need a validator to enforce meaning.
**Underestimating accelerator-to-CPU transfer overhead.** Teams target ANE or NNAPI for raw inference speed and are surprised when constrained decoding cuts into their latency budget. Benchmark with grammar enforcement active — the per-token overhead is real and backend-dependent. (On that note: if you are building inference tooling for long sessions, [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) is a useful reminder to actually step away from the machine while your benchmarks run.)
---
## Conclusion
Here is the minimal checklist to get this right in production:
1. Match your constraint enforcement to your backend — ANE and NNAPI carry a 10–20% logit-masking overhead. Benchmark before you commit to a UX contract.
2. Layer GBNF/EBNF grammar constraints with incremental schema validation. Syntax and semantics are separate concerns.
3. Automate grammar generation from your JSON Schema now. Discovering grammar drift in a shipped product is a bad day.
The ecosystem is converging on first-class JSON Schema support in the sampler pipeline — several open llama.cpp PRs are already pushing in that direction. Wiring up the transpiler step once costs far less than auditing drift across a production release.
**Further reading:**
- [llama.cpp GBNF grammar docs](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md)
- [lm-format-enforcer](https://github.com/noamgat/lm-format-enforcer)
- [MLX-LM structured generation](https://github.com/ml-explore/mlx-lm)
Top comments (0)