Structured Output Grammars for On-Device LLMs

---
title: "Structured Output Grammars for On-Device LLMs on Android"
published: true
description: "A hands-on guide to using GBNF grammars in llama.cpp to guarantee valid JSON from on-device LLMs on Android — no retry loops, no post-processing."
tags: kotlin, android, architecture, mobile
canonical_url: https://blog.mvpfactory.co/structured-output-grammars-on-device-llms-android
---

## What We Will Build

By the end of this tutorial, you will have a working setup where your on-device LLM on Android produces **structurally valid JSON every single time** — not 7 out of 10, not 9 out of 10, but 10 out of 10. We will write a custom GBNF grammar, wire it into llama.cpp through JNI, and watch grammar-guided sampling eliminate the retry loops you have been writing around malformed model output.

Let me show you a pattern I use in every project that runs inference on-device.

## Prerequisites

- Android project with llama.cpp integrated via its JNI bridge
- A quantized GGUF model (Q4_K_M or similar) deployed to device
- Familiarity with Kotlin and basic BNF notation
- A Snapdragon 8 Gen 3 (or comparable) device for realistic benchmarks

## Step 1: Understand Why This Is a Sampling Problem

Here is the gotcha that will save you hours: most teams treat malformed LLM output as a post-processing problem. They wrap inference in a try-parse-retry loop. On mobile, where every millijoule counts, that is the wrong approach entirely.

The fix belongs in the **decoder**. GBNF (GGML BNF) is a grammar format supported natively by llama.cpp. At each token generation step, the sampler checks which tokens are valid continuations given the current grammar state. Invalid tokens get their logits masked to negative infinity before softmax. The model literally cannot produce an invalid sequence.

The pipeline:

Logits → Grammar Mask → Temperature → Top-K/Top-P → Token Selection

This is not regex validation after the fact. The grammar creates a finite-state automaton that walks forward with each generated token, pruning the search space in real time.

## Step 2: Write a Custom GBNF Grammar

Suppose your API expects this schema from the model:

json
{"intent": "string", "confidence": 0.0, "entities": [{"name": "string", "type": "string"}]}

Here is the minimal setup to get this working — a GBNF grammar that locks every field name as a literal:

bnf
root ::= "{" ws "\"intent\":" ws string "," ws "\"confidence\":" ws number "," ws "\"entities\":" ws entities ws "}"
string ::= "\"" [a-zA-Z0-9_ ]+ "\""
number ::= "0" "." [0-9]+
entities ::= "[" ws (entity ("," ws entity))? ws "]"
entity ::= "{" ws "\"name\":" ws string "," ws "\"type\":" ws string ws "}"
ws ::= [ \t\n]

The model has zero freedom to hallucinate keys like `"conf"` or `"entity_list"`. It fills in values; the grammar enforces structure. Write grammars that match your **exact schema**, not generic JSON. A generic `json.gbnf` guarantees valid JSON but not valid *responses*. Schema-specific grammars give you both.

## Step 3: Integrate with Kotlin via JNI

The llama.cpp Android example exposes grammar support through its JNI bridge. Pass the grammar string when you configure the sampler:

kotlin
external fun setupGrammarSampler(
contextPtr: Long,
grammarString: String
): Long

// In your inference wrapper
val grammar = assets.open("schema.gbnf").bufferedReader().readText()
val samplerPtr = setupGrammarSampler(ctxPtr, grammar)

On the C++ side, the grammar is parsed once into a `llama_grammar` instance and reused across tokens within a single generation. The per-token cost is the automaton state advance and logit masking — both O(V) where V is vocabulary size. On 32K-vocab models, that adds roughly 0.2 ms per step.

## Step 4: Measure the Real-World Tradeoff

Here are representative numbers from a Q4_K_M quantized 7B model running on a Snapdragon 8 Gen 3:

| Metric | Unconstrained | Grammar-guided | Delta |
|---|---|---|---|
| Tokens/sec (decode) | 12.4 t/s | 10.8 t/s | -12.9% |
| Time-to-first-token | 280 ms | 295 ms | +5.4% |
| Valid JSON rate | ~72% | 100% | +28pp |
| Avg retries needed | 0.4 | 0 | -100% |
| Effective latency (incl. retries) | 1,480 ms | 1,120 ms | -24.3% |

You pay roughly 13% on raw decode speed, but you eliminate retries entirely. Net effective latency drops by nearly a quarter. On battery-constrained devices, avoiding redundant inference passes matters even more than the raw throughput number suggests.

## Gotchas

**Token boundary misalignment on quantized models.** The docs do not mention this, but it will bite you in production. Consider generating `"confidence": 0.85`. A BPE tokenizer might encode `0.85` as `["0", ".", "8", "5"]` or as `["0", ".85"]` depending on the merge table. Aggressive quantization (Q2_K, Q3_K_S) shifts probability mass in ways that interact poorly with grammar masking.

What this looks like in practice:

- Numeric values truncated at unusual boundaries (`0.` followed by EOS)
- Strings ending mid-token because no valid continuation exists in the grammar
- Repeated whitespace tokens when the grammar allows `ws` as a fallback

**The fix: defensive grammar design.** For numeric fields, allow broader patterns than your schema strictly requires, then validate semantically in Kotlin after parsing:

bnf
number ::= "-"? [0-9]+ ("." [0-9]+)? ([eE] [+-]? [0-9]+)?

Let the grammar guarantee structure. Your application layer handles meaning. Keep those responsibilities separate and you will save yourself a lot of debugging.

**Do not use generic JSON grammars.** A generic grammar guarantees valid JSON but not valid responses. If the model hallucinates a key name that is syntactically valid JSON, a generic grammar will happily allow it. Lock your field names as literals.

## Wrapping Up

Move validation into the decoder. The 10-15% decode overhead pays for itself by removing retry loops, and effective latency drops 20%+ on real workloads. Write schema-specific GBNF grammars with literal field names. Design those grammars defensively for quantized models — permissive value patterns, strict structural rules. The grammar handles structure; Kotlin handles semantics.

That separation of concerns is what makes this pattern production-ready on mobile.