How I use GBNF grammars to make small local models stop hallucinating JSON

Or: why your 8B model keeps breaking your agent's tool protocol, and what you can do about it at the token level.

Originally published on eris-system.dev/blog/gbnf-grammars

---

I've been building Eris, a local-first AI agent written in Rust. It sits on your machine, reads and writes an Obsidian-compatible Markdown vault, and calls tools (memory, reminders, web fetch, email, calendar) through a structured JSON protocol. The LLM backend is llama.cpp, the models are 8-26B GGUFs, and nothing leaves your machine unless you explicitly opt in.

The single hardest engineering problem in the whole project wasn't the orchestrator, the TUI, or the semantic memory layer. It was this:

Getting a small local model to reliably emit valid, schema-conformant JSON.

This post is about how I solved it with GBNF grammars (compiled per-session and narrowed per-turn) and why I think this approach generalises to anyone building tool-calling agents on local models.

The problem: small models are sloppy with JSON

If you've tried to build an agent on a local 8B or even 12B model, you know the drill. You tell the model to respond in JSON. It does, mostly. Then on turn 47 it:

• Forgets a closing brace
• Adds a trailing comma in an array
• Invents a tool name that doesn't exist (vault_read instead of vault:read)
• Wraps its JSON in a markdown code fence
• Emits valid JSON followed by a paragraph of conversational text
• Returns "args": "path=notes/today.md" instead of "args": {"path": "notes/today.md"}

Ollama's format: json helps a bit. It constrains output to some valid JSON, but it says nothing about which JSON. Your tool protocol schema isn't enforced. The model can return {"sure": "here is your file"} and that's perfectly valid JSON. It just isn't a valid tool call.

OpenAI and Anthropic solve this with server-side function-calling APIs. But we're running locally. We don't have that luxury. We have something better.

GBNF: constraining generation at the token level

llama.cpp supports GBNF grammars, a BNF-like format that constrains which tokens the sampler is allowed to emit. If a token would produce a string that can't be extended into a valid parse of the grammar, it gets masked out before sampling. The model literally cannot generate malformed output. Not 'usually doesn't'. Cannot.

This is a fundamentally different thing from prompt-engineering the model to 'please respond in JSON'. It's a hard constraint on the sampler, enforced at every single token position.

Eris compiles a GBNF grammar at session start that defines the exact shape of its protocol envelope. Here's the skeleton from src/engine/grammar/envelope.rs:

const STATIC_GRAMMAR: &str = r#"root ::= "{" ws thought-kv "," ws status-kv "," ws message-kv "," ws toolcalls-kv ws "}"

ws ::= [ \t\n]*

thought-kv ::= "\"thought\"" ws ":" ws json-string
status-kv  ::= "\"status\"" ws ":" ws status-enum
message-kv ::= "\"message_to_user\"" ws ":" ws (json-string | "null")
toolcalls-kv ::= "\"tool_calls\"" ws ":" ws "[" ws tool-call-list ws "]"

status-enum ::= "\"Task\"" | "\"Reflect\"" | "\"Idle\"" | "\"Process\""
"#;

Every LLM response must be a JSON object with exactly four keys: thought (chain-of-thought, always present), status (one of four enum values), message_to_user (string or null), and tool_calls (array of name+args objects). The model can't deviate. It can't add a fifth key. It can't misspell tool_calls. It can't emit a status value that doesn't exist in the enum.

The tool call shape itself gets appended depending on whether we're in static mode (any JSON object for args) or dynamic mode (per-tool typed args):

/// Per-tool dynamic-args version: tool-call dispatches into `tool-with-args`.
const TOOL_CALL_RULES_DYNAMIC: &str = r#"tool-call-list ::= "" | tool-call ("," ws tool-call)*
tool-call ::= "{" ws "\"name\"" ws ":" ws tool-with-args ws "}"
"#;

Notice tool-with-args here. That's where it gets interesting.

Compiling JSON Schema to GBNF rules

Tool arguments are where small models really struggle. A vault:read call needs {"relative_path": "some/file.md"}. A web:fetch call needs {"url": "https://..."}. Without constraints, the model will happily call vault:read with {"file": "notes.md"} (wrong key name, schema validation fails, recovery turn wasted).

Eris goes further: it compiles each tool's JSON Schema into GBNF rules at startup. The schemas come from Rust structs via schemars, so the types are the source of truth. Here's the compiler entry point in src/engine/grammar/schema_to_gbnf.rs:

pub fn schema_to_gbnf_rule(
    tool_name: &str,
    schema: &RootSchema,
) -> Option<(String, Vec<GbnfRule>)> {
    let rule_name = tool_name_to_rule_name(tool_name);
    let mut ctx = CompileCtx {
        definitions: &schema.definitions,
        extra_rules: Vec::new(),
        depth: 0,
    };

    let body = compile_schema_object(&schema.schema, &rule_name, &mut ctx)?;
    let mut rules = vec![(rule_name.clone(), body)];
    rules.extend(ctx.extra_rules);
    Some((rule_name, rules))
}

A GbnfRule is just a (String, String) tuple: rule name and rule body. The compiler walks the JSON Schema tree and emits GBNF productions for each type it finds. Objects become key-value sequences with required fields unconditional and optional fields wrapped in (...)?. Strings with enums become GBNF alternations. Arrays get a helper list rule. $ref pointers get resolved against the definitions map.

The object compiler is the meatiest part. It sorts properties (required first, then optional, alphabetically within each group) and emits them with proper comma handling:

fn compile_object(
    validation: Option<&ObjectValidation>,
    parent_rule_name: &str,
    ctx: &mut CompileCtx<'_>,
) -> Option<String> {
    // ... validation checks, depth guard (MAX_DEPTH = 2) ...

    let required_set: std::collections::HashSet<&str> = validation
        .required.iter().map(|s| s.as_str()).collect();

    // Sort: required first (alphabetically), then optional (alphabetically).
    let all_fields: Vec<&FieldInfo> = required_fields
        .iter().chain(optional_fields.iter()).copied().collect();

    let mut parts = Vec::new();
    for (i, field) in all_fields.iter().enumerate() {
        let kv = format!(
            "\"\\\"{}\\\"\" ws \":\" ws {}",
            field.key, field.expr
        );
        if i == 0 {
            if field.required {
                parts.push(kv);
            } else {
                parts.push(format!("({kv} )?"));
            }
        } else if field.required {
            parts.push(format!("ws \",\" ws {kv}"));
        } else {
            parts.push(format!("(ws \",\" ws {kv} )?"));
        }
    }

    let body = format!("\"{{\" ws {} ws \"}}\"", parts.join(" "));
    Some(body)
}

So for a Rust struct like this:

#[derive(JsonSchema, Deserialize)]
struct VaultWriteArgs {
    relative_path: String,
    content: String,
    mode: WriteMode,  // enum: "overwrite" | "append"
}

The compiler produces GBNF rules roughly like:

vault-write-args ::= "{" ws "\"content\"" ws ":" ws json-string
                      ws "," ws "\"mode\"" ws ":" ws ("\"overwrite\"" | "\"append\"")
                      ws "," ws "\"relative_path\"" ws ":" ws json-string ws "}"

If the model emits "vault:write" as the tool name, the sampler forces the args to conform to vault-write-args. The model can't use the wrong key. It can't pass a number where a string is required. It can't use a mode value that isn't overwrite or append. The schema is literally part of the grammar.

Arrays work too. For a tool with tags: Vec<String>, the compiler emits a helper list rule:

fn compile_array(
    validation: Option<&ArrayValidation>,
    parent_rule_name: &str,
    ctx: &mut CompileCtx<'_>,
) -> Option<String> {
    let items_expr = match validation.and_then(|v| v.items.as_ref()) {
        Some(SingleOrVec::Single(item_schema)) => match item_schema.as_ref() {
            Schema::Object(obj) => {
                let item_rule = format!("{parent_rule_name}-item");
                compile_schema_object(obj, &item_rule, ctx)
                    .unwrap_or("json-value".into())
            }
            Schema::Bool(true) => "json-value".into(),
            Schema::Bool(false) => return Some("\"[\" ws \"]\"".into()),
        },
        // ...
        None => "json-value".into(),
    };

    let list_rule_name = format!("{parent_rule_name}-list");
    let list_body = format!("{items_expr} (\",\" ws {items_expr})*");
    ctx.extra_rules.push((list_rule_name.clone(), list_body));

    Some(format!("\"[\" ws ({list_rule_name})? ws \"]\""))
}

Which produces something like:

test-array-args-tags-list ::= json-string ("," ws json-string)*

The depth guard (MAX_DEPTH = 2) prevents the compiler from chasing recursive schemas forever. If a schema is too complex (deeply nested oneOf, recursive types, free-form additionalProperties with no fixed keys), the compiler returns None and the tool falls back to a generic json-object rule. Graceful degradation, not a crash.

Per-turn grammar narrowing: the real trick

Here's the thing that really moves the needle for small models: Eris doesn't give the model the full 50-tool grammar on every turn.

Before each LLM call, the orchestrator runs a ToolRouter, a semantic similarity search using the same embedding model as vector memory. The user's message gets embedded and compared against precomputed vectors for each tool (built from tool names, descriptions, and routing hints). Only tools above a cosine similarity threshold make the cut.

The GBNF grammar is then recompiled to include only those tools. If the user says 'what time is it', the grammar's tool-with-args alternation might contain only clock:now. The model can't call vault:write even if it wanted to. Those tokens simply aren't available to the sampler.

The subset cache lives in src/orchestrator/core/llama_gbnf_subset.rs:

pub(crate) fn get_or_compile_subset(
    &self,
    gatekeeper: &Gatekeeper,
    tool_names: &[String],
) -> Result<Arc<str>> {
    let mut sorted: Vec<String> = tool_names.to_vec();
    sorted.sort();
    let key: String = if sorted.is_empty() {
        CACHE_KEY_NO_TOOLS.to_string()
    } else {
        sorted.join("\x1e")  // ASCII record separator as cache key
    };

    let mut guard = self.inner.lock().map_err(|_| {
        FcpError::EngineFault("GBNF subset cache mutex poisoned".to_string())
    })?;

    if let Some(hit) = guard.get(&key) {
        return Ok(Arc::clone(hit));
    }

    // Cache miss: compile grammar for exactly these tools
    let entries: Vec<ToolGrammarEntry> = sorted
        .iter()
        .map(|name| {
            let per_tool_rules = gatekeeper
                .parameters_root_schema_for(name)
                .and_then(|schema| schema_to_gbnf_rule(name, &schema))
                .map(|(_rule_name, rules)| rules);

            ToolGrammarEntry {
                name: name.clone(),
                per_tool_rules,
            }
        })
        .collect();

    let compiled = compile_fcp_envelope_grammar_dynamic(&entries);
    let arc: Arc<str> = Arc::from(compiled.into_boxed_str());
    guard.insert(key, arc.clone());
    Ok(arc)
}

Same tool set on a different turn? Cache hit, no recompilation. The Arc<str> keeps the compiled grammar around cheaply.

And here's how the orchestrator picks which grammar to use on each LLM call, from the main cognitive loop in src/orchestrator/core/step.rs:

let (grammar_override, attach_session_grammar) = if !self.config.is_llamacpp() {
    (None, true)  // Ollama: no grammar support, skip
} else if !tools_needed {
    // No tools matched this turn: compile empty grammar (forces tool_calls: [])
    let g = self.gbnf_subset_cache
        .get_or_compile_subset(&self.gatekeeper, &[])?;
    (Some(g), true)
} else if !targeted_tools.is_empty() {
    // Recovery/retry: only the specific tools we're retrying
    let names: Vec<String> = targeted_tools.iter().cloned().collect();
    let g = self.gbnf_subset_cache
        .get_or_compile_subset(&self.gatekeeper, &names)?;
    (Some(g), true)
} else if slim_assembly {
    // Normal path: pre-LLM router's top-K matched tools
    let offered = slim_offered_tool_names(
        &pre_llm_matched_tools,
        self.tool_map_offer_cap,
        moltbook_overlay_latched,
        &self.gatekeeper,
        &self.state,
    );
    if offered.is_empty() {
        (None, true)  // fall back to full session grammar
    } else {
        let g = self.gbnf_subset_cache
            .get_or_compile_subset(&self.gatekeeper, &offered)?;
        (Some(g), true)
    }
} else {
    (None, true)  // full tool roster, session grammar
};

Four branches, four different grammar scopes. The !tools_needed case is fun: when the router decides the user is just chatting (no tools relevant), the grammar still enforces the protocol envelope but compiles with an empty tool list. That means tool_calls can only be []. The model simply cannot hallucinate a tool call when the router says 'this is just conversation'.

This has two effects:

1. Reliability goes way up. Fewer choices means fewer mistakes. An 8B model choosing between 3 tools is dramatically more reliable than one choosing between 50.
2. Generation is faster. A smaller grammar means less work for the constrained sampler at each token position.

The gatekeeper: defence in depth

Grammar enforcement is necessary but not sufficient. Even with a perfect grammar, the model can still call a tool with semantically wrong arguments (a valid path that doesn't exist, or an action that's not appropriate in the current agent state).

Eris layers a Gatekeeper on top (src/tools/gatekeeper.rs). Every tool call passes through it before execution:

pub async fn execute_tool(
    &self,
    state: &AgentState,
    name: &str,
    args: Value,
) -> Result<String> {
    // 1. State-based authorization
    if !Self::state_allows_tool(state, name) {
        return Err(FcpError::ToolFault {
            tool_name: name.to_string(),
            reason: format!("Tool not authorized in state {:?}", state),
        });
    }

    // 2. Normalize common LLM arg-naming mistakes before validation
    let args = normalize_tool_args(name, args);

    // 3. Runtime JSON Schema validation (belt and suspenders with grammar)
    let compiled_schema = JSONSchema::options()
        .compile(&schema_value)
        .map_err(|e| FcpError::Config(...))?;
    if let Err(errors) = compiled_schema.validate(&args) {
        return Err(FcpError::SchemaViolation(...));
    }

    // 4. Execute
    let result = tool.execute(args).await?;

    // 5. Semantic guard: empty results in Recover state are suspicious
    if state == &AgentState::Recover && result.trim().is_empty() {
        return Err(FcpError::ToolFault {
            tool_name: name.to_string(),
            reason: "Semantic Guard: Tool returned zero logic results during recovery"
                .to_string(),
        });
    }
    Ok(result)
}

The arg normalization is worth a mention. Small models frequently use slightly wrong key names: q instead of query, path instead of relative_path, top_n instead of max_headlines. Rather than wasting a recovery turn, the gatekeeper silently maps common aliases before validation:

fn normalize_tool_args(tool_name: &str, mut args: Value) -> Value {
    let Some(obj) = args.as_object_mut() else { return args };

    if tool_name == "web:search" {
        for alias in ["q", "search_query", "search"] {
            if let Some(val) = obj.remove(alias) {
                if !obj.contains_key("query") {
                    obj.insert("query".to_string(), val);
                }
            }
        }
        obj.retain(|k, _| WEB_SEARCH_KEYS.contains(&k.as_str()));
    }
    // ... similar blocks for web:fetch, web:find, news:today,
    //     vision:see, media:catalog (path -> relative_path) ...
    args
}

This is a pragmatic concession. The grammar enforces structure. The normalizer handles naming drift. The schema validator catches whatever slips through both.

What this looks like in practice

On the Ollama backend (no grammar), a long session with a 12B model hits about 3-5% protocol failures per turn. Malformed JSON, invented tool names, trailing prose after the JSON object. Each failure costs a recovery turn (re-prompt with the parse error), which burns context window and adds latency.

On the llama.cpp backend with GBNF, protocol failures drop dramatically. Short to medium sessions are essentially clean. But I won't claim zero failures: on long contexts where the model starts losing coherence, even grammar-constrained output can degrade. The grammar guarantees syntactic validity (the JSON will parse, the keys will be right, the enum values will exist), but it can't fix a model that fills every string field with garbage or repeats the same tool call in a loop because it's lost the plot at 28k tokens.

That's why Eris doesn't rely on the grammar alone. There's a bounded recovery loop that catches schema violations, empty tool results, and semantic nonsense, feeds the model a concrete correction, and retries with a capped budget. The grammar killed the easy failures (malformed JSON, invented tool names). The recovery loop handles the rest. I'll write more about that mechanism in a follow-up post.

The difference in session stability is still massive. A 100-turn session on Ollama reliably degrades; the same session on llama.cpp with grammar stays usable far longer. Just don't let anyone tell you it's a silver bullet.

Bounded generation: the other half

Even with a perfect grammar, generation can go wrong if the model fills its context window mid-response. llama.cpp will truncate, and your valid-so-far JSON becomes invalid because the closing braces never got emitted.

Eris sets n_predict (max tokens to generate) per call, calibrated to leave room in the context window for the response to actually finish. The grammar ensures structure; bounded generation ensures the structure has room to close.

The architecture, briefly

For context on where the grammar fits in the bigger picture:

• Single Rust binary. ~270 source files, ~50 tools, three presentation surfaces (ratatui TUI, localhost web UI via Axum+SSE, optional Discord sidecar), all sharing one orchestrator.
• Markdown vault as memory. Notes in an Obsidian-compatible directory tree. Tiered memory: ephemeral staging (in-process), committed (written to vault), semantic recall (Qdrant vector search over vault chunks). Turn-start prefetch embeds the user message and injects relevant memories into the system prompt before the LLM even runs.
• No unsafe, no panics, no println. Production code uses ? propagation through a typed error taxonomy (FcpError). Diagnostics go through tracing to rotating log files under .fcp/telemetry/. Never to the TUI buffer. println! would corrupt the ratatui render.
• Actor model concurrency. No Arc<Mutex<T>> shared state between threads (well, except the grammar cache, which is a cold path). Communication via tokio::sync::mpsc channels.

Should you do this?

If you're building an agent on local models and you need structured output: yes, absolutely. GBNF grammar enforcement is the single biggest reliability win I've made on this project. More impactful than prompt engineering, more impactful than model selection, more impactful than retry logic.

The key insight is that the grammar is not a prompt. It's not a suggestion. It's a constraint on the sampler. The model can't violate it any more than it can output a token that doesn't exist in its vocabulary. That's a qualitative difference from anything you can do with system prompts.

The tradeoff: you're coupled to llama.cpp (or any sampler that supports grammars; VLLM has experimental support, Outlines does it differently). You need to write or compile the grammar. And very complex schemas (recursive types, polymorphic unions) may not compile cleanly. But for the typical agent tool protocol (an envelope with a discriminated union of tool calls) it works beautifully.

Try it

git clone https://github.com/janpauldahlke/eris
cd eris && cargo build --release
docker run -d -p 6333:6333 -p 6334:6334 \
  -v eris-qdrant-data:/qdrant/storage qdrant/qdrant
./target/release/eris chat   # first run launches the setup wizard

The code is at github.com/janpauldahlke/eris. The project site is eris-system.dev. Apache 2.0. Stars and contributions welcome.

---

• If you're working on local AI agents, structured generation, or Rust+LLM integration, reach out or open an issue.*