I Had a GBNF Grammar File and Nowhere to Point It. So I Added Support for It.

I was building a classification step that had to run entirely on a local model - no cloud calls, straight through llama.cpp. Read a chunk of text, output exactly one of a handful of fixed labels. It ran in a tight, low-latency loop with basically no retry budget, and local models - even good quantized ones - drift more on output formatting than the big hosted APIs do. Wrap the label in a sentence, add a stray quote, whatever. Small thing, but multiplied over a few thousand calls it's a real failure rate.

The fix already exists: llama.cpp grammars, better known as GBNF. Hand llama.cpp a grammar and it masks the token logits at every generation step, so the model literally cannot produce a token that would break the grammar. Not "probably won't" - cannot.

I already had a grammar file. What I didn't have was a way to hand it to the structured-output library I was using (shapecraft) - it had five schema-input types at the time (Zod, JSON Schema, regex, a validator function, XML), and none of them were built for "apply this formal grammar directly."

Why the existing schema types didn't fit

• Not JSON Schema - GBNF describes a context-free grammar over an arbitrary string, not a JSON shape. My output was going to be a matched string, not an object.
• Not a regex pattern - GBNF grammars are recursive (rules referencing rules, alternation, repetition). You can't flatten that into one flat regex.
• Not a validator function - a validator checks output after generation. I wanted the model physically unable to produce the wrong shape, not a check that catches it afterward.

So I added a schema type that did fit.

What I built

A new schema-input type, { gbnf: grammarString }:

import { generate, llamaCpp } from "@aviasole/shapecraft";

const sentimentGrammar = `root ::= "positive" | "negative" | "neutral"`;
const local = llamaCpp({ modelPath: "./models/llama-3.2-3b-instruct.gguf" });

const result = await generate(local, { gbnf: sentimentGrammar }, "Classify: 'I love this!'");

console.log(result.data);           // "positive" - cannot be anything else
console.log(result.guaranteeLevel); // "constrained"

result.data is a raw string, not a parsed object - GBNF describes a string language, not a JSON shape.

A llamaCpp() backend (node-llama-cpp) - the thing that actually applies the grammar at the token level.

A bundled GBNF interpreter for every other backend (openai, groq, anthropic, ollama), none of which have a grammar parameter. There, the grammar gets injected into the prompt instead, and the returned string is checked against it - mismatch, retry:

import { generate, openai } from "@aviasole/shapecraft";

const date = await generate(openai({ model: "gpt-4o-mini" }), {
  gbnf: `root ::= year "-" month "-" day\nyear ::= [0-9]{4}\nmonth ::= [0-9]{2}\nday ::= [0-9]{2}`,
}, "When did WWII end in Europe?");

console.log(date.data);           // "1945-05-08"
console.log(date.guaranteeLevel); // "best-effort" - no grammar param on this API

Same grammar, same call, two very different guarantee levels depending on the backend. GBNF ended up being the one schema type where the guarantee genuinely depends on which backend is running it, not just retry count.

Then I tried to break my own validator

I don't trust a validator only tested against grammars that were supposed to pass.

Catastrophic backtracking - the classic thing that blows up naive regex engines, like ("a" "a"?)* "b" against a long run of as with no trailing b. Didn't blow up - milliseconds, every time. The matcher dedupes by position reached, not path taken, so it doesn't re-explore the same dead end repeatedly.

A genuinely adversarial grammar did slow it down - a ~26-way ambiguous alternation against a 300,000-character input with no valid terminator. That trips a step-budget guard in under a second with a clear error instead of hanging.

Deep right-recursion through a rule reference found a real limit. */+ repetition is iterative, no limit tested up to 50,000+ repeats. But a grammar like list ::= item "," list | item recurses through the JS call stack once per repetition, and breaks around 900-1000 reps. Instead of a raw stack-overflow trace, it now throws a clear message telling you to rewrite with */+ instead.

A real bug this surfaced

Stress-testing also found something unrelated but genuinely broken: the Groq backend was forcing response_format: json_object on every call, including GBNF ones. Groq's API rejects that mode if the prompt doesn't literally contain the word "json" - which a GBNF grammar prompt has no reason to. Every GBNF call against Groq was failing before the model even ran. Two-line fix, but only found by trying to break the feature instead of just checking the happy path.

Where that leaves it

From "I have a grammar file and nowhere to point it" to a schema type that's genuinely valid by construction on a local llama.cpp model, and checked (not just trusted) everywhere else - same call shape, same retry behavior as every other schema type.

If you've got a .gbnf grammar sitting around, or want a local model physically incapable of drifting off a fixed format:

import { generate, llamaCpp } from "@aviasole/shapecraft";

Repo's at github.com/aviasoletechnologies/shapecraft, package is @aviasole/shapecraft on npm. Curious if anyone else here is running GBNF grammars against local models, and what for - I only had the one use case in mind.