I Kept Retrying a Local Model Into the Right Shape. Turns Out I Didn't Have To Retry At All.

Naitik Kapatel — Fri, 07 Aug 2026 12:28:55 +0000

The constraint on this one wasn't the AI part, it was the client: nothing leaves the device. No API calls, no cloud model, full stop, because the documents being classified weren't allowed off the machine they landed on. So the whole pipeline runs on a quantized model through llama.cpp, and its one job is boring on purpose - read a chunk of text, output exactly one of five category labels, nothing else.

Boring is exactly what a local 3B model is bad at, it turns out.

First pass: just ask nicely and hope

I asked for one of the five labels, plainly, in the prompt, the way you'd ask any model. It worked most of the time. "Most of the time" is not a number you get to keep when a pipeline runs unattended over a folder of a few thousand files overnight. The failures were never wild, either, which almost made it worse - a trailing period, the label wrapped in a sentence ("The category is: invoice"), once in a while a label that wasn't even in my list of five. Small drift, but small drift at that volume means a batch job you can't trust without babysitting it.

Second pass: validate after, retry on failure

I was already using shapecraft for the schema side of other parts of this project, so the obvious next move was its regex schema type - match the output against ^(invoice|receipt|contract|memo|other)$, and if it doesn't match, retry. That worked, in the sense that the final output was always correct. What it cost me was time, and on this hardware, time is the thing I had the least of. No GPU on the target machine, CPU inference only, and a chunk that fails the pattern means paying for a full second generation just to get a label I'd already told the model was one of five options. Multiply that by however many chunks drift, over a few thousand files, and the "safety net" was quietly doubling my runtime on the exact runs where it triggered.

Retrying into correctness is still trusting the model to eventually get lucky. I wanted it structurally unable to answer wrong in the first place.

What was already sitting in the schema-input list

Turns out shapecraft has a schema type for exactly this, and I'd walked past it twice without registering what it did:

import { generate, llamaCpp } from "@aviasole/shapecraft";

const labelGrammar = `root ::= "invoice" | "receipt" | "contract" | "memo" | "other"`;
const local = llamaCpp({ modelPath: "./models/llama-3.2-3b-instruct.gguf" });

const result = await generate(local, { gbnf: labelGrammar }, chunkText);

console.log(result.data);           // one of the five, every time
console.log(result.guaranteeLevel); // "constrained"

{ gbnf: ... } hands llamaCpp() a real GBNF grammar - the same grammar format llama.cpp itself has supported for ages - and it gets applied at the token level during generation. The model isn't producing "invoice." and getting caught after, it's mechanically incapable of producing a token outside invoice | receipt | contract | memo | other in the first place. No retry loop, because there's nothing left for a retry to catch.

That also explained the trailing-period and wrapped-sentence failures from pass one in a way "the model is being sloppy" never quite did: those weren't sloppiness, they were the model doing exactly what an unconstrained decoder does, which is treat every token as fair game unless something is physically stopping it. A grammar is that something.

The assumption that cost me twenty confused minutes

llamaCpp() reports guaranteeLevel: "constrained" as its nominal level, and for a minute I read that as "everything I run through this backend is constrained." It isn't - that label only means what it says for actual gbnf inputs. Point the same backend at a plain Zod schema and it's still a prompt-and-hope path underneath, same as any cloud backend without a native grammar hook. Made sense once I thought about it - there's no generic JSON-Schema-to-grammar step doing that translation for you - but I'd assumed the guarantee traveled with the backend instead of the schema type, and briefly wondered why my Zod-typed calls on the same local model weren't as bulletproof as the label ones.

The failure mode I didn't expect to appreciate

I fat-fingered a rule reference in one grammar revision - referenced a rule I'd renamed and forgotten to update. Expected to find that out however these things usually surface: deep into a run, after the multi-gigabyte model had already loaded, staring at a stack trace from whatever node-llama-cpp does with a broken grammar. Instead it errored immediately, before the model load even started, with a plain message pointing at the bad reference. Small thing, but on hardware where loading the model itself takes a noticeable chunk of a minute, catching a typo before that step instead of after is the difference between a five-second fix and a five-minute one, repeated every time I tweak the grammar.

Where that leaves the pipeline

Same five-label job, zero retries, and the failure cases that used to be quiet drift I only caught by spot-checking output are now not reachable in the first place. The overnight batch run that used to occasionally need a rerun over its failed chunks now doesn't have failed chunks to rerun.

If you've got a local model that needs to hit a fixed shape every time and no budget to retry it there, check { gbnf: ... } against llamaCpp() in @aviasole/shapecraft before you build a retry loop to paper over the drift. I built the retry loop first. Didn't need it.