The bug looked like the model being dumb. It wasn't. It was me asking for the wrong thing.
I had an agent with a refund tool — issue_refund(order_id, amount_cents, reason). Somewhere in a long conversation the model emitted "order_id": "ORD-88213". No such order existed. There was 88231 and 88213 was one transposition away. It had interpolated a plausible-looking string out of the digits floating around the context window, the refund service returned a 404, the agent decided the right move was to retry with variations, and forty seconds later I had a loop and a bill.
I spent a while on the prompt. Added "NEVER invent order IDs". Added a few-shot example. Added a reminder at the end of the system prompt. It got better, which is the worst outcome, because "better" means it fails on the tenth request instead of the third, and now you trust it.
It's a decoding problem
Here's the thing I already knew and wasn't applying. When I ask a model to write a string, it does exactly what it's trained to do: produce the most probable continuation. ORD-88213 is a perfectly probable continuation of a context full of order IDs. I was asking for generation and then being upset that I got generation.
But I wasn't in a generation situation. At the moment the agent decides to call the refund tool, I already know everything that could possibly be correct. The order IDs in scope come from the session. The legal amounts come from the order record. The reasons come from my own enum. There is a finite, known, small set of valid arguments. I was asking a language model to guess its way into a set I could have just handed it.
So I stopped asking for strings. Now I compile the choices first.
def compile_options(tool_name, session):
order = session.focused_order
return {
"order_id": [o.id for o in session.orders_in_scope],
"amount_cents": [str(x) for x in refundable_amounts(order)],
"reason": ["damaged", "not_delivered", "duplicate_charge", "late", "other"],
}
Then I either build a JSON schema with enum on every field, or — more often — I break the call into one decision per field. The model never writes an identifier. It picks index 3.
That's it. That's the tip. It sounds too simple to be worth a post, but the number of production agents I've read that ask a model to spell out a UUID is not small, and mine was one of them until recently.
Why field-by-field beats one JSON blob
I tried the obvious version first: one schema, all fields enums, generate the whole object in one constrained decode. It works and it's fewer round trips, but I get better accuracy splitting it up, and the reason is boring — the option sets are per-field. order_id has maybe 5 candidates, reason has 5, amount_cents has 3. Asking a 3B model to make one 5-way choice at a time is a task it's genuinely good at. Asking it to make a 75-way joint choice inside a nested structure is asking for trouble, and the failure modes get harder to read.
Splitting also means each decision is individually loggable and individually testable. When a refund goes wrong I can see which field was picked wrong, which turns a vague "the agent messed up" into a one-line diff.
If you don't want to touch grammars, you don't have to. Any server that returns token logprobs will do it:
scores = {
opt: logprob_of_completion(prompt_for_field, opt)
for opt in options[field]
}
choice = max(scores, key=scores.get)
Two caveats I hit. Options that tokenize to different lengths need length normalization or you'll systematically prefer short ones. And options that aren't a single token — anything with punctuation or mixed case — are better handled by a real grammar/constrained decode than by logprob arithmetic. I use grammars when I'm already running llama.cpp and logprobs when I'm not. Both work; the grammar one is less code.
Keep an escape hatch, and watch it
The failure mode of a closed set is that it's closed. If the correct value isn't in the options, the model picks the least-wrong one and you never find out. Your tool silently refunds the wrong order instead of loudly failing.
So every compiled field gets an explicit "none_of_these" option. It costs one slot. When the model takes it, the tool call aborts, the agent asks the user, and — the part that matters — I log it. Abstain rate per field is now a metric. When order_id abstains on more than about 2% of requests, my candidate compiler is missing a case, usually orders that got attached to the session late or a filter I wrote three months ago and forgot. That's a real signal about my code, not about the model, and I would never have seen it if I'd let the model freehand the string.
What this doesn't fix
Picking the wrong option is still on the table. Enum-constrained order_id can't invent ORD-88213, but it can absolutely choose 88214 when it should have chosen 88213 because the conversation mentioned both and the model's attention landed on the wrong one. That's a ranking error, not a hallucination, and constrained decoding does nothing for it.
What it does give you is something you can measure. A free-text argument is nearly impossible to evaluate against a labeled set without fuzzy matching that hides the exact errors you care about. A closed set is a classifier. Two hundred logged requests with the right answer, run the same prompts, get accuracy per field. That's a number you can put in a PR and defend.
Also, don't do this everywhere. Half the arguments my agents pass are legitimately free-form: a search query, a code edit, a summary, the body of a message. Those aren't arguments, they're content. Content needs generation, and then it needs a validator on the other side — schema check, length check, whatever. The rule I've landed on is: if a value can be enumerated, enumerate it; if it can't, it's content and it needs a validator, not a prayer. The structural args are the ones that break pipelines because they propagate — a wrong order ID doesn't fail at the tool, it fails six steps later in a report nobody's watching.
The side effect I didn't expect
The latency and cost win was bigger than I expected. On my box, a 3B model with a constrained decode over four small fields lands around 150ms p50 — one datapoint, not a benchmark, don't quote me. The hosted 70B-class call it replaced was closer to 800ms, mostly because I was shipping the whole tool spec and the whole conversation every time. Picking from five options was never a reasoning task. I was renting a large model to do a lookup.
The genuinely hard parts of the agent — deciding whether to refund at all, reading an angry email and figuring out what the person actually wants — still go to the big model. That's where the intelligence is doing work. It was never in the spelling.
Try it on whichever tool of yours has an ID argument. That's where the win is, and it's a thirty-line change.
Top comments (0)