Trust the parser, not the prompt: what running a cheap LLM in production taught me

Alexander Likhachev — Wed, 05 Aug 2026 15:20:57 +0000

I run an LLM in the core loop of a small production app: Mening corrects language learners' writing every day. Unit economics put the everyday call on a cheap, fast model tier, and the output feeds a database, so it has to obey a contract: strict JSON, a closed set of error categories, explanations in the learner's UI language, no invented "fixes".

The cheap tier is genuinely good at the task. It is terrible at following the rules around the task. This post is about the escalation ladder I climbed before accepting the house rule in the title, with real numbers from the two fights that taught me it.

Fight one: the model that wouldn't stop "correcting" 了

Chinese learners kept getting a specific non-correction: the model would insert 了 into sentences that were already fine, then explain why the learner "needed" it. A native-speaker check said the original sentences were correct.

I climbed the prompt ladder one rung at a time, deploying and watching live traffic after each:

A judgment-style rule ("only flag genuine errors").
Concrete examples of what not to flag.
Moving the rule later in the system prompt, close to the generation point.
An absolute rule, in caps, no exceptions.

Four iterations, and the model still did it. Not always - which is worse than always, because it looks fixed until it isn't.

The fix that held wasn't wording. The parsing boundary now drops two classes of edits before anything reaches the database: no-op edits, where the "wrong" and "correct" strings are equal, and pure 了-insertion edits. A few lines of Go, zero regressions since. The prompt still asks nicely; the parser doesn't care whether the model listened.

Fight two: explanations drifting into the wrong language

The contract says: explain errors in the user's UI language (say, Russian), never in the target language (say, Finnish). A real user reported Finnish explanations. Intermittent, of course.

Before touching anything I measured it: the same seven real submissions, 21 runs, exact production request shape. Six out of 21 responses drifted (29%), and drift was all-or-nothing per response - the model commits to one language for the entire JSON.

Then I measured the fixes everyone reaches for first:

Temperature to 0.0: still 5/21 drifted. Temperature is not the lever - the coin flip isn't sampling noise, it's the instruction losing to the payload. When the input is dominated by Finnish text, "answer in Russian" fades.
Moving the language rule to the end of the system prompt: 3/21. Better, still broken.
Repeating the rule as the last line of the user turn, after the learner's text: 0/21.

That last one is the cheapest lesson in this post: the system prompt is far from the generation point, and for a small model, distance matters. A rule that must survive belongs in the user turn, next to the data that fights it.

But 0/21 on a 21-run sample is not a guarantee, so the boundary got a guard anyway. offScriptExplanations checks the writing system of every explanation in the response against the UI language's script. Deliberately dumb on purpose:

It's a script check, not language identification - no model call, no dependency, just Unicode ranges.
It only fires when the UI and target scripts differ (Cyrillic UI vs Latin target). Same-script pairs are inert, because there a script check can't tell the languages apart, and guessing is how guards become bugs.
It requires the whole response to be off-script before acting. A quoted Finnish snippet inside a Russian explanation must never trip it.

When it fires, the correction runs one repair retry with the violation spelled out. And if the retry is still wrong, the wrong-language answer is kept, not dropped - a correction the user has to squint at beats no correction at all. Guards should degrade, not destroy.

The quiet third fight: the schema itself

Every error the model reports lands in a closed set of six categories, enforced twice - a CHECK constraint in SQLite and validation at the parse boundary. Anything outside the set is rejected, and any fields the model invents are dropped on the floor.

That last part turned out to be a free security property. The model output is the only untrusted input in the system, and the parser treats it accordingly: no tool calls to hijack, no extra fields to smuggle instructions through, a hard token cap. Prompt injection against this pipeline mostly has nowhere to go, not because the prompt says "ignore injections", but because the boundary only accepts the shape it expects.

The ladder, summarized

What I now do, in order, when a cheap model breaks a rule:

Rewrite the rule with concrete names, not abstractions ("in Russian, never in Finnish" beats "in the UI language").
Move it late in the system prompt.
Repeat it as the last line of the user turn, after the data.
Measure on real inputs - 20 runs tells you more than any prompt review.
If it still leaks even rarely: stop prompting. Enforce it at the parse boundary, with a repair retry if the output is salvageable and a degrade path if it isn't.

Rungs 1-4 reduce the failure rate. Only rung 5 sets it to zero, and the rules that reach rung 5 are exactly the ones where "rarely" is unacceptable.

The prompt is a request. The parser is a contract.

The app this comes from is mening.app - daily writing practice that remembers which mistakes you keep repeating. The memory side of it is written up in error memory under the hood.

DEV Community: Alexander Likhachev

Trust the parser, not the prompt: what running a cheap LLM in production taught me

Fight one: the model that wouldn't stop "correcting" 了

Fight two: explanations drifting into the wrong language

The quiet third fight: the schema itself

The ladder, summarized