Keeping the LLM out of the verdict
I've been building a system that audits AI-generated fitness training plans against published sports-medicine guidelines. A fitness app sends in a plan its LLM produced, and the system returns a verdict: pass, flagged, or rejected — with the specific rule and source it was checked against.
The obvious way to build this is to hand the plan to a good model with a well-written prompt: "Here are the ACSM guidelines. Does this plan comply?" It would work. Most of the time.
I didn't do that, and this series is mostly about why, and what it costs.
The problem with asking a model to be the judge
If an LLM produces the verdict, the verdict inherits every property of an LLM. It's non-deterministic — the same plan can pass on Monday and get flagged on Tuesday. It's unfalsifiable — when it says a plan violates a guideline, you can't check which threshold it thinks was crossed, because there wasn't one; there was a probability distribution over tokens. And it's unauditable in the sense that matters most here: you can't diff it. Change the prompt and you can't tell what else you changed.
For a lot of applications this is fine. For an audit tool, it's fatal. The entire value proposition is "here is independent evidence that this content was checked." An LLM-produced verdict is not evidence. It's a second opinion with no paper trail.
So the constraint I started from: no LLM call may sit in the judgment path.
Where the LLM actually belongs
That constraint doesn't mean no LLM at all. It means being precise about what an LLM is genuinely good at.
Reading a paragraph of free-form English written by another model and pulling out "3 sets, 8-10 reps, 75% of 1RM, twice a week" — that's a language task. There's no rule engine on earth that does it well. That's what the model is for.
Deciding whether "75% of 1RM" falls within "70-85% of 1RM as recommended in Table 1" — that is not a language task. That's a comparison between two numbers, and Python does it perfectly, every time, for free.
So the system splits into two zones:
free text ──▶ [ LLM: extraction ] ──▶ structured data ──▶ [ code: rules ] ──▶ verdict
↑ probabilistic ↑ deterministic
Everything to the left of the arrow is allowed to be uncertain. Everything to the right of it must not be. I've started calling these Zone A and Zone B in my own notes, and the whole architecture is really just an ongoing effort to keep the boundary between them from leaking.
The boundary leaks constantly
This sounds tidy in a diagram. In practice, almost every bug I've hit in this project has been the boundary quietly failing.
An early one: the extractor was correctly pulling age_years: 14 out of a request — the number was right there, the model got it — but the routing logic that decides which rule set applies never checked it. It only looked at a separate minor boolean, which the model had left null because the text never used the word "minor." So a plan for a 14-year-old was being evaluated against adult resistance-training guidelines, cheerfully passing.
The fix is embarrassingly small:
python
Age is a number we already have. Do not ask the model
to also tell us what that number means.
if plan.get("age_years") is not None and plan["age_years"] < 18:
plan["minor"] = True
But the category of bug is the interesting part. The model had done its job — it extracted the age. The failure was that a deterministic implication (age < 18 → minor) had been left on the model's side of the boundary, where it was subject to interpretation, instead of on the code's side, where it isn't. Any time you find yourself hoping the model will infer something that follows mechanically from data you already have, you've put the boundary in the wrong place.
What this costs
Two things, and they're both real.
It's slower to build. A prompt that says "check this against the guidelines" takes an afternoon. Extracting the guidelines into explicit, individually-sourced rules took me weeks, and I'm still not done. Every number in every rule has to be traced back to a direct quote in the source PDF before it ships. There is no shortcut where the model reads the paper and writes the rules for me — I tried, and the failure mode is that it produces plausible rules, which is the worst possible outcome for an audit tool.
It's more brittle at the edges. A deterministic rule engine has no common sense. If a case falls outside every rule, it doesn't gracefully guess — it returns nothing. That's the correct behavior for an auditor, but it means the coverage gaps are yours to find, and they don't announce themselves. I'll write about one of those in a later post: a design change that silently switched off a whole category of existing rules without a single test failing.
What you get in exchange is that the verdict means something. When the system says a plan is fine, it can show you the specific facts it checked and the specific thresholds they were compared against. When it says a plan has a problem, you can look up the sentence in the source document. And when I change a rule, git diff tells me exactly what changed about the system's judgment — which is not a thing you can say about a prompt.
What's next in this series
This is the first post in a series about the specific engineering problems that fall out of that one constraint. Planned so far:
Extraction confidence is not symmetric. Two fields in the same schema can need opposite uncertainty policies, depending on which direction the failure mode runs. Getting this backwards silently produced false "pass" verdicts.
Splitting one LLM call into two made it 3x faster — but only for a third of requests, and the reason why turned out to be more interesting than the speedup.
The coverage gap that no test caught. How making two rule sets mutually exclusive silently disabled rules that had been working for months.
The rule engine layer is open source (MIT) if you want to see the actual implementation rather than my description of it: https://github.com/guardamos-developer/audit-engine
I'd genuinely like to hear from anyone who's drawn this boundary somewhere else, especially if you put more on the model's side and it held up. My priors here are strong but they're built on one project.
Top comments (0)