Every team shipping an LLM feature eventually hits the same wall: the thing you built is non-deterministic, and your whole testing culture assumes it isn't. assertEqual(output, expected) is meaningless when the output is a paragraph of generated prose that will be slightly different next time.
The usual responses are both bad. One is to shrug and ship it with no verification at all: "it looked good when I tried it." The other is to test the model itself, chasing a moving target that changes every time the prompt or the weights do.
I built an internal tool recently that leans on an LLM for exactly one step, and I wanted a real answer to "how do you know it works?" that I could defend out loud. The answer turned out to have two moves. First: make the LLM's job as small as you can get away with, so most of the system stays deterministic and ordinary unit tests still apply. Second: for the irreducibly non-deterministic part that's left, build an actual eval harness, and be honest about what each layer of it can and can't catch.
Here's how that played out.
The problem the tool solves
During portfolio onboarding, an upstream platform detects data gaps: a missing utility bill for a building, a meter that shows up in a utility feed but isn't matched to any known space, an incomplete equipment inventory, an ambiguous "is this tenant-paid or owner-paid?" It produces an anomaly report.
Someone on the internal team then has to turn each detected gap into a specific, correctly-addressed request to the right person at the customer (the property manager for this building, the asset manager for that meter), and chase every one of them to resolution before a 30-day onboarding clock runs out. That's read-the-report, figure-out-who-owns-it, draft-the-email, track-who-responded, repeat across dozens of buildings.
The tool takes the anomaly report and, for each gap: routes it (building, account, owner, severity), creates a tracked record, drafts the outreach, and holds that draft for human approval before anything is sent. The specialist reviews and approves instead of starting from a blank page.
Notice how much of that is not an LLM problem.
Move one: shrink the LLM's job
Routing is a lookup. Severity comes straight off the upstream signal. Tracking is a state machine. Approval is a guarded transition. None of that is ambiguous, and none of it should ever be delegated to a model that might get creative.
So the architecture splits cleanly in two:
- Deterministic spine: routing, tracking, state transitions, and every send-safety check are plain code.
- LLM-assisted step: exactly one thing, the drafting of the outreach message, which is the only part where "turn this structured gap into a polite, specific paragraph a human would actually send" genuinely benefits from a language model.
Routing is pure functions over dictionaries, with no model I/O anywhere near it:
ROLE_BY_GAP_TYPE = {
GapType.MISSING_UTILITY_BILL: OwnerRole.PROPERTY_MANAGER,
GapType.UNMATCHED_METER: OwnerRole.ASSET_MANAGER,
GapType.INCOMPLETE_EQUIPMENT_INVENTORY: OwnerRole.BUILDING_ENGINEER,
GapType.TENANT_OWNER_PAID_AMBIGUITY: OwnerRole.ASSET_MANAGER,
}
Severity is a good illustration of the discipline. The free-text detail field often hints at urgency, and it would be tempting to let the model infer it. I don't. Severity is normalized from the explicit upstream signal, and if it's missing or invalid the anomaly fails loudly rather than getting a guessed value. The rule is simple: trust the deterministic signal, never let the model manufacture one.
The LLM sits behind a provider interface (a one-method protocol), so business logic never touches an SDK directly:
class LLMProvider(Protocol):
def complete(self, prompt: str) -> str:
...
That interface is a boundary and a swap point. Locally it calls Claude directly; in production it becomes a Bedrock call by changing one factory function. And in tests, it becomes a fake that returns canned strings, which means the entire deterministic spine can be verified with no API key and no network calls at all.
That's the payoff of move one. Because the LLM's territory is small and walled off, the majority of the system is just software. It gets 28 ordinary unit tests: given this anomaly, routing produces exactly this record; an unroutable anomaly raises; an approval on a gap in the wrong state is rejected. Pass/fail, checked by code, run on every push in CI. No cleverness required.
Move two: the one step that's left is still non-deterministic
You can't make the drafting step deterministic without defeating the point of using a model for it. So this is where the eval harness lives, scored against a golden dataset of hand-labeled examples, including deliberately adversarial ones. It checks three dimensions, and the interesting part is that they are not equally trustworthy, and I don't pretend they are.
Groundedness Eval: deterministic
The most important property is that a draft may reference only the single gap it was drafted for. A draft that name-drops a different building's ID, or another gap entirely, is a hard failure: leaked context, and it must never reach a human as if it were correct.
This one I can check deterministically, because leaked identifiers have structure. A regex over the draft finds any G-#### or BLD-### token that isn't the one allowed for this gap:
def check_groundedness(draft: str, gap: dict) -> GroundednessResult:
violations = []
foreign_gap_ids = _foreign_ids(draft, gap["gap_id"])
if foreign_gap_ids:
violations.append("references gap ID(s) not allowed: " + ", ".join(foreign_gap_ids))
foreign_building_ids = _foreign_ids(draft, gap["building_id"])
if foreign_building_ids:
violations.append("references building ID(s) not allowed: " + ", ".join(foreign_building_ids))
return GroundednessResult(is_grounded=not violations, violations=violations)
What it honestly does not catch: a prose-level scope violation with no ID attached, such as a building referred to only by description, or a speculative note in the source restated as confirmed fact. Those require semantic judgment. This guardrail catches structured leaks, and I'm precise about that rather than claiming it catches all of them.
Because it's deterministic, the same function does double duty: it's a live guardrail in the request path and a scored dimension in the eval harness.
Hallucination Eval: heuristic
This layer flags numeric and date-like tokens in the draft that have no basis in the source data: an invented kWh figure, a capacity number, a date outside the stated window.
I want to be very clear about its limits, because this is where it's easy to fool yourself. This is string matching, not semantic detection. It catches a fabricated figure. It cannot catch a fabricated assertion with no number attached, the classic case being a gap that raises the question "is this meter tenant-paid or owner-paid?" and a draft that quietly answers it. There's no number to flag there; the hallucination is a claim, not a digit. Several traps in the dataset are exactly this shape and sit outside this eval's reach on purpose. A pass here is a partial signal, not proof.
Tone Eval: LM judge
The last dimension is genuinely subjective (is this the voice of an institutional asset manager, specific and not apologetic, with one clear ask?), so it's scored by a model. Each example carries its own tone rubric, some of which describe traps to avoid rather than just a style, and the judge returns 1-5 scores across specificity, professionalism, single-clear-ask, and rubric adherence, with a passing threshold of 4.
An LM judge is the least deterministic tool in the box, which is exactly why it's confined to the most subjective dimension and nowhere near the send-safety checks.
The dataset is where the judgment lives
The harness is only as good as what it scores against. The golden set is small (eleven hand-labeled examples), but each one carries must_mention, forbidden_references, must_not_invent, and tone_notes labels, and several are built as traps.
The tenant/owner-paid ambiguity case is my favorite. The correct draft surfaces the question and asks the recipient to clarify. An incorrect draft resolves it, picking an answer the source never supported. That's the single most valuable thing the whole system has to get right, and note that it's precisely the failure the heuristic hallucination eval can't see. It lives in the dataset and the tone rubric because that's where the semantic judgment has to happen. The harness isn't magic; it's a place to encode the traps you already understand.
Reject, don't show
One design choice I'm happy with: when a draft fails the groundedness guardrail in the live path, the draft text is never persisted and never returned. The gap record still exists so a human can see that routing succeeded and the draft attempt failed, but the ungrounded text itself doesn't get shown, on the theory that a plausible-looking wrong draft is more dangerous than no draft. A failure should be visible as a failure, not as a suggestion.
Why the evals don't gate CI
The unit tests run on every push. The evals do not. Two reasons: they make live API calls (cost and flakiness), and the tone dimension is a non-deterministic judge that would make the pipeline red for reasons that have nothing to do with a regression. So the deterministic spine is CI-gated, and eval runs are a local step whose results (full model I/O, latency, token usage, and scores) get written to a per-run observability directory instead. You get the history without pretending a subjective judge is a build gate.
What I'd do differently
The harness is a starter, and I'd rather say so than oversell it. Eleven examples run sequentially with no parallelism, no retries, and no eval-set versioning is enough to catch regressions while developing a prompt; it is not a continuously-maintained eval set, and it shouldn't ship as if it were. The hallucination heuristic wants to grow toward semantic checking of unsupported claims, not just unsupported numbers. And the tone threshold is a guess until real drafts have been reviewed by an actual human and the number is tuned against their judgment rather than my intuition.
The pattern, extracted
If you're putting an LLM into a system where being wrong has a cost, the shape that worked for me was:
- Shrink the model's job until it owns only the genuinely ambiguous step, and keep everything deterministic (routing, state, safety checks) in code where unit tests still work.
- Put the model behind an interface so it's swappable and, more importantly, fakeable in tests.
- Eval the one non-deterministic step across layers of decreasing trust: deterministic checks first, heuristics second, an LM judge only for what's truly subjective, and be explicit about what each layer can't catch.
- Encode the traps in the dataset, because that's where your actual domain judgment lives.
- Reject bad output rather than display it, and keep the subjective judge out of your build gate.
You can't unit-test the model. But you can build a system where most of it doesn't need a model at all, and put a harness with honest limits around the part that does.
The code is on GitHub: github.com/amirmarcel/partner-strategy-copilot. Django REST Framework, Claude behind a provider interface, the eval harness in evaluations/, and the golden dataset with its traps in golden_dataset/.
I'm a software engineer interested in the architecture of AI-assisted systems: where the deterministic boundaries should sit and how you verify the parts that cross them. Feedback and pushback welcome.
Top comments (1)
The split I like here is making the model write only the part that is actually ambiguous. Routing, state, approval, and safety checks are better as plain code because you can diff them and make them fail closed. The test I use is whether a bad model output can change who owns the next action. If yes, the boundary is too soft.