When your AI agent is a demo, "it works" means it gave a good answer when you tried it. When your AI agent quotes prices and takes orders for a real business, "it works" has to mean something you can actually measure — because a 2% failure rate that's charming in a demo is a stack of wrong invoices in production. We build agents that run live operational work for businesses, and the question we get asked least but should get asked most is: how do you know it's right? This is how we test them.
You can't unit-test a conversation, so stop trying to
The first instinct from a normal engineering background is to write assertions: given this input, expect this output. It falls apart immediately, because there are twenty valid ways for the agent to ask "which timber grade did you want?" and a string match will fail nineteen of them. Testing a language agent isn't testing a function; it's testing a behaviour under variation. So the unit of testing isn't an input/output pair — it's a scenario plus a set of properties that must hold no matter how the wording lands.
A scenario for our sales agent looks like: "A trade customer asks for materials for a small decking job, one item is out of stock at their branch." The properties that must hold: it resolved every product to a real SKU or asked a clarifying question, it never quoted a price that didn't come from the pricing engine, it flagged the out-of-stock item rather than silently dropping it, and it never confirmed an order the validation layer would reject. Notice none of those are about exact words. They're about what must be true of the outcome.
Two layers: deterministic checks and judged checks
Some of those properties are cheap and binary, so we check them in code. Did every price in the transcript match a get_price call result? That's a hard assertion — the agent physically cannot pass unless the numbers trace to the system of record. Did it ever call create_order on a basket containing an invalid line? Binary. These deterministic checks catch the failures that actually cost money, and they never flake.
The softer properties — was the reply clear, did it correctly understand an ambiguous request, was the tone right for a trade counter — get checked by an LLM-as-judge, scoring each transcript against a rubric. This is where people get nervous, and rightly: a judge model has its own error rate. So we don't treat its score as truth, we treat it as a filter. It surfaces the transcripts most likely to be wrong, a human reviews those, and the human labels become the regression set. The judge scales the reviewing; it doesn't replace the reviewer.
The regression set is the actual product
Every real conversation that went wrong — every misread request, every clarifying question the agent should have asked and didn't — becomes a permanent test case. Over time this library of real failure modes is worth more than any synthetic test suite, because it's drawn from how actual customers actually talk, which is never how you imagined they would. A new model version, a new prompt, a new integration: it has to pass the entire history of things that once broke before it ships. That library, not the model, is what makes the thing trustworthy enough to run unattended, and it's a lot of what you're really buying when you buy a vertical AI product that's already in production.
What this means if you're building one
- Test scenarios and invariants, not input/output pairs. The unit of correctness is "what must be true," not "what exact words came out."
- Push every money-critical property into a deterministic check. If a wrong value is structurally impossible, you don't need to hope the model behaved.
- Use an LLM judge to triage, never to certify. It decides what a human looks at; the human decides what's true.
- Turn every production failure into a permanent regression case. Real transcripts are the highest-value test data you will ever have, and they're free — you just have to capture them.
The demo is the easy 5%. The reason an operational AI agent can be trusted with a real customer and a real price is not that the model is clever — it's that there's a test harness underneath saying, on every change, "prove you didn't break any of the two hundred things that have broken before." Build that harness early. It's the difference between a party trick and a product.
Chris Fagan is the founder of OptiFlow Technologies, building agentic AI for builders merchants, energy suppliers and UK SMEs.
Top comments (0)