There's a specific failure that kills AI in operational software, and it isn't hallucinating a fact in a chat window. It's quoting a customer £4,200 for materials that should have been £4,700, because the model "reasoned" about pricing instead of reading it. In a builders merchant, that mistake ships a real order at a real loss. So when we built an agent that takes trade quotes and orders, the entire design question was: how do you let a language model run the conversation without ever letting it be the thing that decides a price or a stock level?
Here's the pattern we landed on. It generalises to any agent that has to act on a system of record, so it's worth writing down.
The model proposes, the system disposes
The core rule is a hard split between intent and authority. The language model is allowed to work out what the customer wants — "20 lengths of 4x2 CLS, treated, plus fixings, for collection tomorrow." That's a language problem and the model is genuinely good at it. What the model is never allowed to do is produce the number. It cannot state a price, confirm stock, or write an order from its own output.
Every one of those is a call into a typed tool that hits the real system:
-
resolve_product(description)returns actual SKUs from the catalogue, or an ambiguity the agent has to clear with the customer. -
get_price(sku, account)returns the price for that trade account's pricing tier, straight from the ERP's pricing engine. The model never computes this. -
check_stock(sku, branch)returns real availability at the real branch. -
create_order(lines, account)runs the identical validation a counter order runs, and rejects anything that fails it.
The model's job is to fill in the arguments. The system's job is to decide whether the result is legal. If the model asks for a price on a SKU that doesn't exist, it doesn't get a plausible-looking number back — it gets an error it has to handle. There is no path where a hallucinated value becomes a customer-facing figure, because the customer-facing figures don't come from the model at all.
Why not just prompt it to be careful?
Because "be careful" is not an architecture. You cannot prompt your way out of a model occasionally producing confident, wrong output — that's a property of how they work, not a bug you can instruct away. The only reliable guarantee is to make the wrong action structurally impossible rather than discouraged. If the price can only ever come from get_price, then the price can only ever be right, no matter what the model was thinking. You've moved correctness from "the model behaved" to "the system enforced," and only the second one holds up on a bad day.
This also makes the thing debuggable. When a human queries a quote, we can show exactly which SKU resolved, which pricing tier applied, and which stock figure was live at that second — because every one of those was a real call with a real result, not a stretch of generated text. Try auditing "the model said so."
The uncomfortable part: resolution is most of the work
The unglamorous truth is that resolve_product — turning "some treated 4x2 and the usual screws" into exact catalogue SKUs — is where the real engineering lives, and it's deeply specific to the business. This merchant's catalogue, this merchant's naming, this merchant's substitution rules when something's out of stock. There's no general model that knows your product tree. That work doesn't transfer between customers, which is exactly why a vertical agent that has actually done it is worth more than a general assistant that hasn't. If you want to see how that plays out on a real trade counter, we wrote up the after-hours trade counter product it powers — but the transferable lesson is the boundary, not the product.
If you're building one of these
The takeaways, stripped down:
- Split intent from authority. The model decides what's wanted; typed tools decide what's true and what's allowed.
- Never let the model emit a value that has a source of truth. Route it through a call every time, even when it's slower.
- Make bad actions impossible, not discouraged. Structure beats prompting for anything that touches money or stock.
- Expect the domain-specific resolution layer to be most of the build. That's not a detour from the AI work — it is the AI work.
The agent feels like magic to the customer because it holds a natural conversation. It's safe to run because, underneath, it can't actually make anything up.
Chris Fagan is the founder of OptiFlow Technologies, building agentic AI for builders merchants, energy suppliers and UK SMEs.
Top comments (1)
"Be careful is not an architecture" should be on a poster. We landed on the same split building agents over systems of record — the model fills arguments, the system decides legality — and the surprise was how much it simplified the prompt. Once the model physically cannot emit a price, you stop writing paragraphs of "do not guess, do not estimate, always verify" and the prompt shrinks to the actual language task. One thing worth adding: the error you hand back on a bad SKU matters a lot. A flat "not found" makes the model retry the same hallucinated SKU; a structured error — "no match, here are 3 near matches, ask the customer" — turns the failure into a clarifying question instead of a loop. Did you find create_order's rejection messages needed the same treatment, shaped for the model to recover from rather than just for a human log?