I want to describe one architectural rule that has mattered more than any model choice on the enterprise AI systems we have shipped. It comes from building supplier intelligence platforms, but it generalises to anything where a number produced by your system will eventually be disputed by someone with a commercial interest in the answer.
The rule: models handle language, code handles numbers and state.
It sounds obvious written down. It is violated constantly, usually for good-seeming reasons, and the failure surfaces about eighteen months later when someone challenges an output and nobody can reconstruct how it was produced.
The concrete version
Take supplier performance scoring. The system needs to produce, per supplier, an on-time delivery rate, a quality index, a contractual compliance flag, and a composite risk score.
The tempting implementation is to give a model the relevant context and ask for the scores. It works in a demo. It is wrong.
Here is the split that actually holds up:
Model territory:
- Resolving that
ACME Industrial Ltd,Acme Industrial, andACME IND (Shanghai)are the same legal entity (or deciding, with reasoning, that they are not) - Extracting the payment terms, liability cap, and service credit thresholds from a 40-page contract PDF
- Classifying a free-text quality incident into a defect taxonomy
- Summarising why a set of external signals suggests elevated financial risk
- Answering a category manager's natural language question by translating it into a query
Code territory:
- Computing the on-time delivery rate from receipt records
- Evaluating whether a service level threshold was breached
- Summing committed spend
- Calculating the weighted composite score
- Deciding whether an alert crosses a severity boundary
Why the boundary is not negotiable
Three reasons, in increasing order of how expensive they are to learn the hard way.
Reproducibility. A deterministic calculation returns the same answer today and in nine months. A model call does not, and it definitely does not after the provider updates the model underneath you. When someone asks why this supplier's score changed, "the model said so" ends the conversation badly.
Explainability that survives scrutiny. A computed score decomposes into an arithmetic chain: these 340 receipts, of which 31 were late against the promised date, against a threshold defined in clause 7.2. A model-generated score decomposes into a plausible-sounding paragraph. Only one of those survives a dispute where money is attached.
The failure mode. When code is wrong, it is wrong loudly and consistently — you find it in testing. When a model is wrong about arithmetic, it is wrong quietly, occasionally, and confidently, in a way that passes review because the answer looks reasonable. That is the worst possible failure profile for a system people rely on.
Crossing the boundary safely
Model outputs frequently need to feed calculations. A contract's SLA threshold, extracted by a model, is an input to a deterministic breach check. That is fine, provided the crossing is explicit.
The pattern we use:
- Extract to a strict schema. Not free text. A typed structure with defined fields and value constraints.
- Attach a citation. Every extracted field carries a pointer to its source — document ID, page, clause. Not for decoration; for the dispute.
- Score the confidence. The extractor returns a confidence per field.
- Route by threshold. Above threshold, auto-accept. Below, queue for human review. Never silently default a missing value — a defaulted liability cap is a landmine.
- Persist as a reviewed fact. Once accepted, the value is a row in your governed model with provenance and a timestamp. Downstream code reads the row, never the model.
That last step is the one teams skip. If your calculation calls the model at read time, you have a non-reproducible system regardless of how good your prompts are.
The same rule for agents
We build monitoring agents that watch external signals for supplier distress — filings, adverse media, certification registries, sanctions lists. The equivalent boundary applies to actions rather than numbers.
Agents propose. Humans dispose.
The agent detects a signal, summarises it with sources, assigns a severity through deterministic rules, and routes it to the person who owns that relationship. It does not open a corrective action, does not email the supplier, does not change a qualification status.
This is not excessive caution. A false positive that triggers an automated corrective action notice damages a commercial relationship in a way that costs more than the entire monitoring capability saved. Reserve automated state changes for rules that are unambiguous and low-stakes.
Where entity resolution fits
One nuance worth stating: entity resolution is probabilistic work with permanent consequences, so it gets extra constraints.
Tune for precision over recall. A wrong merge — combining two genuinely distinct suppliers — corrupts every downstream metric for both and is hard to detect. A missed merge leaves a duplicate, which is visible and fixable. Optimise accordingly.
Make every automated merge reversible, with the evidence retained. You will need to unwind some of them.
Queue ambiguous pairs for a human rather than forcing a decision. The model explaining its reasoning is what makes bulk human approval practical — reviewers can scan fifty justifications far faster than fifty raw record pairs.
The short version
If you are building an enterprise system with an LLM in it, write down which side of the line each component sits on before you write the code. Retrofitting auditability into a system that let a model compute its outputs is close to a rewrite, and you will discover the need for it at the worst possible moment.
Frequently Asked Questions
Why can't an LLM compute business metrics?
It can produce numbers, but not reproducibly or explainably. The same input can yield different outputs across calls and across model versions, and the result decomposes into prose rather than an arithmetic chain. When a metric carries commercial consequence, both properties are disqualifying.
How do you use model output in a calculation then?
Persist it first. Extract to a strict schema with a citation and a confidence score, route low-confidence fields to human review, and store accepted values as reviewed facts with provenance. Downstream code reads the stored row, never calls the model at read time.
Should AI agents be allowed to take actions autonomously?
Only for unambiguous, low-stakes rules. For anything touching a commercial relationship, use propose-and-notify: the agent detects, summarises with sources, assigns severity, and routes to a human owner who decides.
What confidence threshold should extraction use?
It depends on the cost of an error in that field, not on a global number. A payment term that is wrong is an inconvenience; a liability cap that is wrong is a legal exposure. Set thresholds per field and never silently default a missing value.
Why precision over recall in entity resolution?
An incorrect merge silently corrupts every metric for both entities and is difficult to detect afterwards. A missed merge leaves a visible duplicate that someone will eventually notice and fix. The asymmetry in damage justifies the asymmetry in tuning.
This is drawn from a longer piece on building supplier intelligence platforms — full architecture, implementation sequence and cost model here: SRM Software in 2026: A Build vs Buy Guide for CTOs.
More on how we approach LLM integration and agentic workflow development.
Top comments (0)