Hallucination Is a Design Problem, Not a Model Problem

Most procurement conversations about hallucination open with the wrong question: which model hallucinates least?

It is the wrong question because it treats a structural property as a vendor feature, and it produces systems that fail in the one way nobody planned for. The better question is blunter. When this system produces a confident, fluent, incorrect answer, what happens next?

If the answer is “someone notices,” describe the mechanism. If the answer is “it flows into the next step,” you do not have a model problem. You have a design problem, and no version number is going to fix it.

Better Models Narrow the Gap. They Do Not Close It.

Be precise here, because the careless version of this argument is wrong. Part of the cause really does sit inside the model.

OpenAI’s own research on the subject argues that models hallucinate because standard training and evaluation reward guessing over admitting uncertainty. On an accuracy scored benchmark, a confident guess has some chance of scoring. Saying “I do not know” guarantees zero. The incentive points one way, and models are trained into it.

Two findings from that work matter for anyone building on top. First, some questions cannot be answered from patterns in text at all, so accuracy will never reach 100 percent. Second, and more usefully: hallucination is not therefore inevitable, because a model can abstain when it is uncertain.

That is a fix to how models are trained and scored. It is real work and it will help. But it is not a lever you hold, and it will not arrive as a release note that ends the problem. What you hold is the system around the model: what it is given, what it is permitted to say, and what gets checked before anyone acts on it.

Grounding Moves the Failure. It Does Not Remove It.

Retrieval is the first mitigation most teams reach for, and usually the right one. Hand the model the actual document, and it no longer has to reconstruct a fact from training data.

It also creates a failure mode that is harder to catch than the one it replaced.

An ungrounded model that invents a figure often produces something a knowledgeable reader can smell. A grounded model that retrieves the wrong document produces a confident answer, with a citation, drawn from a genuine source that happens to be the wrong one. The citation does not make the answer correct. It makes it more persuasive, and it quietly changes the reviewer’s job from reading critically to checking that a link exists.

Three consequences for design. Measure retrieval quality separately from answer quality, because an answer can be perfectly faithful to a wrongly retrieved passage. Show the retrieved passage itself, not just a link to its source, so a reviewer can see what the answer was actually built from. And decide explicitly what happens when retrieval finds nothing relevant, because the default behaviour of most implementations is to answer anyway from general knowledge, silently, in exactly the same confident tone.

Abstention Is the Cheapest Mitigation and the Least Built

A system that can say “I am not confident here” converts expensive silent errors into cheap routed exceptions. That is most of the value, and it is rarely implemented.

The reason is structural, and it mirrors the benchmark problem precisely. Internal evaluations usually measure how often the system answered correctly. A system that abstains scores worse on that measure than one that guesses and occasionally gets lucky. Teams then tune toward confidence, reproducing inside their own organisation the exact incentive that causes the problem upstream.

Decide before building what an acceptable abstention rate is, and measure confident errors separately from abstentions instead of pooling both into one accuracy number. A system that declines 15 percent of cases and is dependable on the rest is usually worth more than one that answers everything and is wrong 5 percent of the time without telling you which 5 percent.

Verify Against Something Deterministic

Where a claim can be checked, check it in code before a person ever sees it.

If the model states a balance, read the balance. If it names a contract clause, confirm the clause exists. If it computes a figure, recompute it. This is unglamorous, and it is the most effective pattern available, because it converts a probabilistic claim into a deterministic one at the point where it matters.

The limitation is real and worth stating: it only works where ground truth is reachable. For summarisation, drafting and judgement, there is nothing to check against. Those are exactly the applications where you should be most conservative about what the output is permitted to trigger.

Where Probabilistic Output Must Not Reach At All

Some decisions should not sit downstream of a model, however well designed the system is.

Anything irreversible, anything that moves money or assets, and anything carrying a compliance obligation belongs in deterministic code, with the model confined to preparing input that a person or a rule engine then acts on. In tokenization work, this is concrete rather than abstract: investor eligibility and transfer restrictions belong inside the token standard, enforced at transfer time under something like ERC-3643, with KYC and AML checks wired into identity rather than inferred by a model. A prompt that describes a compliance rule is not a control. It cannot be tested exhaustively, and it cannot be explained in a dispute.

The same reasoning governs keys. If an agent can trigger a transaction, it holds scoped keys under threshold approval and nothing more, and the contract logic it can reach has been through independent audit before any value moves. Key handling causes more incidents than contract logic does, and an automated caller is not an exception to that.

When the Ledger Is the Wrong Tool Here

We build blockchain systems, so take this as the section worth reading twice.

Writing model output to an immutable ledger is a common instinct and usually a mistake. Immutability is actively hostile to a system with a known error rate: you have permanently recorded a wrong answer, and the only remedy is appending a correction so that both now exist forever. For an auditable trail of what an AI system did, an append-only signed log in your own infrastructure is simpler, cheaper, reversible where it needs to be, and entirely sufficient.

The ledger earns its place at a different point in the pipeline. Verify deterministically first, then record the verified fact. What goes on chain should be the outcome a rule confirmed, never the claim a model made.

Scope Honestly, Then Own It

Much of what arrives labelled as a hallucination problem is a scoping problem. The system was asked something for which no single correct answer exists, and it produced one anyway. Three narrow calls with validation between them usually beat one broad call with a long prompt asking the model to be careful.

And for a meaningful share of these requests, the honest answer is a rule engine, better search, or conventional automation, with no language model at all. Cheaper, exactly correct, fully testable. A partner who never raises that option is telling you something about their incentives.

Whatever you build, keep it: the prompts, the evaluation suite, the deterministic controls, the integrations, the keys. Those artefacts are the system. If they live with a vendor, your ability to change model, change provider, or prove what your controls do leaves with them.

Five questions worth asking before anyone writes code:

What does the system do when it does not know?
How does a wrong output get caught, by whom, within what time?
What is verified deterministically before anything acts on it?
Are confident errors measured separately from abstentions?
What is the blast radius of one wrong answer, and is anything downstream irreversible?

A system that cannot answer these is not made safer by a better model. It is only less legible about how it will fail.

RWaltz is a technology partner building blockchain, AI, and enterprise software, including smart contracts, tokenization platforms, and custom integrations. Our approach is security-first and build-to-own: honest scoping, independent audits, and client-owned IP and keys, so what we deliver belongs to you.

📖 Read the full blog: https://www.rwaltz.com/blogs/hallucination-is-a-design-problem-not-a-model-problem

Connect with RWaltz:

LinkedIn: https://www.linkedin.com/company/rwaltzsoftware
X (Twitter): https://twitter.com/rwaltzsoftware
Facebook: https://www.facebook.com/RWaltz-Software-PvtLtd-255590135349493
Telegram: https://t.me/RWaltzCrypto
GitHub: https://github.com/rwaltzsoftware
Clutch: https://clutch.co/profile/rwaltz-software
Website: https://www.rwaltz.com