cavitnation

Posted on Sep 12

How I stop an LLM from hallucinating in production (RAG + entity-match + MCP)

#ai #webdev #llm #machinelearning

How do you make an LLM answer questions about real-world entities — companies, people, records — without it confidently making things up?

I build Verivello, a live AI agent that answers questions about any UK company from official registers (Companies House, HM Land Registry, FCA, The Gazette, sanctions lists). In that domain, a wrong director or ownership figure is worse than no answer at all. So "don't hallucinate" isn't a nice-to-have — it's the whole product.

Here's the system design I use to keep an LLM grounded in production. No prompt-magic, no fine-tuning. Just architecture.

The problem

LLMs are fluent, and fluency reads as confidence. Ask a model "who is the director of Acme Ltd?" and it will happily produce a plausible name — whether or not it actually knows. For casual use that's fine. For anything a business acts on, it's a liability.

The usual first instinct is "add RAG." Retrieval-augmented generation helps, but on its own it isn't enough: retrieval can pull the wrong record, and the model can still paraphrase it into something subtly false. Grounding is a pipeline, not a single step.

The system at a glance

flowchart TD
    U[User question] --> C[Classifier: intent + entity]
    C --> R[Engine router]
    R --> T[Tool / function-calling loop]
    T --> M[MCP tool servers]
    M --> S1[Companies House]
    M --> S2[Land Registry]
    M --> S3[FCA / ICO]
    M --> S4[Gazette / Sanctions]
    S1 & S2 & S3 & S4 --> V[Grounding layer:<br/>verbatim output + entity-match]
    V --> A[Streamed, source-cited answer]
    A --> U

Two rules in that grounding layer do most of the anti-hallucination work.

Rule 1 — Verbatim tool output, not model memory

The model is never allowed to recall a fact about an entity. Every fact in an answer must come from a tool call whose raw output is passed through unchanged. The model's job is to explain and cite retrieved records — not to know them.

Concretely:

Tools return structured records (JSON), and that JSON is placed in context verbatim.
The system prompt instructs the model to answer only from provided records and to cite them.
If a fact isn't in the retrieved records, the model must not supply it.

This one rule kills the most common failure mode: the model "filling in" a field from training data.

Rule 2 — Entity-match verification

This is the rule most people miss. Name-search APIs happily return a record for a query — often the wrong one. Search "Smith Consulting" and you might get any of dozens. If you feed that straight into context, you've grounded the model in the wrong company. Now it's confidently wrong with a citation, which is worse.

So before any record is allowed into the grounded context, it's re-verified against the entity the user actually asked about. Illustrative shape (not production code):

// Reject records that don't actually match the entity the user asked about.
function verifyMatch(query, record) {
  // Strongest signal: exact identifier match.
  if (query.companyNumber && record.companyNumber) {
    return query.companyNumber === record.companyNumber;
  }
  // Otherwise: normalised name + a corroborating signal.
  const a = normalise(record.name);
  const b = normalise(query.name);
  return a === b || (similarity(a, b) >= 0.92 && sameIncorporationYear(query, record));
}

// Only verified records are allowed into the grounded context.
const grounded = toolResults.filter(r => verifyMatch(query, r));

Mismatches are dropped, not shown. The model only ever sees records that provably belong to the queried entity.

The golden rule — fail closed

If nothing verifies, the agent says so. No verified record → an honest "I don't know," never a plausible guess.

This is a product decision as much as an engineering one. In due diligence, a confident wrong answer is the expensive failure mode — far more costly than an honest gap. So the whole pipeline is built to fail closed: when in doubt, refuse.

Why MCP for the tool layer

Each data source is exposed as a typed Model Context Protocol (MCP) tool server. That gives a few real benefits:

Isolation & testability — each source is an independent server with a typed schema, unit-testable on its own.
Reuse — the same tool servers work across clients (Claude Code, Claude Desktop, the product itself).
A clean boundary — the model requests what it needs; the tool layer owns how it's fetched and verified.

I open-sourced a small, keyless example of this pattern — an MCP server exposing UK public-data tools, with unit tests + CI — here: github.com/cavitnation/mcp-uk-tools.

Takeaways

If you're wiring an LLM up to real data, grounding is a pipeline:

Verbatim tool output — the model explains and cites; it never recalls.
Entity-match verification — prove each record belongs to the queried entity before using it.
Fail closed — no verified record, no answer.
MCP tool servers — a clean, testable, reusable boundary between model and data.

That's the difference between a demo and something a business can actually trust to act.

I wrote up the full architecture (with the diagram, no product code) here: github.com/cavitnation/verivello-architecture. I'm an AI-native full-stack engineer — happy to talk shop in the comments.

DEV Community