I have tested a lot of website chatbots over the past two years, and the failure I see most often is confidence. An assistant quotes a price nobody agreed to. It promises a feature that does not exist. It explains a refund policy the business never wrote down. Every one of these answers arrives in the same calm, helpful tone as the correct ones, which is exactly what makes them dangerous.
We have been building a configurable assistant platform at JarvisBitz for a while now, and the question I get asked most about it is how we stop it inventing answers. This article is my answer. It is an engineering answer rather than a product pitch, because the fix lives in the architecture, and it applies whether you build your own assistant or buy one.
Where invented answers actually come from
A language model answers every question it is given. That is what it was trained to do, and it is very good at it. Put it behind a chat window, give it your web pages as context, and it will answer the questions your pages cover. It will also answer the questions your pages do not cover, using whatever it absorbed during training, in the same confident voice.
Retrieval augmented generation, the technique most teams reach for, narrows the problem without removing it. Retrieval finds the passages from your material that look most similar to the question, and the model writes an answer from them. The trouble starts when retrieval comes back thin. The passages are only loosely related, but the model still has to say something, so it fills the gap from general knowledge. The output looks grounded. It has the shape of an answer drawn from your documents, and part of it is fiction.
I have seen this with prices, with features that did not exist, and with policies the business never wrote down. In each case the retrieval step had worked exactly as designed, which is the uncomfortable part. The design itself was the problem.
Rule one: not enough evidence is an outcome, and it needs its own path
The most important decision we made was to treat "we do not have a reliable answer" as a real result with its own branch in the system, instead of something the model should smooth over.
In practice it works like this. After retrieval and before any text is generated, a separate check scores the evidence. How well do the retrieved passages cover the question. Whether they come from approved sources. Whether there are enough of them and whether they agree with each other. If that score falls under a threshold the business has set, the model is never asked to write an answer. The conversation takes a different route: the assistant says plainly that it does not have that information, offers to connect the visitor with a person, and, if the visitor is willing, collects a way to follow up.
It sounds like a small change, and it changes almost everything downstream, because once the model only ever generates from evidence that passed a check, it is no longer in the position of having to guess. The invented price problem mostly disappears on its own.
The threshold is a business decision rather than an engineering one. A law firm wants it strict. A company selling gift boxes can afford to be looser. So we expose it as a setting, and we let the business watch what happens at each level during testing, which I will come back to.
Give "I do not know" its own path through the system. Most of the other rules follow from that one.
Rule two: the assistant may only know what the business gave it
The second rule is about the boundary of knowledge. The assistant answers from approved sources and from nothing else. The website, the product documentation, the policy PDFs, the notes someone in sales wrote down. Whatever the business attaches is the whole world as far as the assistant is concerned.
This is easy to say and surprisingly easy to get wrong. The most common leak I see is a system prompt that tells the model to use its general knowledge if the documents do not cover the question. Someone added it because the assistant felt unhelpful in testing, and it is the single instruction most responsible for confident nonsense in production. We do not allow it. If the business wants the assistant to answer a question, the business adds a source that answers it.
The second leak is subtler. The model is allowed to reason from the documents, and reasoning drifts. Asked whether the product integrates with a tool the documents never mention, a model reading a page that lists eight integrations will sometimes conclude that a ninth is probably supported as well. The evidence check catches most of this, because the passages do not actually contain the claim. Requiring the model to cite the passage behind every factual statement catches most of the rest, since a claim with nothing to cite stands out immediately.
Rule three: business rules belong in code, where a visitor cannot argue with them
Grounding governs what the assistant knows. There is a separate question of what it is allowed to say, and this is where I see a second family of failures that have nothing to do with hallucination.
Whether this assistant may discuss pricing at all. Whether the support assistant may suggest an upgrade, or that belongs to the sales assistant alone. Which topics are off limits. At what point in a conversation it should stop and bring a person in. These are policies. Most teams put them in the system prompt, and I understand why, because it takes two minutes. But a prompt instruction is advice the model can be talked out of by a determined visitor, and visitors are more determined than you expect. A rule checked in code on every turn cannot be talked out of anything.
So in our architecture the persona layer has two parts. Tone, which the prompt handles perfectly well. And policy, which ordinary code checks before and after generation. If the policy says no pricing, the draft answer is checked for pricing and stopped before it reaches the visitor, who gets a polite redirect and, where it makes sense, an offer to talk to someone who can quote.
Rule four: the hard cases go to a person, and that is a feature
An assistant that hands off to a human is often described as one that failed. I think that view is wrong, and it pushes teams into making the assistant answer things it should not.
A visitor who is angry, a question that touches a contract, a request for a discount, a medical or legal detail, a question the knowledge base cannot support: these should reach a person, and the assistant's job is to get them there gracefully and with context. The handoff carries the conversation so far, the intent it detected, and whatever contact details the visitor offered, so the person picks it up without asking anyone to start again.
Designed this way, the handoff is where the highest value conversations end up. It deserves to be built properly rather than treated as the apology screen.
The part most teams skip: learning from the questions you could not answer
Every time the evidence check fails, we record the question. Over a month, that log becomes one of the most useful documents the business owns.
If thirty visitors ask whether you work with a particular accounting system and your site never mentions it, that is thirty pieces of market research delivered for free. It tells the business what to write next, what belongs in the FAQ, what sales keeps hearing on calls, and sometimes what to build. An assistant that had invented a plausible answer would have destroyed that signal. The visitor would have left, satisfied or misled, and nobody would have learned that the question was ever asked.
We surface this as a report: unanswered questions grouped by theme and ranked by how often they came up. I expect it to be the feature businesses value most once they have lived with it, and it only exists because we refused to let the model guess.
Test it before it meets a customer
None of the above is worth much if the business cannot see what the assistant is doing. Our test environment shows the decision behind every reply: which intent matched and with what confidence, which sources were retrieved, whether the evidence threshold passed, which policy rule fired, which lead detail was being collected.
This is how a non technical owner tunes the system. A wrong answer with no trace is a mystery that needs an engineer. A wrong answer with a trace usually shows that a document was missing or a threshold was too loose, and the owner fixes that directly. It is also how they choose the evidence threshold with some confidence, by watching real questions flow through at different settings before anything goes live.
What this means if you are buying rather than building
If you are evaluating an assistant for your website, from us or from anyone else, the rules above turn into a short checklist. Ask the vendor:
- What happens when the knowledge base does not contain the answer. Ask to see it, using a question you know is not covered.
- Whether you can restrict the assistant to your sources only, with no fallback to general knowledge.
- Whether rules like "never discuss pricing" are enforced in code or only written into a prompt.
- Whether you can see why the assistant gave a particular answer.
- Whether you get a report of the questions it could not answer.
A vendor who cannot answer those clearly is selling you the chat window, and the chat window is the easy part. If you already run an assistant and want a second opinion on it, our free AI audit covers exactly these checks.
Where this leaves us
We describe what we are building as a platform for domain specific assistants, and it is still a prototype, so I am not going to claim results we have not measured. What I will claim is that this architecture is the only approach I have seen hold up once real visitors arrive. The model handles the execution: understanding the question, finding the answer, collecting the details. The business keeps control of what the assistant knows, what it may say, and when it must stop. If you take one thing from this piece, make it the first rule. Give "I do not know" its own path, and most of the rest follows.
Suresh B is the Founder & CEO of JarvisBitz Tech, an AI and software engineering company helping businesses build and scale production AI systems and digital products.
Top comments (1)
Rule one is the load-bearing piece. Once "not enough evidence" is a first-class branch, the model is no longer asked to paper over a thin retrieval set.
The bypass I still see in production is a system prompt that tells the assistant to fall back on general knowledge when the docs are quiet. Someone adds it after a stiff demo, and the invented price problem returns under a friendlier tone. I would treat that instruction as a defect, not as a helpfulness tweak. If the business wants the question answered, it adds a source. If it does not add a source, the refuse path runs.
I would also score agreement, not only similarity. Two passages that both mention shipping but disagree on the window should fail the gate the same way an empty set does. Then the unanswered-question log you describe becomes the product backlog for the knowledge base, not a graveyard of awkward chat transcripts.