How to reduce hallucinations in a business AI assistant (and why "never make things up" in the prompt is not enough)

Adapted from an article originally published in French on closiqode.com.

You cannot remove every hallucination from a generative AI assistant. What you can do is make them rarer, and make sure an unverified answer never triggers a consequence that matters.

In practice, that means combining six things: a narrow scope, validated sources, an explicit refusal, hard tests, citations and human control.

The French data protection authority (CNIL) puts it simply: generative models are not knowledge bases. Their probabilistic logic can produce a result that is wrong but plausible. The real danger is not the visible error. It is the convincing answer nobody thinks of checking.

1. Narrow the scope

The more an assistant tries to answer everything, the further it drifts from its job.

Write down the allowed topics, the forbidden topics, who the users are and what the assistant is allowed to do. "Answer HR questions" is too broad. "Find the validated administrative steps for a leave request" is testable.

2. Ground it in verified sources

For a business assistant, the document base matters more than the model:

• documents that are up to date,
• identified versions,
• content that someone has actually validated.

A RAG system connects the model to a search over a separate knowledge base. It can anchor answers in your own information, but a bad retrieval, an obsolete document or a wrong interpretation are still possible. Give every important source an owner and a review date.

3. Make "I don't know" a feature

The assistants I build are designed to refuse explicitly when the knowledge base does not support a reliable answer.

A good refusal is useful. It says what is missing, suggests which source to provide, or hands over to a person. A vague "I'm not sure" is not enough if the user can still take the answer at face value.

A minimal instruction looks like this, but remember that the instruction alone guarantees nothing: the tests in step 4 are what prove it works.

Answer only from the provided excerpts.
If the excerpts do not contain the answer, say so explicitly,
list what information is missing, and suggest who to ask.
Quote the source document and version for every factual claim.

4. Test the hard questions

Your test set should cover:

• ambiguous questions,
• missing information,
• contradictory sources,
• unusual wording,
• edge cases.

A system tested only on obvious cases gives a false sense of reliability.

A minimal protocol

1. Collect real questions, each with the expected answer and the expected source.
2. Add impossible, out-of-scope or deliberately misleading questions.
3. Test several phrasings of every critical question.
4. Score retrieval relevance and answer faithfulness to the source separately.
5. Replay the whole test set after every change of model, prompt or corpus.

A test case can be as simple as:

{
  "question": "How many days in advance must a leave request be submitted?",
  "expected_source": "hr-procedures-v3.pdf, section 2",
  "expected_behavior": "answer_with_citation"
}

with a sibling case where expected_behavior is refuse because the corpus does not contain the answer.

5. Do not automate sensitive decisions blindly

The assistant can prepare, search, summarize or route. That does not mean every decision should happen without a human.

Match the control to the consequence:

Possible consequence	Example	Recommended control
Low and reversible	Drafting an internal note	Simple review
Moderate	Answering a customer about a procedure	Visible source and an escalation path
High	Committing spend or acting on a person	Mandatory human validation and an audit trail

6. Keep the sources visible

When the stakes justify it, let users check where the information came from.

The citation must point to the document that was actually used, with its version if needed. And check that the cited passage really supports the generated sentence: a reference being present is not proof that it is the right one.

7. Monitor errors in production

Launch is not the end of the work. Add a way to flag an answer, keep what you need for diagnosis while respecting data protection rules, and turn every incident into a new test case.

Track useful indicators: answers without a source, appropriate refusals, human corrections, out-of-corpus questions and error severity. A satisfaction score alone does not measure reliability.

Diagnose before you fix

Not every error comes from the model. To fix an assistant for good, find the step that failed:

Observed error	Diagnostic question	Useful check
The right source was not retrieved	Was the document present, current and accessible to this user?	Test retrieval on its own; check permissions, metadata and chunking
The source is right but the answer distorts it	Is the generated sentence really supported by the cited passage?	Compare claim and excerpt, limit rewriting, force a refusal when evidence is weak
The answer is fine but the action would be risky	What happens if it is misread?	Block automatic action, show the source, require proportionate human validation

This keeps you from "fixing" every incident by adding one more line to the prompt. Each error becomes a reproducible test.

What does not work on its own

• Writing "never make things up" in the prompt with no other control.
• Connecting every document without managing versions and permissions.
• Testing only on prepared demos.
• Confusing a fluent answer with a proven one.
• Hiding the limits from users to make the tool look more impressive.

FAQ

Does RAG eliminate hallucinations?
No. It provides external, updatable context, which can reduce some errors. Retrieval and generation still have to be evaluated.

Is a more powerful model enough?
Not to guarantee the reliability of a business process. Sources, scope, tests and controls are still needed.

Should you put a warning everywhere?
A warning does not replace system design. Users should understand the limits, but sensitive decisions mainly need to be blocked or validated.

A good assistant does not always answer

It is the one that also knows when not to.

---

I am Fabien Kost, an independent developer in France (ClosiQode). I build business AI assistants that answer from validated documents and say so when they cannot find the answer. More on how I design them (in French).

Sources (in French): CNIL, questions and answers on using generative AI systems and first guidance on deploying generative AI.