Why Your RAG Chatbot Still Hallucinates: 5 Design Mistakes (and How to Fix Them)

You built a RAG chatbot. The documents are indexed, the vector database is humming, and the demo went well. Then a user asks about the refund policy, and the bot confidently quotes a rule that was retired last
year.

RAG is supposed to reduce hallucinations. It does, but only when retrieval is good. The pipeline has two steps: find the relevant text, then generate an answer from it. If the text you retrieved is wrong, incomplete or outdated, the model will happily write a smooth, confident answer on top of it.

In most projects the problem isn't the model. It's the design upstream of it. Here are five mistakes that show
up again and again.

1. Chunking that cuts the meaning in half

Splitting every document into fixed 500-token pieces is easy. It also slices tables in half, separates a step
from its warning, and cuts a definition away from the term it defines. When the chunk doesn't make sense
alone, the answer built on it won't either.
The fix:
• Split on structure first: headings, paragraphs, list items
• Keep a small overlap between neighbouring chunks
• Attach the document title and section heading to every chunk
• Keep tables and step lists together
A quick test: read 20 random chunks. Would you understand each one without the rest of the document? If
not, your model won't either.

2. Relying on vector search alone

Embeddings are great at meaning and weak at exact matches. Product codes, error IDs, version numbers and names often get lost. A user searching for "error E4021" doesn't want something vaguely about errors.

The fix: use hybrid search. Combine keyword search (BM25 or similar) with vector search, then pass the top 20 to 50 candidates through a re-ranker that picks the best 3 to 5 for the prompt. A good rule: retrieve wide,then re-rank narrow.

3. Stale and conflicting data
Documents change. The index usually doesn't. Worse, the old and new versions of a policy can both sit in the index, and the retriever has no idea which one is current.

The fix:
• Store metadata with every chunk: source, version and last-updated date
• Re-index when a document changes, or on a regular schedule
ScaleDojo | Blog Articles | Draft for publishing Page 2
• Remove or filter out superseded versions
• Show the source and its date in the answer so users can judge

4. No evaluation, just vibes
"It seems to work" is not a metric. Without measurement you can't tell whether a new chunk size helped or hurt.

The fix:
build a small test set. Start with 30 to 50 real questions, each with the correct answer and the
document that contains it. Then measure two things separately:
• Retrieval: did the right chunk come back in the top results?
• Generation: is the final answer actually supported by the retrieved text?
Re-run the set after every change. It takes the guesswork out of tuning.

5. No way to say "I don't know"
Language models try to be helpful. If the retrieved context doesn't contain the answer, many will fill the gap
with something plausible. That is the hallucination users notice.

The fix:
• Instruct the model to answer only from the provided context
• Give it an explicit way out when the answer isn't there
• Require a citation for every claim
• Set a retrieval-score threshold; below it, decline or ask a clarifying question
• Log the declined questions, because they show where your documents have gaps

A starting prompt:

Answer the question using only the context below.
If the context does not contain the answer, reply:
"I couldn't find this in the documentation."
Cite the source title for every claim.
Context:
{retrieved_chunks}
Question: {question}

A debugging order that saves time
When you see a wrong answer, ask these in order:

1. Was the right information in the index at all?
2. Was it retrieved?
3. Was it ranked high enough to make it into the prompt?
4. Did the model actually use it? Most bugs show up in the first three, long before you need to blame the model. For a deeper look at the full architecture (ingestion, embeddings, vector databases and retrieval), the RAG and Vector Databases guide on ScaleDojo walks through each component.

Comments

[Sophia Bennett]

Great 👍

[Ritesh Gupta]

Insightful