Ask a chatbot a question about your own documentation and it answers confidently, fluently, and sometimes completely wrong. The model is not broken. It is doing exactly what it was trained to do: produce plausible text. Retrieval-augmented generation, or RAG, exists to change the job from "produce plausible text" to "answer from these documents." When RAG works, hallucinations drop sharply. When it fails, it usually fails in one of a handful of predictable places, and each one is fixable.
What RAG actually is
A RAG pipeline has four stages, and the model is only the last one. First, your documents are split into chunks. Second, each chunk is converted into a vector embedding, a list of numbers that captures its meaning. Third, at query time, the user's question is embedded the same way and the nearest chunks are retrieved. Fourth, those chunks are pasted into the model's prompt and the model answers from them.
Documents -> chunks -> embeddings -> vector store
Question -> embedding -> top-k chunks -> prompt + model -> answer
Most teams obsess over the model and neglect the first three stages. In practice, retrieval quality decides answer quality. A model given the right three paragraphs answers correctly. A model given the wrong three paragraphs answers confidently and wrong, because nothing in its prompt tells it otherwise.
Failure 1: chunks that destroy meaning
The most common RAG bug is chunking by fixed size with no respect for structure. A 500-token window that cuts a table in half, splits a function from its explanation, or separates a warning from the procedure it warns about produces chunks that are individually meaningless. Retrieval then returns fragments, and the model stitches fragments into fiction.
Fix it by chunking on structure: split by headings, by function, by paragraph, and keep small overlaps between adjacent chunks so context survives the boundary. Overlap of 10 to 20 percent is a reasonable starting point. Then read retrieved chunks for real queries and judge them by eye: would a human given only these chunks answer correctly? If not, the chunking is wrong, not the model.
Failure 2: retrieval that returns the wrong neighbors
Vector similarity is not relevance. A question about "refunds for annual plans" can retrieve a chunk about "refunds for monthly plans" because the words are close, while the chunk about annual plans sits just outside the top-k cutoff. Pure vector search also misses exact terms: a product code, an error string, a person's name.
The standard fix is hybrid retrieval: combine vector search with keyword search, typically BM25, and merge the results. The vector search catches paraphrases; the keyword search catches exact terms. Reranking the merged set with a small cross-encoder model before handing it to the generator is the second upgrade, and it is usually the highest-return change in the whole pipeline.
# Hybrid retrieval in sketch form: keyword hits plus vector hits, reranked.
keyword_hits = bm25_search(query, top_n=20)
vector_hits = vector_search(embed(query), top_n=20)
candidates = dedupe(keyword_hits + vector_hits)
ranked = cross_encoder_rerank(query, candidates)[:5]
answer = llm.generate(prompt_with(ranked))
Failure 3: prompts that invite invention
Even with perfect retrieval, a prompt that says "answer the question" invites the model to fill gaps from its training data. The fix is boring and effective: instruct the model to answer only from the provided context, and to say it does not know when the context does not contain the answer. "I don't know" is a feature, not a failure, and teams that treat it as one will keep shipping hallucinations.
Add citations to the output: require the model to quote or reference the chunk each claim comes from. Citations do two jobs at once. They let users verify answers, and they make hallucination visible during evaluation, because an uncited claim is a claim the pipeline cannot defend.
Failure 4: never measuring anything
The deepest RAG failure is building the pipeline and never evaluating it. Without a test set of real questions with known-good answers, every change is a guess. Build a set of 50 to 100 questions your users actually ask, with the correct answer and the source document noted. Then measure: how often does retrieval return the right chunk, and how often does the final answer match? Most teams discover that retrieval, not generation, is the bottleneck, and that a week of chunking and reranking work beats a month of model tuning.
Evaluation also catches data drift. Documents change, products rename, policies update. A chunk store built from last quarter's documentation answers last quarter's questions. Rebuild embeddings when source documents change, version the index, and re-run the test set after every rebuild. A RAG system without a refresh process is a snapshot pretending to be a service.
The honest summary
Hallucination in a RAG system is usually a retrieval failure wearing a model's clothes. Chunk on structure, retrieve with hybrid search, rerank before generating, prompt for grounded answers with citations, and measure against real questions. None of that is glamorous, and all of it is the difference between a chatbot that helps and one that confidently misleads. The model is the last 10 percent of the system. The pipeline is the other 90, and it is where the work belongs.
When RAG is the wrong tool
Not every question-answering problem is a RAG problem, and forcing it causes a different class of failure. If the answer requires computation, aggregation, or live data, retrieval cannot help. "How many open tickets do we have?" is a database query, not a document lookup, and a RAG pipeline that retrieves a stale wiki page about ticket counts will answer confidently and wrong. Route those questions to tools: let the model call a function that queries the live system, and reserve RAG for knowledge questions with stable, document-backed answers.
The same logic applies to questions with a single authoritative answer that changes often. Prices, quotas, and policy thresholds belong in a structured store queried at answer time, not in embeddings rebuilt nightly. A useful rule: if the answer would change between index rebuilds, do not put it in the index. Put a pointer in the index and fetch the value live.
Finally, respect the case where the honest answer is that no document covers the question. A RAG system that always answers, even when retrieval returns nothing relevant, is a hallucination machine with good branding. Build the abstention path deliberately: when the best retrieved chunk scores below a threshold, say so, and offer the closest related topic or a human contact. Users forgive "I don't know" far faster than they forgive a wrong answer delivered with confidence.
Start with one pipeline, not ten
If your team is new to RAG, resist the urge to build one pipeline per department. Build one pipeline for one well-understood document set, with a test set of real questions and a named owner for the index. Run it for a month, fix the chunking problems you find, add hybrid retrieval and reranking, and only then expand. A single pipeline that answers correctly teaches the organization what good looks like. Ten pipelines built at once teach it that RAG is unreliable, and that reputation is hard to reverse. Quality first, coverage second, always.
Top comments (0)