A client called us last month with a simple complaint: "Our support agent confidently quotes the wrong refund policy." The model was fine. The prompt was fine. The problem was three layers down, in the part nobody demos: retrieval. The agent was pulling the wrong chunk of text and then reasoning beautifully over the wrong facts.
This is the quiet truth about Retrieval-Augmented Generation (RAG). When an agent gives a wrong answer, the instinct is to blame the model or "prompt it harder." But in production, the majority of bad answers we debug are retrieval failures, not generation failures. The model did exactly what it was told - it just got handed the wrong context. Here are the five failure modes we see most often, and how we fix them.
1. Chunking that splits a fact in half
The default move is to slice documents into fixed 500-token windows. That works until a fact straddles a boundary - the eligibility rule is in chunk 14, the exception that voids it is in chunk 15, and your retriever returns only chunk 14. The agent now states a rule with total confidence and zero awareness of the exception.
The fix: chunk on structure, not character count. Split on headings, table rows, clauses, and list items. Add a small overlap (10-15%) so a fact and its caveat never get cleanly severed. For policy and contract data, we often store the whole section as one chunk even if it is long - a slightly bloated context beats an amputated fact.
2. Embeddings that confuse "similar words" with "same meaning"
Vector search retrieves what is semantically near the question. But "Can I cancel my subscription?" and "Can I cancel my appointment?" live close together in embedding space while meaning entirely different things in your system. Pure semantic search will happily hand back the appointment policy.
The fix: hybrid retrieval. Combine dense vector search with old-fashioned keyword (BM25) search and merge the results. Keywords catch the exact terms - product names, error codes, SKUs - that embeddings smudge together. In our experience this single change removes a large share of "close but wrong" retrievals.
3. No re-ranking, so the best chunk sits at position seven
Your retriever returns the top 20 candidates. The genuinely correct chunk is in there - at rank 7. But you only pass the top 3 to the model, so it never sees it. Recall was fine; ranking failed.
The fix: add a re-ranker. Pull a generous candidate set (say 20-30), then run a cross-encoder re-ranker that scores each chunk against the actual question and reorders them. Pass the top few after re-ranking. It is one extra step and it consistently lifts answer quality more than swapping to a bigger LLM.
4. Stale or duplicated documents
The 2023 pricing PDF and the 2026 pricing PDF both live in the index. Retrieval finds the 2023 one because it happens to be a tighter semantic match. Now your agent quotes prices from three years ago, and it is not wrong about the document - it is wrong about which document.
The fix: treat the index as a living dataset, not a one-time dump. Attach metadata (effective date, version, source) and filter on it at query time. Run a de-duplication pass. Re-index on a schedule. The most expensive RAG bugs we have untangled were not algorithmic - they were a forgotten stale file nobody removed.
5. No "I don't know" path
If retrieval returns nothing relevant, a naive pipeline still stuffs whatever it found into the prompt, and the model dutifully invents an answer. That is the hallucination everyone fears - except it was avoidable.
The fix: score the retrieval. If the top result's relevance is below a threshold, do not answer from it - say you do not have that information, or hand off to a human. An agent that knows the edge of its own knowledge is worth far more than one that bluffs.
How we test this before it ships
You cannot eyeball your way to a reliable RAG system. We build a small evaluation set - 50 to 100 real questions with known-correct source documents - and measure two things separately: retrieval accuracy (did we fetch the right chunk?) and answer accuracy (was the final response correct?). Splitting them tells you where the failure actually lives. Nine times out of ten, fixing retrieval fixes the answer, and you never needed a more expensive model at all.
RAG is not magic and it is not plumbing you can ignore. It is the layer that decides whether your agent is grounded in your business or improvising. Get it right and a mid-sized model outperforms a frontier model running on bad context. Get it wrong and no model on earth will save you.
About Shanti Infosoft: Shanti Infosoft is a CMMI Level 5 AI development company that has delivered 700+ projects across 16+ industries. We help teams move from AI ideas to dependable, production-grade software - shantiinfosoft.com | machine learning development services.
If your agent is confidently retrieving the wrong context, we can audit your retrieval pipeline and tune it against your own documents. Talk to our team.
Related reading: Can You Trust AI's Output? Hallucinations, Biased Evals, and the Human-QA Layer
Rishabh Jain is a Director at Shanti Infosoft, where the team builds AI agents and automation for real business operations.
Top comments (6)
Good list. The stale-document case is the one I see teams underestimate most, because the retrieval stack looks healthy while the corpus is quietly wrong.
One detail I would add: I like separating retrieval confidence from source validity. A chunk can be highly relevant and still invalid because its effective date, product version, or permission scope is wrong. So I usually treat metadata filters as a hard gate before reranking, not just extra text for the model to reason over.
For the "I don't know" path, thresholding also works better when it is evaluated per intent. A low-confidence pricing answer and a low-confidence FAQ answer should not have the same handoff threshold.
The metadata-as-hard-gate point is one we end up making to clients directly - a confidently wrong answer on stale pricing erodes trust far more than a system that admits it isn't sure. Per-intent thresholds are underrated for the same reason: the cost of being wrong isn't uniform across question types, so the handoff bar shouldn't be either. Have you found many teams willing to tune thresholds per intent, or do most still ship one global number and hope?
Most teams still ship one global number, honestly. Per-intent thresholds get resisted less because they're technically hard and more because nobody owns the decision - it needs someone who understands both the cost of a wrong answer per intent and the retrieval scores, and that's usually split across a PM and an ML engineer who never sit in the same meeting. The teams that do tune per-intent almost always got there after a specific incident (a bad pricing answer, a compliance-adjacent question answered wrong), not proactively. Same pattern as most RAG maturity, honestly - measurement discipline shows up after the first embarrassing failure, not before.
You've named the real blocker, and it's organizational more than technical. The PM knows the cost of a wrong answer, the ML engineer knows the retrieval scores, but nobody owns the number, so it defaults to one global threshold no one has to defend. The incident that finally forces the issue is usually what frees up the budget too, because now there's a name attached to the failure. When we get in early, half the job is just getting someone to own that call before the bad answer instead of after. Have you seen making the PM the owner actually stick, or does it tend to drift back to the ML side?
Drift is the more common outcome in my experience. Making it stick usually needs the ownership tied to a metric someone is actually evaluated on, not just a title on a doc. If the PM "owns" the threshold but nothing measures whether their calls were right, it slides back to whoever's on call during the next incident. The setups where it held were the ones where per-intent retrieval quality showed up in a weekly review, not just in postmortems after something broke.
That weekly-review point is the part most teams skip. We ended up putting per-intent recall on the same dashboard the on-call engineer already stares at, so a bad retrieval class surfaces before anyone files a ticket about a wrong answer. Tying it to a number in someone's review is what stopped the ping-pong for us too — postmortems only ever caught the incidents loud enough to notice. Do you sample the weekly eval set, or run it across the full intent taxonomy?