Retrieval-augmented generation looks trivial in a tutorial: embed some documents, drop them in a vector database, stuff the top matches into a prompt, done. Then you point it at real company data and real users, and you discover that the demo was the easy 10%.
We build RAG systems over private knowledge for companies, and almost every painful bug traces back to the same handful of failure modes. Here they are, and what actually fixes them.
1. Retrieval returns the wrong chunks — and the model uses them anyway
The single biggest source of "wrong" RAG answers isn't the LLM. It's retrieval handing it irrelevant or partial context, which the model then summarizes with total confidence. Naive fixed-size chunking splits a table from its header, or a clause from the sentence that negates it.
The fix is unglamorous data engineering: chunk on semantic boundaries, not character counts; add a reranking step so the top-k you actually pass is the top-k by relevance, not by raw vector distance; and store enough metadata to filter before you search. Retrieval quality sets the ceiling on everything downstream.
2. The model answers beyond its context
Even with perfect retrieval, an LLM will happily fill gaps with plausible invention. In a RAG system that's worse than no answer, because it looks sourced.
Force grounding: instruct the model to answer only from the retrieved context and to say "I don't know" when the context doesn't cover it — then verify that with citations that point back to specific chunks. If you can't trace a sentence to a source, treat it as a hallucination, not an answer.
3. The knowledge goes stale, and nobody notices
RAG is only as good as the index behind it. Documents change, get duplicated, get deleted — and a pipeline that ingested once at launch quietly serves last quarter's truth. The unsexy work is the ingestion pipeline: incremental re-indexing, de-duplication, and a freshness signal so old content can be down-weighted or expired.
4. You have no evals, so you're guessing
"The new embedding model feels better" is not an engineering statement. Without a held-out set of real questions with known-good answers, every change is a coin flip — you fix one query and silently break five. Build an eval set early, measure retrieval hit-rate and answer faithfulness on every change, and treat a regression like a failing test.
5. Latency and cost sneak up on you
Embedding the query, searching, reranking, and stuffing a large context into a big model adds up — in both seconds and dollars. Cache embeddings and frequent queries, retrieve fewer-but-better chunks rather than dumping everything, and reserve the largest model for the steps that genuinely need it.
The pattern underneath all of these
Notice what's missing from that list: clever prompting. Production RAG is a data and retrieval engineering problem wearing an AI costume. The teams whose RAG holds up aren't the ones with the fanciest prompt — they're the ones who treat ingestion, chunking, retrieval, and evaluation as real systems with real tests.
That's the lens we bring from years of building data systems at scale before this wave. If you're moving a RAG prototype toward something users can actually trust in production, that's the kind of RAG and knowledge-AI work we do at Krazimo.
What's bitten you hardest in a production RAG system? I'll dig into specifics in the comments.
Top comments (2)
One failure mode I'd add: the silent-drop. When your index is sparse for a query area, retrieval returns 2 chunks instead of the k=5 you asked for — and the LLM doesn't know the context is incomplete. It answers confidently from partial evidence because nothing signals the gap.
The fix that worked for me: pass the retrieved count alongside the chunks in the prompt ("3 of 5 chunks found") and instruct the model to treat low coverage as low-confidence. The grounding check you described breaks down if the model doesn't know it only got 60% of the evidence it needed.
Silent-drop is the right name for that failure mode. What you describe is structurally a watchdog-on-absence problem: the retrieval ledger says k=5 requested, k=2 returned, and nobody downstream knows the gap exists because the missing rows do not write themselves into the prompt. The grounding check is conditioned on completeness it cannot verify.
Passing the count is the right move, and the primitive can be sharpened. Coverage is a column-level fact with a clear counterparty — the retriever knows what it returned vs what was asked. That makes it a real downward weight on confidence, not just a flag. Tell the model the coverage ratio and require it to compress its claim to what k=2 of 5 can actually carry. Cuts the failure mode at the same place you are catching it now, without depending on the model recognizing the warning string in its context window.
Adjacent failure worth not conflating: even with full k, „found 5" is silent about whether those 5 cover the query semantics. Coverage by count is one floor; coverage by topic span is another floor entirely. First is checkable from retrieval metadata; second needs a separate eval. Same signal name hides two different observability problems.