5 failure modes we hit building 12+ production RAG systems, and the architectural fixes that actually worked.
I've spent the last 14 months building production AI systems for fintech, healthcare, and SaaS clients. Of the 12+ RAG pipelines we've shipped, every single one failed in a different way than it did in staging.
Not broke. Failed. Silently degraded. Answered confidently and wrong. Retrieved the right document but extracted the wrong passage. Worked at 10 queries per minute and collapsed at 100.
Here's what we kept hitting, and what we fixed.
01 Naive chunking destroys retrieval quality
The default in most RAG tutorials is fixed-size chunking, splitting every document into 512-token chunks, embedding them, and then done. It works in demos. In production, it silently kills accuracy.
The problem: semantic meaning doesn't respect token boundaries. A contract clause that spans 600 tokens gets split in the middle. A medical report with a critical finding in the second half of a paragraph gets separated from its context. The retriever finds half the answer, and the LLM hallucinates the rest.
❌ Fixed-size chunking at 512 tokens: retrieval precision dropped to 54% on our healthcare client's policy documents after go-live.
What we switched to: a parent-child chunking strategy with semantic boundary detection.
Retrieval precision went from 54% to 81% on the same document set. The LLM gets the full semantic unit, not a fragment.
02 The wrong embedding model for your domain
Most teams default to text-embedding-ada-002 or a generic SBERT model. These are fine for general English. They're inadequate for financial filings, clinical notes, or legal language.
We had a fintech client whose RAG system was scoring 0.87 cosine similarity on retrieved passages, but the answers were wrong 40% of the time. The model was retrieving chunks that were superficially similar in language but semantically different in context. "Risk" in a compliance document does not mean the same thing as "risk" in an earnings call.
The fix: switch to a domain-adapted or domain-fine-tuned embedding model. For finance, BGE-financial or FinBERT embeddings. For clinical, ClinicalBERT or BioBERT as an embedding base. For general enterprise, a hybrid approach:
The instruction asymmetry matters. BGE models were trained with different prefixes for queries vs documents. Skip it, and you lose 8–12% recall on domain-specific content.
03 No reranking layer — cosine similarity isn't relevance
Vector similarity retrieves semantically proximate chunks. But proximity ≠ relevance to the specific question. You need a reranker.
Without a reranker, the top-k retrieved chunks are sorted by embedding similarity, which doesn't account for query-specific intent, negation, or specificity. We consistently saw the most relevant chunk sitting at position 4 or 5 in the retrieval output, behind noisier but "closer" matches.
04 Context window mismanagement at scale
At low query volumes, stuffing 8 retrieved chunks into the prompt works. At production scale with concurrent requests, you hit three problems: cost explosion, latency spikes, and more insidiously, the Lost in the Middle problem.
Research consistently shows that LLMs have lower recall for information buried in the middle of long contexts. If your most relevant chunk ends up at position 3 of 8 in the context, the model may not weight it appropriately.
Our production pattern now:
- Retrieve 20 candidates from the vector store
- Rerank to top 5
- Apply context compression to reduce token count by ~60%
- Place the most relevant chunk first and last (primacy + recency bias in LLMs)
05 No evaluation infrastructure — flying blind
This is the one that hurts the most to admit: most of the RAG systems we inherited had zero evaluation framework. They were shipped, deemed "working" based on informal testing, and degraded silently over weeks as the document corpus grew or the query distribution shifted.
You need three things before you go to production:
A golden dataset — 50–100 question/answer pairs manually verified against your document corpus
RAGAS metrics — faithfulness, answer relevancy, context precision, context recall
A weekly eval run — automated, tracked in a dashboard, with alerts if any metric drops more than 5%
✅ Once you have RAGAS running, you can actually compare chunking strategies, embedding models, and reranker configs quantitatively. It turns RAG tuning from guesswork into engineering.
What the fixed architecture looks like
After applying all five fixes on a healthcare SaaS client's policy document RAG system, here's what the numbers looked like:
The full production RAG stack now looks like: semantic chunking → domain-adapted embeddings → hybrid search (vector + BM25) → cross-encoder reranking → context compression → LLM with structured output + RAGAS eval loop.
Each layer adds ~50–150ms of latency. The tradeoff is worth it when the cost of a hallucinated answer in a healthcare or fintech context is a support ticket, a compliance issue, or a lost contract.
If you've hit any of these, or if your RAG system works great in staging and degrades in production, drop a comment. I'm collecting failure patterns across verticals right now and would love to hear what you're seeing.
We run a technical AI delivery practice called Ailoitte, if you're rebuilding a broken RAG pipeline and want to talk architecture, reach out.
Top comments (0)