Most teams get RAG working in a notebook over a weekend. Very few get it working reliably in production. The gap is not model quality — it is engineering discipline.
The RAG prototype is fifty lines of Python. It works. Then production happens — users ask unexpected questions, retrieval degrades as the corpus grows, and the model confidently synthesizes wrong answers from bad context. Nobody knows, because there is no instrumentation to catch it.
Chunking: The Foundation
A poor chunking strategy cannot be compensated for downstream. If relevant information is split across chunks or diluted into one too large, no retrieval algorithm will recover it.
Hierarchical chunking is the production-grade pattern: maintain parent chunks (full sections) and child chunks (sentences/short paragraphs). Retrieve at child granularity for precision. Return parent text as LLM context for completeness.
Every chunk must carry metadata — source document ID, version, content hash, embedding model version. content_hash tells you when a chunk needs re-embedding because the source changed.
Retrieval: Hybrid Is the Default
Neither BM25 nor vector search alone is sufficient. Hybrid retrieval with Reciprocal Rank Fusion (RRF) is the baseline for production RAG.
The pipeline:
- Dense retrieval (vector similarity) + Sparse retrieval (BM25 keywords) in parallel
- RRF merge — rank-based fusion without score normalization
- Cross-encoder re-ranker — precision pass on top candidates
Skipping the re-ranker is the most common mistake. Initial retrieval optimizes for recall. The re-ranker optimizes for precision — critical when your context window only fits top-5 chunks.
Context Assembly: Where Pipelines Quietly Break
- Token budget management — hard ceiling, never rely on hope that chunks fit
- Deduplication — hierarchical chunking and hybrid retrieval can surface the same content via multiple paths
- Source attribution — every chunk in context must carry its source ID for citation
The "I Don't Know" Instruction Is Not Optional
Without explicit grounding instructions, LLMs fill context gaps with plausible hallucinations. Your system prompt must instruct the model to acknowledge when context is insufficient — and to cite sources for every factual claim.
Evaluate Retrieval Independently
The most common RAG debugging mistake: assuming a bad answer is a generation failure. Most bad RAG answers are retrieval failures — the right chunk was not in the context.
Measure Recall@K and MRR against a ground truth dataset of 50-100 queries. Fix retrieval before you blame the model.
Production Observability
A RAG pipeline without observability is a black box that silently degrades. Key signals:
- "I don't know" rate — drops below 80% signals retrieval degradation
- Chunks dropped rate — rising means context window pressure
- Retrieval latency p99 — vector index performance
- Corpus staleness — content hash mismatches between source docs and stored chunks
Read the Full Article
This is a summary of my deep dive into production RAG engineering. The full article covers every pipeline component with implementation examples:
👉 Building Reliable RAG Pipelines — Full Article
The full article includes:
- Full pipeline architecture diagram (9 stages)
- Three chunking approaches with Python implementations (fixed, semantic, hierarchical)
- Hybrid retrieval with RRF implementation (Qdrant)
- Cross-encoder re-ranking (self-hosted and Cohere API)
- Context assembly with token budget management and deduplication
- Prompt construction with grounding and guardrails
- Retrieval evaluation framework (Recall@K, MRR, context relevance)
- Per-request tracing schema and aggregate alerting metrics
- Corpus staleness detection implementation
- Graceful degradation with BM25 fallback
- Production deployment checklist
Top comments (0)