Six months ago I shipped a RAG prototype that worked on my laptop. Two weeks later in production, it was returning stale answers, blowing through the LLM budget, and getting flagged for hallucinations the retrieval metrics said shouldn't be possible.
Here's the gap between "RAG tutorial that works" and "RAG system in production" — because every tutorial stops at the first one.
1. Retrieval metrics lie in production
Your test set is small, curated, and probably biased toward questions you already know have answers in the corpus. Production queries come in at a different angle: typos, missing context, questions the docs don't actually cover, questions that span multiple documents.
Recall@5 on your test set doesn't predict production quality. What does: a 7-field query log (query, retrieved chunk IDs, generated answer, user action, latency, cost, timestamp) and the discipline to actually look at the bad rows weekly.
2. Chunking strategy quietly costs you 30-50% retrieval quality
Most tutorials show fixed-size chunking because it's simplest. In production you need semantic chunking for narrative docs, structural chunking for code and markdown, parent-child retrieval for PDFs with tables, and a chunker version column so you can re-embed selectively instead of paying to re-embed the whole corpus every time you tweak the strategy.
A small delta-reindex pattern (hash each chunk, only re-embed changed ones) cuts embedding cost 99% on small corpus edits.
3. Vector DB cost shape surprises everyone
Pinecone's per-read-unit billing looks cheap at 100 queries/day. At 10,000 queries/day it crosses Qdrant Cloud's flat-rate. pgvector is free if you're already on Postgres — until you forget to create the HNSW index and every query becomes a sequential scan.
The real decision isn't "which vendor is best." It's "what's my query volume in 6 months, and what's my ops capacity?"
4. Evaluation is what separates "I shipped it" from "I can improve it"
Without an eval harness, every prompt tweak is a vibe check. LLM-as-judge works if you pin the judge model, calibrate the rubric, and check inter-rater agreement (Wilcoxon for statistical significance, not just "looks better"). 50 queries in the test set is enough to get real signal if you built it with a 5-step human review protocol rather than scraping queries blind.
5. Observability on RAG is NOT the same as observability on APIs
You don't want more alerts. You want three alerts: retrieval returned zero chunks, LLM cost per query crossed threshold, p95 latency broke SLO. Everything else goes on a Grafana dashboard you look at weekly, not in your pager rotation.
The debug script that solves "the answer was wrong" complaints in 5 minutes takes the query, replays the retrieval, prints the chunks that came back, and shows which ones the LLM actually used. Without this, "the answer was wrong" takes an hour per incident.
I spent six weeks compressing what I learned (plus three case studies from production RAG systems at different scales) into The RAG Production Handbook — launching today.
What's in it:
- 10 chapters, 5 appendices, ~95 pages — chunking, vector DB selection, evaluation, deployment, cost modelling, observability, 3 real case studies, vendor matrix, migration recipes
- Working code repo — chunkers.py (all 10 strategies), retriever.py (vendor-agnostic abstraction), evaluator.py, logger.py, Dockerfile, deployment configs for Vercel/Fly.io/Kubernetes
- Vendor-agnostic — patterns work with Claude, OpenAI, local models; pgvector, Qdrant, Pinecone, Chroma, Weaviate, LanceDB
Launch pricing:
- First 20 buyers: ₹699 (~$8.50)
- After that: ₹999 through Nov 30
- From Dec 1: ₹1499
7-day no-questions refund. Delivered as PDF + repo, emailed within seconds of payment.
Link: https://aiunplugged.in/products/rag-production-handbook/
Happy to answer questions in the comments — what part of RAG production bit you hardest?
Top comments (0)