Retrieval-augmented generation is the most requested AI feature and the most commonly botched. The failure is almost never the generation step -- it's retrieval quietly returning the wrong context, and the model confidently summarizing garbage. This checklist comes from building research and analysis pipelines where wrong retrieval means wrong business decisions.
Chunking: respect the document's own structure
Fixed 512-token windows sliced mid-sentence are the default and the mistake. Chunk on semantic boundaries -- headings, paragraphs, table rows -- and prepend each chunk with its breadcrumb ("Annual Report 2025 > Risk Factors > Currency"). A chunk that can't be understood alone can't be retrieved reliably.
Hybrid search beats embeddings alone
Dense vectors miss exact identifiers -- SKUs, names, error codes -- that keyword search catches trivially. Run BM25 and vector search in parallel and fuse results (reciprocal rank fusion is fine). This one change fixes a huge share of "the answer was in the corpus but retrieval missed it" tickets.
Rerank before you generate
First-stage retrieval optimizes recall; a cross-encoder reranker optimizes precision on the top 30-50 candidates. Feeding the model 6 highly relevant chunks beats feeding it 20 mediocre ones -- better answers and lower token spend.
Metadata filters are half the battle
- Store structured fields (date, source, doc type, customer) alongside every chunk.
- Resolve filters before semantic search -- "latest pricing" should hard-filter on recency, not hope embeddings notice.
- Let the LLM extract the filter from the question with a typed schema, then apply it in the database.
Answer honesty: refuse to guess
The system prompt must make abstention a first-class outcome: if the retrieved context doesn't contain the answer, say so and show what was found. Require citations to chunk IDs and render them in the UI. Users forgive "not found"; they don't forgive confident fabrication.
Evaluate retrieval separately from generation
- Build a golden set of ~50 real questions with known source passages.
- Measure retrieval hit-rate (is the right chunk in the top-k?) -- this isolates most failures.
- Then grade end-to-end answers for faithfulness and completeness.
- Re-run on every change to chunking, embeddings, or prompts.
RAG is a search-quality problem wearing an AI costume. Treat it with the same rigor as the rest of your production agent stack, and it stops being the feature users quietly distrust.
Originally published at haiderfarooq.dev.
Top comments (0)