Your AI model does not know what happened yesterday. Your AI model does not know your company's internal salary sheet. Your AI model will confidently make up an answer rather than admit ignorance. RAG (Retrieval Augmented Generation) exists to fix these three problems, and understanding RAG properly changes how you build AI systems.
This article covers:
- What RAG is and why the industry depends on it
- Seven real problems that break RAG systems in production
- How to measure whether your RAG system works at all
- Ten advanced techniques, with diagrams for the ones that need one
- When RAG is the wrong tool entirely
The original technique comes from a 2020 Facebook AI paper that combined a language model's internal knowledge with an external document index. That paper is still the right starting point if you want the formal definition:
Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," 2020.
Section 1: What RAG Actually Is
RAG stands for Retrieval Augmented Generation. Before your AI model answers a question, a system fetches relevant facts from a database, then hands those facts to the model along with the question.
- Step 1: the user sends a query in plain language.
- Step 2: an embedding model converts the query into a vector (a grid of numbers representing meaning, not the exact words).
- Step 3: a vector database finds the stored chunks closest in meaning to that vector.
- Step 4: the LLM writes an answer using the question plus the retrieved chunks.
Keyword search vs vector search
This is the comparison that explains why RAG exists at all.
| Keyword search | Vector search | |
|---|---|---|
| Matches on | Exact words | Meaning |
| Speed | Fast | Slightly slower |
| Handles synonyms | No, "Heathrow" misses "London" | Yes, both land close together |
| Setup cost | Low | Needs an embedding model and a vector index |
| Best for | Exact codes, IDs, known terms | Natural language questions |
Real-world case: a telecom support bot took customer questions about "cell coverage" against a knowledge base written with the term "network signal strength." Keyword search returned nothing for roughly 30% of these queries. Switching the retriever to vector search closed most of that gap without touching a single support document.
Key takeaway: vector search doesn't replace keyword search; it covers the gap where users phrase things differently than your documents do. Many production systems run both and merge the results.
Further reading: the original RAG paper →
Section 2: Seven Problems That Break RAG in Production
A demo working on ten test questions tells you almost nothing about production behavior. Here's where the seven failure patterns cluster.
Problem 1: No conversation history
A user asks "Who is Avery?" then "What is her salary?" A basic setup treats every message as brand new, so "her" pulls up whichever salary document ranks first, which could belong to someone else entirely.
Real-world case: an internal HR chatbot answered "What is her salary?" with the wrong employee's number three times before the team traced it to missing conversation history.
Fix: pass every prior message, user and assistant, into the model on every call. One afternoon of work once found.
Problem 2: Search only looks at the last message
Even with full history, if the retrieval step only searches the literal last message, "what is her salary" still has no name in it. Fix: combine the conversation history into one search string before querying. Search "Avery, what is her salary" instead of "what is her salary."
Problem 3: Fixing #2 breaks topic switches
After several turns about Avery, the user asks "Who won the IOTY award?" The system keeps returning Avery chunks because the combined search string is now dominated by her name. No clean fix exists yet: query rewriting, recency weighting, and a dedicated "clean query" model all help partially.
Counter-view: some teams call this overstated, since most conversations stay on one topic for many turns, so recency weighting alone solves most real cases. Others note the remaining failures are exactly the moments users notice and lose trust.
Problem 4: Chunks that are too small lose context
A chunk says "won the award" without the name, because the name sat in a heading, in a different chunk.
Real-world case: a legal search tool returned clause text with no contract name attached; lawyers had to manually trace every result. Adding the document title to each chunk's metadata at ingestion time fixed it without touching chunk size.
Problem 5: Chunks that are too large dilute relevance
A chunk covering someone's entire career, salary, awards, and reviews will never score close to a narrow question like "who won the award," because it's diluted with unrelated text.
Real-world case: a team running a support bot for a SaaS product raised chunk size from 500 to 1500 tokens, expecting richer context and better answers. Accuracy on their eval set dropped instead, because each retrieved chunk now carried three unrelated topics, and the model occasionally answered from the wrong one buried in the middle. They reverted to 600 tokens with a small overlap and recall went back up.
Problem 6: RAG is fundamentally approximate
A system that looks logically sound passes ten questions and fails the eleventh, because vector similarity is a mathematical shortcut, not real understanding. Nothing guarantees the right chunk surfaces every time. The only real fix is measuring everything (Section 3).
Problem 7: Wrong encoder at query time
Build the vector store with one embedding model, query it with a different one, and you get a dimension mismatch crash. Different models place text in different mathematical spaces entirely; a 300-dimension vector isn't comparable to a 3000-dimension one.
Alert: mixing encoder versions is one of the most common production outages in RAG systems. Pin your embedding model version and treat any change as a full re-ingestion, not a config update.
Summary table
| Problem | Root cause | Fix |
|---|---|---|
| No conversation history | Only current message sent to model | Pass full history every call |
| Wrong chunks from pronouns | Search uses last message only | Search using combined history |
| Topic switching fails | History dominates the search string | Query rewriting, recency weighting |
| Context missing (too small) | Name and content split across chunks | Overlap, parent-child chunking, metadata |
| Wrong chunk surfaces (too large) | Broad chunks score low on narrow queries | Smaller, focused chunks |
| Unpredictable failures | Retrieval is probabilistic by design | Build evals, measure, iterate |
| Dimension mismatch crash | Different encoder at ingest vs query | Same model throughout, always |
Pros and cons of RAG as an approach
| Pros | Cons |
|---|---|
| Cheaper than fine-tuning on new data | Retrieval quality is never guaranteed, only probabilistic |
| Keeps the model current without retraining | Requires ongoing engineering: chunking, indexing, evals |
| Keeps private data out of model weights | Struggles with questions spanning many documents |
| Answers can cite exact sources | Adds latency: embed, search, then generate |
| Works well for narrow, factual questions | Breaks down on vague or multi-topic conversations |
Key takeaway: retrieval quality matters more than prompt quality. If the wrong document gets retrieved, the best prompt in the world won't fix the answer.
Section 3: How to Measure Whether Your RAG System Actually Works
Without measurement, RAG work turns into guesswork.
Step 1: Build a golden dataset. Collect real questions and correct answers from your own data.
Question: "Who won the prestigious IOTY award?"
Keywords: ["Maxine", "Thompson"]
Perfect Answer: "Maxine Thompson won the IOTY award in 2023."
Step 2: Measure Retrieval Quality
Before evaluating the final answer, make sure your retrieval system is returning the right context. These metrics help measure retrieval performance.
| Metric | What it measures |
|---|---|
| MRR (Mean Reciprocal Rank) | Measures how highly the first relevant chunk is ranked. A relevant chunk at rank 1 scores 1.0, rank 2 scores 0.5, rank 3 scores 0.33, and so on. |
| Recall@K | Measures how often the relevant information appears within the top K retrieved chunks. Higher recall means the system is less likely to miss important context. |
| Keyword Coverage | Measures the percentage of expected keywords or entities that appear across the retrieved chunks. |
| Precision | Measures what proportion of the retrieved chunks are actually relevant to the user's query. |
Key insight: In most RAG systems, Recall@K is more important than Precision. A few irrelevant chunks usually add only minor noise, but failing to retrieve the correct chunk often leads to an incorrect or incomplete answer.
Recall matters more than precision in most RAG systems: a missing relevant chunk is a wrong answer, an extra irrelevant chunk is just noise.
Step 3: Measure answer quality with LLM as a Judge. A second, stronger model scores the generated answer against your reference on accuracy, completeness ("Maxine Thompson," not just "Maxine"), and relevance.
Real-world case: a support team ran weekly judge scoring on 200 golden questions. A new chunking strategy dropped completeness scores by 12% in that eval run, caught before it ever reached a customer.
Key takeaway: if you aren't scoring MRR, Recall@K, and answer quality on a fixed dataset, you're not doing RAG engineering; you're guessing and hoping.
Further reading: MRR and ranking metrics explained →
Section 4: Ten Advanced RAG Techniques
Alert: none of these work automatically for every dataset. Run evals on your own golden set before and after each change, or you're not measuring anything.
1. Chunking R&D. Test different splitting strategies with a hypothesis first ("this fails because chunks run too big"), not at random. Semantic chunking splits by meaning instead of character count.
LangChain's text splitters →
2. Encoder selection. Test multiple embedding models against your eval set. For images, caption first and vectorize the caption. For PDFs, convert to markdown with a code library before embedding; don't burn a model call on it.
3. Prompt engineering. The most skipped technique. Static context (company details, current date, domain rules) plus correct history handling frequently beats a full architecture rebuild.
4. Document rewriting. A table of numbers vectorizes poorly. Rewrite it as natural language before ingestion: "Rewrite this table matching how users would ask about it."
Query-time techniques change what gets searched for and how the results get filtered before reaching the model:
5. Query rewriting (diagrammed above). Call a model before searching, prompt: "rewrite this as a standalone query, resolving any pronouns from the conversation." Handles the "her salary" problem directly.
6. Query expansion. Instead of one rewritten query, generate several phrasings and search with all of them, then pool the results.
Real-world case: one user question expanded into three phrasings surfaced three genuinely different relevant chunks that a single query missed. The model got a richer, more complete context pool than any one query would have found alone.
7. Reranking. Query expansion produces a lot of chunks fast, many irrelevant. A separate model reorders them by relevance to the original question and does not answer anything itself; ranking is its only job.
8. Hierarchical RAG. Fixes questions spanning many documents, like "how many employees earn more than $60k?" No single chunk has that answer. Build summary documents at multiple levels; search summaries first (broad), then drill into specific chunks (narrow).
Alert: you can only build a hierarchy for a question type you anticipated. Someone will ask "how many documents start with the letter A," and no hierarchy will exist for that. This reduces the whack-a-mole problem; nothing eliminates it.
9. Graph RAG. Documents relate to each other: an employee has a manager, a product belongs to a category. Standard chunk retrieval misses these relationships entirely.
Fix: store relationships in chunk metadata, and pull in 1-2 neighbor chunks whenever a chunk gets retrieved. For heavily interconnected data, a dedicated graph database handles this properly.
Microsoft's GraphRAG research → uses this same idea at scale, building a knowledge graph from the corpus before retrieval. For most use cases, metadata alone covers it
Neo4j → is the common choice when you need the full graph database.
10. Agentic RAG. Instead of a fixed pipeline, give a model tools and let it decide what to call, in what order, looping until it has enough context.
Strength: handles questions a fixed pipeline fails on completely.
Weakness: the same question can produce different results on different runs, which makes testing harder and reliability lower for production. Agentic RAG is really techniques 5, 6, and 7 run by the model itself instead of your code.
The full survey on Agentic RAG → covers the design patterns (reflection, planning, tool use, multi-agent collaboration) in more depth.
Key takeaway: every technique here amplifies a solid foundation. None of them fix a broken chunking strategy or a missing eval set. Build the simple pipeline first, measure it, then add complexity where the numbers say you need it.
When NOT to Use RAG
RAG solves a specific problem: too much data to fit in a prompt, changing too often to bake into model weights. Outside that problem, it's often the wrong tool.
- Small datasets. If your entire knowledge base fits comfortably in the context window, retrieval adds latency and failure points for no real gain. Paste it in directly.
- Static documentation that rarely changes. A small, stable FAQ or policy doc doesn't need a vector database and an eval pipeline. A well-written prompt with the content inline is simpler and just as accurate.
- Structured SQL data. "How many orders shipped late last month" is a database query, not a retrieval problem. Vector search on rows of numbers performs badly; a text-to-SQL approach or a direct query is the right tool.
- Exact mathematical calculations. RAG retrieves text. It doesn't compute. A calculation belongs in code or a calculator tool, not in a retrieved chunk.
Key takeaway: RAG is for grounding language answers in a large, changing body of unstructured text. If your data is small, static, structured, or the question is a computation, reach for a simpler tool first.
Is RAG Dead?
Two arguments circulate regularly, and both fall apart under scrutiny.
"Context windows got huge, just paste everything in." Knowledge bases grow faster than context windows do. Even when everything technically fits, filtering out the irrelevant 90% before generation stays more efficient in cost and accuracy.
"Agents replaced RAG." Agents still call vector search, still use encoders, still run retrieval techniques. The orchestration changed. A model deciding when to search is still generation augmented by retrieval, wearing a different name.
Final Takeaways
- RAG grounds a model in real data instead of letting it guess.
- Every stage (encoding, retrieval, chunking, prompting) fails in its own distinct way.
- Measurement through a golden dataset separates real engineering from guesswork.
- Advanced techniques amplify a good foundation; none of them fix a broken one.
- Know when not to use it: small, static, structured, or purely computational data doesn't need retrieval.
| # | Technique | What it fixes |
|---|---|---|
| 1 | Chunking R&D | Wrong chunk size |
| 2 | Encoder selection | Wrong model for your data type |
| 3 | Prompt engineering | Missing context, poor history handling |
| 4 | Document pre-processing | Documents not shaped for search |
| 5 | Query rewriting | Poor standalone search queries |
| 6 | Query expansion | One query missing relevant chunks |
| 7 | Reranking | Too many chunks, polluted context |
| 8 | Hierarchical RAG | Questions spanning many documents |
| 9 | Graph RAG | Relationship-heavy data |
| 10 | Agentic RAG | Complex, multi-step retrieval decisions |
Top comments (0)