- Book: RAG Pocket Guide
- Also by me: Database Playbook
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
The global RAG market is on track to clear $40 billion in 2026. According to vendor write-ups, somewhere between 70% and 80% of the enterprise RAG projects feeding that number never reach production (Blits.ai, 2026, Ragaboutit, 2026). The demos are great. The pilots ship a deck. Then someone tries to point the chatbot at the actual document corpus and the thing falls apart in a small, embarrassing way.
Picture the kind of failure a team I talked to ran into. Their internal HR bot quoted an old parental-leave policy to an employee who was checking what they were entitled to. The current policy had been live on the intranet for months. The chatbot had never re-indexed it. The team that owned the bot did not know the bot had drifted, because nothing on the dashboard tracked source freshness against retrieval. The link works. The model answers. The number is wrong.
Demos hide this. Production exposes it. The failures cluster into five repeatable patterns that show up across contracts-QA, customer support, internal knowledge bases, and developer assistants. Below is each pattern, the anchor that makes it concrete, and where it actually lives in your pipeline.
1. Knowledge staleness
The freshness gap is the most common silent failure I hear about from teams running RAG against intranet content. A document gets re-published. The vector store still holds the old chunks. The retriever scores by semantic similarity, not by ingestion timestamp, so the old chunk wins because its language matches the query better.
A reranker that ignores recency will keep returning the 18-month-old chunk forever. A 2025 paper formalised this — a simple recency prior beats heuristic trend-detection on freshness benchmarks (arXiv 2509.19376). The fix is not exotic. You add a recency multiplier to the relevance score before final ranking.
import math
from datetime import datetime, timezone
def freshness_score(
base_score: float,
indexed_at: datetime,
half_life_days: float = 90.0,
) -> float:
now = datetime.now(timezone.utc)
age_days = (now - indexed_at).total_seconds() / 86400
decay = math.exp(-math.log(2) * age_days / half_life_days)
# Mix: 70% semantic, 30% recency. Tune per corpus.
return 0.7 * base_score + 0.3 * decay
This sits between your vector search and your final top-k cut. Each chunk needs an indexed_at field carried through the pipeline as metadata. If you cannot answer "when was this chunk last refreshed" inside your retriever, you cannot ship a serious RAG system against documents that change.
2. Retrieval precision against tables and figures
The second failure is structural. You retrieve five chunks. Three are loosely on-topic prose. The actual answer was in a table on page 14 that your PDF parser flattened into a row of pipe characters that no embedder treats as meaningful. The model now confidently answers from prose context that was tangentially related.
Tables, figures, and footnotes are where naive ingestion bleeds the most precision. The fix is replacing your text extraction with a structure-aware parser. unstructured and docling both segment a PDF into typed elements: headings, paragraphs, tables, images. Each gets embedded with the right strategy.
from unstructured.partition.pdf import partition_pdf
elements = partition_pdf(
filename="vendor_contract.pdf",
strategy="hi_res",
infer_table_structure=True,
extract_image_block_types=["Table", "Image"],
)
for el in elements:
if el.category == "Table":
# Preserve the HTML so the embedder sees structure.
chunk_text = el.metadata.text_as_html
chunk_kind = "table"
else:
chunk_text = el.text
chunk_kind = el.category.lower()
yield {
"text": chunk_text,
"kind": chunk_kind,
"page": el.metadata.page_number,
"indexed_at": datetime.now(timezone.utc),
}
text_as_html matters. If your table chunk goes into the embedder as raw cell text with no structure, the embedding represents a bag of cell values. If it goes in as HTML, the embedder picks up column-row relationships, and a query like "what is the late-payment penalty in tier 3" has a chance of matching the actual cell.
docling is the IBM-backed alternative and tends to do better on multi-column academic PDFs. Pick one, benchmark on your real corpus, do not skip this layer.
3. Document boundary loss
A 30-page contract gets split into 60 chunks. The retriever finds chunk 47, which contains "the Licensee shall indemnify the Licensor for…" — except "Licensee" is defined on page 2 and the indemnity scope is qualified by clause 12.3 on page 21. The retrieved chunk reads as a complete clause to the embedder. To a lawyer it is a fragment with three open references.
This is what parent-document retrieval and hierarchical chunking exist to solve. Match small (a clause). Return large (the section). The short version: production teams who care about contract-style documents almost always end up with a child-parent split, and the ones who don't end up explaining to a customer why the bot omitted a clause.
4. The eval gap
You will not detect any of the three failures above without a retrieval eval rig. The depressing finding from multiple 2026 surveys is that roughly 70% of teams running RAG in production have no systematic eval for retrieval quality (Ragaboutit, 2026). They have an LLM judge for end-to-end answer quality, sometimes. They have nothing that catches a recall regression at the retriever layer.
Two metrics carry the weight: context recall (did the retrieved chunks contain the facts needed to answer) and context precision (what fraction of retrieved chunks were relevant). Ragas gives you both for free over a labelled set. Library choice is not the hard part. The hard part is committing to a 200-question gold set that you re-run on every embedding-model change, every chunking-config change, every reranker change. Without it, you are flying blind and your dashboard is green because the dashboard does not measure what is broken.
5. Naive RAG architecture for enterprise complexity
The fifth failure is architectural drift. The team copies a tutorial that wires a single embedder, a single vector store, and a single LLM call. Then product asks for source citations (now you need chunk-level metadata). Legal wants access controls per document (now you need pre-filtering on user identity). Ops wants to know why the bill spiked (now you need cost tracking per query). Someone in the all-hands wants to know why answers about Q3 financials cite Q1 docs (back to staleness).
The honest take from the 2026 production RAG architecture write-ups is that a working system has seven moving parts: a structured ingestion layer, a chunking layer with metadata, a hybrid retriever (dense + BM25), a reranker, a freshness/access filter, an LLM with cited generation, and an eval loop running async on production traffic. None of those are optional. All of them are what teams skip in the demo and pay for in week three of the rollout.
A pipeline that fixes the worst two
Putting freshness-aware reranking and structure-aware extraction together gives you the single biggest jump for the least engineering cost. The combined retrieve path looks like this in practice.
from dataclasses import dataclass
from datetime import datetime
@dataclass
class Chunk:
text: str
kind: str # "table" | "paragraph" | "heading"
page: int
indexed_at: datetime
score: float = 0.0
def retrieve(query: str, k: int = 20) -> list[Chunk]:
# 1. Hybrid retrieve top-k * 5 candidates.
candidates = hybrid_search(query, k=k * 5)
# 2. Cross-encoder rerank.
candidates = cross_encoder_rerank(query, candidates)
# 3. Apply freshness multiplier.
for c in candidates:
c.score = freshness_score(c.score, c.indexed_at)
# 4. Boost tables when query mentions structured terms.
if any(t in query.lower() for t in ["price", "fee", "tier"]):
for c in candidates:
if c.kind == "table":
c.score *= 1.15
candidates.sort(key=lambda c: c.score, reverse=True)
return candidates[:k]
This is not clever. It is the unglamorous middle of the pipeline that nobody writes a Medium post about. It is also where the recall regressions live. Every chunk has a kind. Every chunk has a timestamp. Every score gets adjusted before the top-k cut. The query path can answer the three questions a debugger needs to ask: was the source fresh, was the structure preserved, did the right chunk make the cut.
Where the 70% number actually comes from
It is not a single survey. It is a cluster of consultancy and industry-blog reports converging on the same range: Flexsin, Blits.ai, and Ragaboutit. All cite teams that ran a pilot, hit the failures above, and stalled between demo and production. The shape of the failures is consistent. The fix is consistent. The teams that ship are the ones who treat retrieval as a system with five layers, each instrumented.
The teams that don't ship are still in the demo room, watching the answer come back fast and confident, not asking the question that breaks it.
If this was useful
The RAG Pocket Guide walks through retrieval, chunking, and reranking patterns for production end-to-end — the same five failure modes above with code, evals, and the operational details that demos skip. If your retrieval layer is the weak link, start there. The Database Playbook is for the layer underneath: which vector store, which hybrid setup, which metadata schema actually holds up when your corpus grows past a million chunks.
Top comments (0)