The Retrieval Pipeline Is Lying to You: How RAG Fails Before the LLM Sees Anything

Your RAG system did not fail because the model hallucinated.

It failed because the only “facts” the model saw were a mangled PDF table, an outdated policy, a chunk with missing context, and three near-duplicate paragraphs that pushed better evidence out of the top-k results.

The LLM was downstream of a retrieval pipeline that had already distorted reality.

This is the part of RAG that is easy to miss. Teams spend a lot of time choosing models, tuning prompts, and debating context windows. But in production, a shocking number of failures happen earlier: during ingestion, chunking, indexing, filtering, ranking, and query transformation. By the time the LLM receives the retrieved context, the answer may already be impossible.

The retrieval pipeline is not a neutral search layer. It decides what the model is allowed to know.

TL;DR

• Most RAG failures are not prompt failures; they are retrieval-pipeline failures.
• Ingestion and parsing can destroy the meaning before embedding begins.
• Chunking is context surgery, not text splitting.
• Vector similarity is not the same as evidence relevance.
• Metadata, permissions, versioning, and time are where production trust lives.
• If you only evaluate final answers, you cannot diagnose retrieval failures.

📋 Table of Contents

• The LLM never sees reality
• 1. Your source of truth is already damaged at ingestion
• 2. Chunking is context surgery not text splitting
• 3. Generic embeddings flatten your domain
• 4. Similarity is not evidence
• 5. Metadata is where trust and permissions live
• 6. Old versions quietly poison the index
• 7. The user query is not the real question
• 8. Duplicates make retrieval confidently wrong
• 9. If you only evaluate final answers you are blind
• A retrieval contract for production RAG

The LLM never sees reality

A retrieval-augmented generation system gives the model a narrow slice of the world: the retrieved chunks.

If that slice is incomplete, outdated, duplicated, unauthorized, or structurally broken, the model cannot reason its way out of that problem. It can only work with what it was given.

That is why the phrase “the retrieval pipeline is lying” is not just dramatic wording. The pipeline creates the model’s operating reality.

A bad prompt can usually be fixed with a better prompt.

A bad retrieval pipeline is worse. It silently supplies bad evidence.

This matters even more now because many teams are moving toward hybrid retrieval, rerankers, metadata filtering, and agentic query planning. Those techniques improve things, but they also add more places where the system can fail before generation starts.

The rest of this article walks through the failure modes I would check first when a RAG system produces wrong answers even though “the document is in the database.”

1. Your source of truth is already damaged at ingestion

Scenario:

Your knowledge base contains the correct answer in a table inside a PDF. The user asks a question that should be easy to answer. The model returns nonsense because the retrieved chunk looks like this:

Plan A 10 20 50 Plan B 15 25 75 Monthly Annual

The table was technically retrieved. But its meaning was destroyed during extraction.

Why it matters:

A lot of RAG teams treat ingestion as a boring preprocessing step: extract text, chunk it, embed it, move on. That is a mistake. If the extraction layer loses structure, the embedding layer embeds garbage, and the generation layer receives garbage with confidence.

Documents are not just bags of words. They contain:

• headings,
• lists,
• tables,
• code blocks,
• captions,
• footnotes,
• page context,
• and document hierarchy.

When parsing flattens all of that into one-dimensional text, the model loses the relationships that make the content meaningful.

Solution:

Make ingestion document-type aware. Preserve structure where it matters, and convert tables into a representation the model can actually read.

A practical ingestion layer should produce structured blocks, not just raw text.

from dataclasses import dataclass

@dataclass
class ParsedBlock:
    block_id: str
    doc_id: str
    kind: str  # "text", "table", "code", "heading"
    section_path: list[str]
    content: str
    source_location: str

For tables, do not dump cells as a single line. Convert them into Markdown, CSV, or a compact JSON structure.

def table_to_markdown(headers: list[str], rows: list[list[str]]) -> str:
    header_line = "| " + " | ".join(headers) + " |"
    separator = "| " + " | ".join(["---"] * len(headers)) + " |"
    body = [
        "| " + " | ".join(row) + " |"
        for row in rows
    ]
    return "\n".join([header_line, separator, *body])

Then store both the searchable text and the structured representation.

Why this works:

The LLM is much better at reading preserved structure than reconstructing it from flattened text. A Markdown table gives the model row and column relationships. A flattened string does not.

🚨 Production warning:

If your corpus contains PDFs, scanned images, slides, or HTML with heavy navigation boilerplate, parsing is not a solved problem. It is one of the highest-impact parts of your RAG system.

2. Chunking is context surgery not text splitting

Scenario:

The retrieved chunk says:

“The limit is 50 per workspace. Exceeding it triggers a soft stop.”

The user asked: “What is the API rate limit for Enterprise?”

The chunk is from the right document. But it does not say what “the limit” refers to, because the previous chunk contained the subject.

Why it matters:

Naive chunking breaks references.

If you split text by fixed token counts, you will routinely cut between:

• a heading and its content,
• a question and its answer,
• a definition and its usage,
• a list and its introduction,
• a pronoun and its antecedent.

The retriever may find the chunk because the words are similar. But the chunk is semantically orphaned.

Solution:

Chunk with context boundaries, not just length boundaries.

A good chunk should usually carry:

• document title,
• section path,
• nearby heading context,
• and enough surrounding text to be self-contained.

from dataclasses import dataclass

@dataclass
class Chunk:
    chunk_id: str
    doc_id: str
    parent_id: str | None
    section_path: tuple[str, ...]
    text: str
    searchable_text: str


def contextualize_chunk(doc_title: "str, chunk: Chunk) -> str:"
    path = " > ".join([doc_title, *chunk.section_path])
    return f"{path}\n\n{chunk.text}"

For retrieval, you can embed the contextualized version while still storing the original text.

Even better, use a parent-child pattern:

• retrieve using small, precise chunks,
• but send the larger parent section to the LLM.

def build_context_from_hits(hits: list[Chunk], chunk_store, parent_store) -> list[str]:
    parent_ids = {hit.parent_id for hit in hits if hit.parent_id}
    parents = [parent_store.get(pid) for pid in parent_ids]
    return [parent.text for parent in parents if parent]

Why this works:

The small chunk gives retrieval precision. The parent chunk gives the LLM enough context to understand what the precise chunk actually means.

💡 Practical note:

If your chunks often begin with “This”, “It”, “The above”, or “As described”, your chunking strategy is probably breaking referential context.

3. Generic embeddings flatten your domain

Scenario:

A user asks about “credit limits”. The retriever returns documents about “credit scores”, “credit cards”, and “credit risk” because they are semantically close. The exact policy about account credit limits is ranked too low to matter.

Why it matters:

Embedding models are powerful, but they are not magical. They encode general semantic similarity. They do not automatically understand the distinctions that matter in your product, your legal language, your codebase, or your internal terminology.

In production, this creates subtle failures:

• “environment” means deployment environment in your docs, but the embedding model leans toward general computing environments;
• “workspace” means tenant container in your product, but the model treats it as a generic UI concept;
• “policy” means insurance policy in one corpus and access policy in another.

The retriever returns plausible text. It is just not the right text.

Solution:

Do not rely on vector similarity alone.

A production retrieval pipeline usually needs at least three layers:

1. keyword search for exact terms, IDs, error codes, and names;
2. vector search for semantic similarity;
3. metadata filtering for source, version, permissions, and product area.

A simple way to combine keyword and vector results is reciprocal rank fusion.

def reciprocal_rank_fusion(
    ranked_lists: list[list[str]],
    k: int = 60,
) -> list[str]:
    scores: dict[str, float] = {}

    for ranked_list in ranked_lists:
        for rank, doc_id in enumerate(ranked_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)

    return sorted(scores, key=lambda doc_id: scores[doc_id], reverse=True)

You can then pass the fused list to a reranker.

Why this works:

Keyword search rescues exact matches that embeddings can blur. Vector search rescues paraphrases that keyword search misses. Fusion gives you both.

⚠️ Gotcha:

If your corpus contains product names, error codes, SKUs, ticket IDs, or legal clause numbers, pure vector search will disappoint you. Those are often exact-match problems.

4. Similarity is not evidence

Scenario:

The top retrieved chunk is about refunds. The user asked about refunds. The chunk is relevant. But it does not contain the answer to the specific question: “Can I get a refund after 60 days?”

The model sees relevant text, but not sufficient evidence. It guesses.

Why it matters:

Retrieval systems often optimize for similarity, but the LLM needs evidence.

Those are not the same thing.

A chunk can be:

• topically relevant but answer-irrelevant;
• relevant but outdated;
• relevant but incomplete;
• relevant to a different product tier;
• relevant but contradictory to another chunk.

If your retrieval pipeline stops at “these chunks are similar to the query”, it has not done enough.

Solution:

Add a reranking and evidence-selection stage.

A cross-encoder reranker is a common approach: it scores query-chunk pairs more carefully than bi-encoder vector similarity can.

from dataclasses import dataclass

@dataclass
class CandidateChunk:
    chunk_id: str
    text: str


def rerank_candidates(
    query: str,
    candidates: list[CandidateChunk],
    cross_encoder,
    top_k: int = 6,
    min_score: float = 0.30,
) -> list[CandidateChunk]:
    scored: list[tuple[float, CandidateChunk]] = []

    for candidate in candidates:
        score = cross_encoder.predict(query, candidate.text)
        if score >= min_score:
            scored.append((score, candidate))

    scored.sort(key=lambda item: item[0], reverse=True)
    return [candidate for _, candidate in scored[:top_k]]

The important design decision is this: retrieve broadly, rerank narrowly.

For example:

• vector search returns 100 candidates;
• keyword search returns 100 candidates;
• fusion produces 50 candidates;
• reranker selects 5 to 8 chunks for the LLM.

Why this works:

The first-stage retriever optimizes recall. The reranker optimizes precision. The LLM gets fewer, better chunks instead of a noisy pile.

Retrieval design	Strength	Weakness	Best use
Vector-only	Simple and semantic	Misses exact terms and fine relevance	Prototypes
Hybrid search	Better recall	Still needs ranking discipline	Most production systems
Hybrid + reranker	Strong evidence selection	More latency and complexity	Customer-facing RAG
Agentic retrieval	Can plan multi-step evidence gathering	Harder to control	Complex analytical questions

🔍 Why this matters:

If your answer quality improves when you reduce the number of retrieved chunks, your problem is probably evidence selection, not context length.

5. Metadata is where trust and permissions live

Scenario:

An employee asks the internal assistant about salary bands. The correct document exists. But the assistant retrieves a document from another department, another country, or a draft that was never approved.

Or worse: it retrieves something the user should not be allowed to see.

Why it matters:

Vector databases are often treated as if similarity is the only query dimension. In real systems, retrieval must also respect:

• tenant boundaries,
• user groups,
• document status,
• product version,
• geography,
• customer tier,
• publication date,
• and access control rules.

If metadata is an afterthought, your retrieval pipeline becomes a security and correctness problem.

Solution:

Model retrieval filters as first-class citizens.

Every indexed chunk should carry metadata such as:

{
    "doc_id": "policy-123",
    "tenant_id": "acme",
    "acl_groups": ["hr", "managers"],
    "status": "published",
    "product": "billing",
    "version": 4,
    "effective_at": "2026-01-01T00:00:00Z",
    "superseded_at": None,
}

Then build filters from the user context and request context.

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class UserContext:
    tenant_id: str
    groups: frozenset[str]


def authorized_retrieval_filter(user: UserContext, as_of: datetime) -> dict:
    return {
        "tenant_id": user.tenant_id,
        "acl_groups_overlap": list(user.groups),
        "status": "published",
        "effective_at_lte": as_of.isoformat(),
        "not_superseded_at": as_of.isoformat(),
    }

The exact query syntax depends on your vector database, but the architectural idea is the same: retrieval must be authorized before it is ranked.

Why this works:

It prevents the model from seeing information it should not use. That is better than trying to make the model “be careful” after the fact.

🧠 The important part:

If access control is enforced only in the UI, but not in retrieval, your RAG system can become a permission bypass machine.

6. Old versions quietly poison the index

Scenario:

Your company updated the refund policy in 2026. The old 2024 policy is still in the vector index because nobody removed it. The user asks about refunds. The retriever returns both versions. The model picks the older one because it ranks slightly better.

Now your assistant confidently gives outdated advice.

Why it matters:

Retrieval systems are sensitive to semantic match, not historical truth. If multiple versions of a document exist, the index does not automatically know which one is authoritative.

This problem shows up in:

• policies,
• pricing pages,
• API docs,
• runbooks,
• legal terms,
• release notes,
• internal SOPs.

The failure is especially nasty because the retrieved document may be “right” in a historical sense. It is just wrong for the current question.

Solution:

Version your documents and filter by time.

Use fields like:

• effective_at,
• superseded_at,
• is_current,
• doc_version,
• source_updated_at.

Then query with a point-in-time filter.

def current_version_filter(as_of: str) -> dict:
    return {
        "effective_at_lte": as_of,
        "superseded_at_gt_or_null": as_of,
    }

If your system supports it, prefer explicit version graphs:

{
    "doc_id": "refund-policy",
    "version": 5,
    "supersedes": "refund-policy-v4",
    "effective_at": "2026-02-01T00:00:00Z",
}

When a new version is published, mark the old version as superseded instead of leaving both equally retrievable.

Why this works:

The retrieval pipeline stops treating stale knowledge as equally valid. The LLM sees the version that is active for the relevant time window.

💡 Practical note:

“Delete old documents” is often not enough. You may need historical answers for old incidents, audits, or customer disputes. Versioning beats deletion.

7. The user query is not the real question

Scenario:

A user asks:

“Why did my deploy break?”

The actual logs say:

“Pipeline failed due to container image pull timeout.”

The user’s vocabulary and the system’s vocabulary do not match. The retriever searches for “deploy break” and returns generic deployment docs instead of the relevant incident record.

Why it matters:

Users ask questions using their own mental model. Documents are written using the author’s mental model. Retrieval has to bridge that gap.

If you send the raw user query directly to the index every time, you are assuming the user knows the correct terminology. In production, that assumption fails constantly.

Solution:

Transform the query before retrieval.

This can be as simple as deterministic expansion, or as sophisticated as an LLM-assisted query planner.

A safe starting point is to create a structured query object:

from dataclasses import dataclass, field


@dataclass
class RewrittenQuery:
    canonical_query: str
    expansions: list[str] = field(default_factory=list)
    filters: dict = field(default_factory=dict)

Then build multiple retrieval queries from it.

def rewrite_deploy_question(raw_query: str) -> RewrittenQuery:
    return RewrittenQuery(
        canonical_query=raw_query,
        expansions=[
            "deployment failure",
            "CI pipeline error",
            "release pipeline timeout",
            "container image pull failure",
        ],
        filters={
            "doc_type": ["incident", "runbook", "log_explanation"],
        },
    )

For more advanced systems, you can use the LLM to generate:

• a normalized question,
• likely synonyms,
• missing entities,
• subquestions,
• and metadata hints.

But do not let query rewriting become a black box. Log the rewritten query and use it in evaluation.

Why this works:

It increases recall across vocabulary mismatch. The retriever no longer depends on the user accidentally using the same words as the documentation.

⚠️ Gotcha:

Query rewriting can drift into hallucinated constraints. Always ground rewrites in the user’s original intent and validate them with retrieval evals.

8. Duplicates make retrieval confidently wrong

Scenario:

Your corpus contains the same onboarding guide copied across five product folders. The user asks an onboarding question. The top five results are all near-duplicates of the same paragraph. A better answer from a different document never makes it into the context.

Why it matters:

Duplicate content distorts ranking.

If the same idea appears many times, it can look artificially important. The retrieval system may return multiple variants of the same chunk, reducing diversity and crowding out complementary evidence.

This happens when:

• documents are copied across spaces;
• multiple versions are indexed;
• templates generate repeated text;
• web crawls include repeated headers, footers, and navigation;
• support articles are duplicated for different brands.

The result is not just inefficiency. It is biased evidence selection.

Solution:

Deduplicate at ingestion and retrieval.

At ingestion, exact duplicates can be caught with content hashes.

import hashlib


def chunk_signature(text: str) -> str:
    normalized = " ".join(text.lower().split())
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

Store the signature with the chunk and skip or canonicalize duplicates.

def dedupe_chunks(chunks: list[dict]) -> list[dict]:
    seen: set[str] = set()
    unique: list[dict] = []

    for chunk in chunks:
        signature = chunk_signature(chunk["text"])
        if signature in seen:
            continue

        seen.add(signature)
        unique.append(chunk)

    return unique

For near-duplicates, exact hashing is not enough. You may need:

• canonical document IDs,
• section-level deduplication,
• similarity clustering,
• or retrieval diversification.

A simple diversification rule is: do not return more than one chunk from the same document unless they contain clearly different sections.

Why this works:

It improves evidence diversity. The LLM gets a broader set of relevant facts instead of five copies of the same fact.

9. If you only evaluate final answers you are blind

Scenario:

The assistant gives a wrong answer. The team debates whether the prompt is bad, the model is bad, or retrieval is bad. Nobody knows, because the only thing being measured is the final response.

Why it matters:

RAG systems have multiple failure stages. If you only evaluate the final answer, you cannot tell whether:

• the right document was missing;
• the right document was retrieved but ranked too low;
• the right chunk was retrieved but lacked context;
• the reranker removed the correct chunk;
• the prompt ignored the evidence;
• or the model hallucinated despite good evidence.

Those require different fixes.

Solution:

Evaluate retrieval separately from generation.

At minimum, build a dataset where each question has:

• expected answer,
• required evidence chunks or documents,
• and known negative chunks that should not be retrieved.

Then measure retrieval quality directly.

def retrieval_recall_at_k(
    retrieved_ids: list[str],
    relevant_ids: list[str],
    k: int,
) -> float:
    if not relevant_ids:
        return 1.0

    retrieved = set(retrieved_ids[:k])
    relevant = set(relevant_ids)
    return len(retrieved & relevant) / len(relevant)

Other useful metrics include:

• precision@k: how many retrieved chunks were actually useful;
• MRR: how early the first correct chunk appears;
• evidence coverage: whether all required facts are present;
• context contamination: whether wrong or outdated chunks appear;
• permission violations: whether unauthorized chunks were retrievable.

A practical eval case might look like this:

{
    "question": "What is the refund window for Enterprise plans?",
    "relevant_chunk_ids": ["policy-v5-enterprise-refunds"],
    "forbidden_chunk_ids": ["policy-v3-legacy-refunds"],
    "expected_answer_contains": ["90 days"],
}

Why this works:

It separates retrieval failures from generation failures. That makes debugging possible instead of speculative.

🔍 Why this matters:

If your retrieval recall@5 is poor, no amount of prompt engineering will reliably save the system.

A retrieval contract for production RAG

The way to stop being surprised by RAG failures is to define a retrieval contract.

A retrieval contract is the set of guarantees your pipeline must satisfy before the LLM is allowed to see anything.

Before shipping a production RAG system, I would want answers to these questions:

Ingestion

• Are tables preserved in a model-readable format?
• Are headings and section paths retained?
• Are headers, footers, and navigation boilerplate removed?
• Are code blocks preserved as code, not flattened prose?

Chunking

• Can each chunk be understood without reading the previous chunk?
• Does each chunk carry document and section context?
• Are small chunks used for retrieval and larger chunks for generation where appropriate?
• Are chunks connected to parent documents?

Retrieval

• Is vector search combined with keyword search for exact terms?
• Is there a reranking stage?
• Is top-k selected based on evidence quality, not just similarity score?
• Are duplicate and near-duplicate chunks controlled?

Metadata and trust

• Are permissions enforced in retrieval filters?
• Are tenant boundaries enforced?
• Is document status included: draft, published, archived?
• Are effective dates and supersession handled?

Query handling

• Is the raw query transformed or expanded?
• Are filters inferred safely?
• Are rewritten queries logged and evaluated?
• Are ambiguous queries routed to clarification instead of blind retrieval?

Evaluation

• Do we know retrieval recall@k?
• Do we know how often outdated chunks appear?
• Do we know whether forbidden documents are retrievable?
• Do we have golden questions with required evidence chunks?

If you cannot answer those, your RAG system may work in demos, but it is not telling you the truth in production.

The most important mental shift is this:

Do not ask, “Why did the LLM hallucinate?” Ask, “What evidence did the retrieval pipeline allow the LLM to see?”

In many systems, that question reveals the real failure immediately.