The Retrieval Stack Nobody Shows You: Chunking, Reranking, Filtering, and Context

Most RAG failures are not model failures.

The model did not forget how to read. The prompt is not necessarily bad. The embedding model is not always the culprit.

The answer was often lost earlier, in the part of the system nobody demos:

• how documents were parsed,
• how chunks were created,
• which chunks were allowed by filters,
• how candidates were ranked,
• and how the final context was assembled.

The model only sees the final context window. Everything before that moment decides whether the model receives evidence, noise, stale policy, duplicated fragments, or a mix of all four.

That is the retrieval stack.

TL;DR

• Chunking is not just text splitting; it defines the unit of evidence.
• Filtering is where permissions, freshness, product scope, and trust live.
• Vector search alone misses exact terms, IDs, and error codes.
• Hybrid search plus reranking usually beats naive top-k vector search.
• Context assembly is a budgeting and ordering problem, not concatenation.
• If you only evaluate final answers, you cannot debug retrieval quality.

📋 Table of Contents

• The part of RAG that actually decides quality
• 1. Chunking is a unit-of-evidence problem
• 2. Retrieve small, answer big
• 3. Filtering is where production reality lives
• 4. Hybrid search fixes semantic blind spots
• 5. Reranking turns candidates into evidence
• 6. Context assembly is budgeting not concatenation
• 7. Evaluate the stack before blaming the model
• 8. Choosing the right retrieval stack
• The checklist I would use

The part of RAG that actually decides quality

A production retrieval system is not one component. It is a pipeline.

At minimum, it usually includes:

1. document parsing,
2. chunking,
3. metadata extraction,
4. indexing,
5. query filtering,
6. candidate retrieval,
7. reranking,
8. context assembly.

Each step can quietly damage the final answer.

Parsing can destroy tables. Chunking can sever definitions from examples. Filtering can exclude the right document or include the wrong one. Ranking can promote a plausible chunk instead of the correct one. Context assembly can bury the best evidence under weaker text.

The uncomfortable truth is this:

The LLM is often the easiest part of the RAG system to replace. The retrieval stack is where the product behavior is actually designed.

1. Chunking is a unit-of-evidence problem

Scenario:

Your documentation contains a clear answer:

“For Enterprise customers, webhook retries happen every 5 minutes for up to 24 hours.”

But your chunker split the text so that one chunk says:

“For Enterprise customers…”

and another chunk says:

“…every 5 minutes for up to 24 hours.”

The retriever finds one fragment. The model guesses the rest.

Why it matters:

Chunking is usually treated as a token-limit problem: split the document into 500-token pieces, add overlap, move on.

That is too narrow.

Chunking defines the smallest unit of evidence your retrieval system can provide. If a chunk cannot stand on its own, it is not a good evidence unit.

A good chunk should usually answer:

• What is this about?
• Which document and section does it belong to?
• Does it contain enough context to be interpreted correctly?
• Can it be cited?
• Can it be filtered by metadata?

Solution:

Chunk with document structure, not just length.

For Markdown-like content, headings are a natural starting point.

import re
from dataclasses import dataclass

HEADING_RE = re.compile(r"^(#{1,4})\s+(.*)$", re.MULTILINE)


@dataclass
class Section:
    heading: str
    text: str


def split_markdown_by_heading(text: str) -> list[Section]:
    sections: list[Section] = []
    current_heading = ""
    current_lines: list[str] = []

    for line in text.splitlines():
        match = HEADING_RE.match(line)

        if match:
            if current_lines:
                sections.append(
                    Section(
                        heading=current_heading,
                        text="\n".join(current_lines).strip(),
                    )
                )

            current_heading = match.group(2).strip()
            current_lines = []
        else:
            current_lines.append(line)

    if current_lines:
        sections.append(
            Section(
                heading=current_heading,
                text="\n".join(current_lines).strip(),
            )
        )

    return sections

This is not a complete production chunker, but it shows the right instinct: preserve structure.

In a real system, you may then split long sections by paragraph, list item, table, or semantic boundary. But the heading path should stay attached to the chunk.

@dataclass
class Chunk:
    doc_id: str
    chunk_id: str
    heading_path: tuple[str, ...]
    text: str

Why this works:

The chunk keeps enough context to be understood. The heading path also helps retrieval, reranking, and citations.

💡 Practical note:

If many of your chunks begin with “This”, “It”, “The above”, or “As described earlier”, your chunking is probably creating orphaned evidence.

2. Retrieve small, answer big

Scenario:

A user asks how to configure SSO. The retriever finds a small chunk with the exact sentence the user needs. But that sentence only makes sense when the surrounding section is visible.

The model receives the right fragment and still gives a weak answer because the fragment lacks surrounding context.

Why it matters:

Small chunks are good for retrieval precision. Large chunks are often better for comprehension.

This creates a tension:

• small chunks match queries more precisely,
• large chunks give the model more coherent context.

The mistake is choosing one and forcing it to do both jobs.

Solution:

Use a parent-child or small-to-big pattern.

Retrieve using small chunks, but expand to a larger parent section before building the final prompt.

@dataclass
class StoredChunk:
    chunk_id: str
    doc_id: str
    parent_id: str | None
    text: str


def expand_to_parents(
    hits: list[StoredChunk],
    chunk_store,
    max_parents: int = 4,
) -> list[StoredChunk]:
    parent_ids: list[str] = []

    for hit in hits:
        if hit.parent_id and hit.parent_id not in parent_ids:
            parent_ids.append(hit.parent_id)

    parent_ids = parent_ids[:max_parents]

    return [chunk_store.get(parent_id) for parent_id in parent_ids]

The chunk_store here can be any persistence layer: database, object storage, in-memory map, or document store. The important part is the relationship between child and parent chunks.

Why this works:

The child chunk gives the retriever a precise match target. The parent chunk gives the model enough context to interpret the answer correctly.

This pattern is especially useful for:

• long documentation,
• legal policies,
• runbooks,
• technical specs,
• API guides,
• and procedural content.

⚠️ Gotcha:

Parent expansion can pull in too much text if parent sections are huge. Use section limits, token budgets, or secondary filtering.

3. Filtering is where production reality lives

Scenario:

A customer asks about pricing. The system retrieves a pricing document. The answer is still wrong because the document was for a different region, an old plan, or a deprecated product version.

The retrieval was semantically successful. It was operationally wrong.

Why it matters:

Real knowledge bases are not flat collections of truth. They contain:

• old versions,
• drafts,
• archived pages,
• internal-only notes,
• customer-specific policies,
• regional differences,
• environment-specific instructions,
• and access-controlled content.

If your retrieval system only understands similarity, it cannot respect those boundaries.

Solution:

Make metadata filtering a first-class part of retrieval.

Every chunk should carry metadata that describes its scope and trust level.

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class RetrievalFilter:
    tenant_id: str
    product: str
    audience: str
    environment: str
    user_groups: frozenset[str]
    as_of: datetime


def build_metadata_filter(filter: RetrievalFilter) -> dict:
    return {
        "tenant_id": filter.tenant_id,
        "product": filter.product,
        "audience": filter.audience,
        "environment": filter.environment,
        "lifecycle": "active",
        "effective_at_lte": filter.as_of.isoformat(),
        "acl_groups_overlap": list(filter.user_groups),
    }

The exact query syntax depends on your datastore, but the architectural point is consistent: filtering should happen before or alongside retrieval, not as an afterthought.

Useful metadata fields often include:

• tenant_id,
• product,
• plan_tier,
• region,
• environment,
• lifecycle,
• effective_at,
• valid_until,
• owner,
• source_type,
• acl_groups.

Why this works:

It prevents the retriever from considering chunks that are structurally inappropriate for the request.

A question from a production customer should not retrieve staging docs. An internal engineering question should not be answered with marketing copy unless that is the intended source.

🚨 Production warning:

Do not rely on the model to ignore unauthorized or irrelevant content. Filtering and access control must happen before the context is assembled.

4. Hybrid search fixes semantic blind spots

Scenario:

A user searches for error code ERR_PAYMENT_409. Vector search returns conceptually related payment failure documentation, but not the exact error reference. The answer is close, but not specific.

Why it matters:

Vector search is good at semantic similarity. It is often weak at exact matching for:

• error codes,
• identifiers,
• function names,
• config keys,
• SKUs,
• ticket IDs,
• API endpoints,
• product names,
• and rare technical terms.

Keyword search has the opposite problem. It can find exact strings but misses paraphrases.

In production, you usually need both.

Solution:

Combine keyword and vector search, then fuse the results.

One common approach is reciprocal rank fusion.

def reciprocal_rank_fusion(
    ranked_lists: list[list[str]],
    k: int = 60,
) -> list[str]:
    scores: dict[str, float] = {}

    for ranked_list in ranked_lists:
        for rank, chunk_id in enumerate(ranked_list, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1.0 / (k + rank)

    return sorted(scores, key=lambda chunk_id: scores[chunk_id], reverse=True)

You might use it like this:

vector_hits = vector_index.search(query, top_k=50)
keyword_hits = keyword_index.search(query, top_k=50)

fused_ids = reciprocal_rank_fusion([vector_hits, keyword_hits])

The exact retrieval backends do not matter as much as the design: one path handles semantic matching, the other handles lexical matching, and a fusion step produces a candidate set.

Why this works:

Hybrid search improves recall and reduces the chance that an exact technical term is missed because the embedding model generalized it away.

Retrieval mode	Strong at	Weak at
Vector search	Paraphrase and semantic similarity	Exact terms and rare tokens
Keyword search	Exact names, IDs, and codes	Paraphrase and intent
Hybrid search	Balanced recall	Needs ranking discipline

🔍 Why this matters:

If your corpus contains error codes, API routes, config keys, or legal clause numbers, pure vector search will eventually disappoint you.

5. Reranking turns candidates into evidence

Scenario:

Your retrieval system returns 20 chunks. Several are relevant. One is exactly right. But the final prompt uses the first six chunks, and the exact one is ranked ninth.

The model gets close, but not precise.

Why it matters:

First-stage retrieval is usually optimized for recall. It should not be trusted to produce the final evidence order.

Vector similarity can produce a good candidate pool, but it is not always good at deciding which candidate is the strongest answer-bearing chunk.

That is what reranking is for.

Solution:

Retrieve broadly, rerank narrowly.

A typical production shape is:

1. retrieve 50 to 200 candidates,
2. rerank them against the query,
3. keep the top 3 to 8 for the prompt.

from typing import Protocol


class Reranker(Protocol):
    def score(self, query: str, text: str) -> float:
        ...


def rerank_candidates(
    query: str,
    candidates: list[str],
    reranker: Reranker,
    top_k: int = 6,
    min_score: float = 0.0,
) -> list[str]:
    scored: list[tuple[float, str]] = []

    for text in candidates:
        score = reranker.score(query, text)

        if score >= min_score:
            scored.append((score, text))

    scored.sort(key=lambda item: item[0], reverse=True)

    return [text for _, text in scored[:top_k]]

The Reranker interface can be backed by a cross-encoder model, a hosted reranking service, or another scoring system. The important part is the pipeline behavior: candidate generation is separated from evidence selection.

Why this works:

Reranking gives the system a second chance to evaluate relevance with more interaction between query and document. It often improves precision more than increasing the number of chunks sent to the model.

💡 Practical note:

Reranking adds latency and cost. In many systems, the right tradeoff is to rerank a moderate candidate pool rather than rerank everything in the corpus.

6. Context assembly is budgeting not concatenation

Scenario:

You retrieve eight great chunks. You paste all of them into the prompt. The answer gets worse because the most important fact is buried under repeated boilerplate.

Why it matters:

The context window is not a storage unit. It is a decision surface.

Everything you include competes for attention. Weak chunks do not just waste tokens; they can actively confuse the model, especially when they overlap or contradict stronger evidence.

Context assembly needs to answer:

• How much evidence is enough?
• Which chunks are redundant?
• Which chunks should appear first?
• Should we prefer one authoritative chunk over several weaker ones?
• How do we preserve citations?
• What happens when the budget is exceeded?

Solution:

Build context with a budget and explicit ordering rules.

from typing import Callable


def assemble_context(
    candidates: list[str],
    estimate_tokens: Callable[[str], int],
    max_tokens: int,
) -> list[str]:
    selected: list[str] = []
    used_tokens = 0

    for text in candidates:
        tokens = estimate_tokens(text)

        if used_tokens + tokens > max_tokens:
            continue

        selected.append(text)
        used_tokens += tokens

    return selected

This is intentionally simple. Production systems often add:

• deduplication,
• source diversity limits,
• section diversity limits,
• authority-based ordering,
• citation IDs,
• and truncation policies.

A more deliberate ordering policy might place the strongest reranked chunk first, then add supporting chunks, then add broader context only if space remains.

Why this works:

It treats the prompt as a scarce resource. The goal is not to include everything relevant; it is to include the evidence that best supports a correct answer.

🧠 The important part:

If your context contains three chunks saying almost the same thing, you are probably paying tokens to increase confusion, not accuracy.

7. Evaluate the stack before blaming the model

Scenario:

The assistant gives a wrong answer. The team changes the system prompt. The answer improves for one case but breaks another. Nobody knows whether the original problem was retrieval, chunking, filtering, ranking, or generation.

Why it matters:

Final-answer evaluation alone hides the failure point.

If you do not evaluate retrieval separately, you cannot tell whether:

• the right chunk was missing,
• the right chunk was retrieved but ranked too low,
• the right chunk was filtered out,
• the chunk was too fragmented,
• the context was overloaded,
• or the model ignored good evidence.

Solution:

Evaluate each layer.

Start with retrieval metrics.

def recall_at_k(
    retrieved_ids: list[str],
    relevant_ids: list[str],
    k: int,
) -> float:
    if not relevant_ids:
        return 1.0

    retrieved = set(retrieved_ids[:k])
    relevant = set(relevant_ids)

    return len(retrieved & relevant) / len(relevant)

Useful metrics include:

Metric	What it tells you
Recall@k	Did the right evidence appear in the top results?
Precision@k	Were the top results mostly useful?
MRR	How early did the first useful chunk appear?
Filter violation rate	Did retrieval include forbidden or stale chunks?
Duplicate ratio	Are repeated chunks crowding out diverse evidence?
Groundedness	Is the final answer supported by the selected context?
Context sufficiency	Does the assembled context contain enough evidence?

A strong eval set should include:

• direct factual questions,
• multi-section questions,
• questions requiring current versions,
• questions where older documents are similar but wrong,
• permission-sensitive questions,
• exact identifier queries,
• and queries where the correct answer is not in the corpus.

Why this works:

It lets you improve the retrieval stack with evidence instead of guessing.

If recall@5 is poor, the problem is candidate retrieval. If recall@5 is good but answers are bad, the problem may be reranking, context assembly, or generation.

8. Choosing the right retrieval stack

Not every system needs the same stack.

A prototype answering questions over one small PDF collection does not need the same architecture as a multi-tenant enterprise assistant with thousands of documents, permissions, and constantly changing content.

Stack	Best for	Main strength	Main risk
Vector-only	Prototypes, small corpora	Simple and fast	Misses exact terms and weak ranking
Vector + metadata filters	Scoped products	Better safety and relevance	Requires metadata discipline
Hybrid search	Technical documentation	Better recall for exact and semantic queries	More infrastructure
Hybrid + reranking	Production assistants	Stronger evidence selection	More latency and cost
Hybrid + reranking + parent-child chunks	Complex documentation	Better comprehension	More pipeline complexity
Full stack with evals and lifecycle controls	Enterprise systems	Trustworthy and maintainable	Requires ongoing ownership

A practical way to choose:

Use vector-only when:

• the corpus is small,
• content is clean and canonical,
• questions are broad,
• and you are still learning the failure modes.

Add metadata filters when:

• content has versions,
• multiple products or tenants exist,
• permissions matter,
• or freshness matters.

Add hybrid search when:

• users search for error codes,
• identifiers matter,
• product names are exact,
• or keyword matching is clearly failing.

Add reranking when:

• top-k vector results are plausible but not precise,
• the corpus is large enough to produce many near-misses,
• answer quality depends on selecting the strongest evidence,
• and you can afford the extra latency.

Add parent-child chunking when:

• small chunks retrieve well but answer poorly,
• documents are structured,
• context is often split across adjacent text,
• or answers require procedural understanding.

The checklist I would use

Before calling a retrieval system production-ready, I would want to verify the following.

Chunking

• Chunks preserve heading or section context.
• Chunks are interpretable without reading the previous chunk.
• Long sections can expand into parent context.
• Tables, code blocks, and lists are not destroyed during parsing.

Filtering

• Every chunk has lifecycle metadata.
• Freshness is enforced.
• Tenant and permission boundaries are enforced before generation.
• Deprecated or archived content is excluded unless explicitly requested.

Retrieval

• Vector search is not the only signal.
• Exact terms, IDs, and error codes can be matched.
• Candidate retrieval is separate from final evidence selection.
• The system retrieves more candidates than it sends to the model.

Reranking

• A reranking step exists for nontrivial corpora.
• Reranked results are thresholded, not blindly accepted.
• The final context is selected from the strongest candidates.

Context assembly

• There is a token or evidence budget.
• Duplicate chunks are reduced.
• Source and section diversity are considered.
• The strongest evidence is not buried.

Evaluation

• Retrieval recall and precision are measured.
• Golden questions include expected evidence chunks.
• Stale or forbidden source hits are tracked.
• Groundedness is evaluated separately from final answer style.

The retrieval stack is not glue code. It is the part of the system that decides what the model is allowed to know.

If you get chunking wrong, the model sees fragments.

If you get filtering wrong, the model sees the wrong truth.

If you get ranking wrong, the model sees plausible noise.

If you get context assembly wrong, the model sees too much and understands too little.

The model can be impressive. But the retrieval stack is what makes it reliable.