RAG Has a Routing Problem: Knowing Which Knowledge Source to Trust

Your RAG system can retrieve the “right” document and still give the wrong answer.

The document may be relevant. It may even be accurate in its own context. But it might be the staging-environment version, a deprecated API guide, an internal design memo, a community forum reply, or a policy that was replaced six months ago.

That is not just a retrieval problem. It is a routing problem.

In multi-source RAG systems, the hard question is not only:

Which chunks are similar to the query?

It is:

Which knowledge source should this query trust, for this user, in this environment, at this point in time?

Most RAG pipelines are built as if all sources are equally valid once they enter the vector store. In production, that assumption falls apart quickly.

TL;DR

• RAG systems often fail because they retrieve from the wrong source, not because the model is weak.
• Similarity is not trust. A relevant chunk from a low-authority or outdated source can still be wrong.
• Route queries by intent, user context, environment, permissions, freshness, and source authority.
• Use explicit source manifests instead of hardcoded collection names.
• Resolve conflicting sources with provenance and precedence, not model confidence.
• Evaluate routing separately from retrieval and generation.

📋 Table of Contents

• The real problem is not retrieval alone
• 1. The one-index trap
• 2. Route intent before retrieving text
• 3. Source manifests are the missing control plane
• 4. Freshness is a trust signal
• 5. Authority should beat similarity
• 6. Permissions must route before relevance
• 7. Conflicts need provenance not vibes
• 8. Know when not to retrieve
• 9. Evaluate routing separately
• A production routing checklist

The real problem is not retrieval alone

A typical RAG pipeline looks like this:

1. User asks a question.
2. Query is embedded.
3. Similar chunks are retrieved.
4. Chunks are inserted into a prompt.
5. The model generates an answer.

That works fine when there is one canonical knowledge base and every document has the same authority.

Real systems are messier.

You might have:

• official product documentation,
• API references,
• billing policies,
• internal runbooks,
• support macros,
• resolved tickets,
• community forum posts,
• design documents,
• marketing pages,
• deprecated docs,
• and third-party integration guides.

Each of those sources has a different relationship to truth.

A support agent answering a customer should trust official billing policy more than an old resolved ticket. An internal engineer debugging an incident should trust runbooks more than public FAQs. A developer asking about an API should trust the current API reference more than a blog post from two years ago.

That is routing.

Routing decides:

• which collections or indexes to search,
• which sources are allowed,
• which sources are authoritative,
• whether freshness matters,
• whether the query needs transactional data instead of documents,
• and what to do when sources conflict.

If you skip this layer, your RAG system becomes a confidence machine built on top of an evidence problem.

1. The one-index trap

Scenario:

You index product docs, old wiki pages, Jira tickets, design proposals, and community posts into one vector store. A user asks, “Why does the webhook retry after 30 seconds?” The top result is from an abandoned design document. The answer is fluent, specific, and wrong.

Why it matters:

Vector search does not know what kind of source it is looking at. It only sees embeddings.

A deprecated design doc can be more semantically similar to a question than the current official documentation. That does not make it more trustworthy.

The one-index approach is attractive because it is simple. One ingestion pipeline. One vector store. One query path. But it collapses several different trust domains into a single retrieval surface.

Solution:

Separate sources by class before you rely on similarity.

You do not necessarily need separate databases. You can use collections, namespaces, metadata filters, or index aliases. What matters is that the system can distinguish source classes at query time.

from enum import Enum


class SourceClass(Enum):
    PRODUCT_DOCS = "product_docs"
    API_REFERENCE = "api_reference"
    BILLING_POLICY = "billing_policy"
    INTERNAL_RUNBOOKS = "internal_runbooks"
    SUPPORT_TICKETS = "support_tickets"
    COMMUNITY_POSTS = "community_posts"
    DESIGN_DOCS = "design_docs"


SOURCE_INDEX_MAP = {
    SourceClass.PRODUCT_DOCS: "docs_current",
    SourceClass.API_REFERENCE: "api_reference_current",
    SourceClass.BILLING_POLICY: "billing_policy",
    SourceClass.INTERNAL_RUNBOOKS: "ops_runbooks",
    SourceClass.SUPPORT_TICKETS: "resolved_tickets",
    SourceClass.COMMUNITY_POSTS: "community",
    SourceClass.DESIGN_DOCS: "design_archive",
}

Now routing can decide that a customer-facing billing question should not search design_archive at all.

Why this works:

It reduces cross-domain contamination. The system no longer asks, “What is similar?” before asking, “What kind of answer is this?”

💡 Practical note:

Do not over-isolate too early. Start with a small number of meaningful source classes. Ten badly defined collections are worse than three well-defined ones.

2. Route intent before retrieving text

Scenario:

A user asks, “How do I cancel my subscription?”

That question could mean several things:

• Where is the cancellation button in the UI?
• What is the refund policy?
• Does the API support cancellation?
• Is the user eligible for a special exception?
• Is this an internal support question or a customer question?

If your router treats all of those as “search the docs,” you will get answers that are technically relevant but operationally wrong.

Why it matters:

Intent determines the source.

A question about billing policy needs a policy source. A question about API behavior needs an API reference. A question about an outage needs operational runbooks. A question about a user’s account may need live account data, not documents at all.

Solution:

Add a routing layer before retrieval.

In simple systems, this can be rule-based. In more advanced systems, it can be a classifier or an LLM-based router. The important part is that the routing decision is explicit, logged, and testable.

from dataclasses import dataclass


@dataclass(frozen=True)
class RoutingDecision:
    source_classes: tuple[SourceClass, ...]
    reason: str
    require_current: bool = False


def route_by_intent(query: str) -> RoutingDecision:
    q = query.lower()

    billing_terms = {"invoice", "charge", "refund", "subscription", "cancel", "pricing"}
    api_terms = {"endpoint", "api", "sdk", "rate limit", "webhook", "token"}
    ops_terms = {"outage", "rollback", "runbook", "on-call", "incident"}

    if any(term in q for term in billing_terms):
        return RoutingDecision(
            source_classes=(SourceClass.BILLING_POLICY, SourceClass.PRODUCT_DOCS),
            reason="billing_intent",
            require_current=True,
        )

    if any(term in q for term in api_terms):
        return RoutingDecision(
            source_classes=(SourceClass.API_REFERENCE, SourceClass.PRODUCT_DOCS),
            reason="developer_intent",
            require_current=True,
        )

    if any(term in q for term in ops_terms):
        return RoutingDecision(
            source_classes=(SourceClass.INTERNAL_RUNBOOKS,),
            reason="ops_intent",
            require_current=True,
        )

    return RoutingDecision(
        source_classes=(SourceClass.PRODUCT_DOCS,),
        reason="generic_product_question",
    )

This example is deliberately simple. Production routers often combine:

• keyword patterns,
• lightweight classifiers,
• user role,
• product context,
• conversation state,
• and fallback rules.

Why this works:

Routing narrows the search space and increases the chance that the answer comes from an appropriate source. It also makes failures easier to diagnose: if the router chose the wrong source class, you know where to look.

3. Source manifests are the missing control plane

Scenario:

Your vector store has collections named docs_final, docs_final_v2, wiki_old, support_macros_2024, and internal_stuff. Nobody remembers which one is canonical. The retrieval pipeline searches all of them because nobody wants to break something.

Why it matters:

If you cannot describe a source’s purpose, owner, authority, and lifecycle, you cannot route to it safely.

Hardcoded collection names are not a control plane. They are archaeology.

Solution:

Create explicit source manifests.

A source manifest is a small metadata record that describes what a source is and how it should be trusted.

from dataclasses import dataclass


@dataclass(frozen=True)
class SourceManifest:
    source_id: str
    source_class: SourceClass
    owner: str
    authority_tier: int
    lifecycle: str  # active, deprecated, archived, draft
    environments: tuple[str, ...]
    audience: tuple[str, ...]
    freshness_sla_days: int | None
    acl_groups: tuple[str, ...]

Example:

BILLING_POLICY_SOURCE = SourceManifest(
    source_id="billing-policy-v5",
    source_class=SourceClass.BILLING_POLICY,
    owner="finance-ops",
    authority_tier=1,
    lifecycle="active",
    environments=("production",),
    audience=("support", "customers"),
    freshness_sla_days=30,
    acl_groups=("support", "finance", "customers-public"),
)

Now your router can make decisions like:

• only use sources with lifecycle == "active",
• exclude sources not meant for the current environment,
• prefer authority_tier == 1 for policy questions,
• warn if a source has missed its freshness SLA,
• restrict internal sources from customer-facing traffic.

Why this works:

Routing becomes policy-driven instead of tribal-knowledge-driven. You can test source selection because the source’s behavior is described explicitly.

⚠️ Gotcha:

Source manifests need maintenance. A stale manifest can be almost as misleading as a stale document.

4. Freshness is a trust signal

Scenario:

A customer asks about pricing. Your system retrieves a pricing page that is semantically perfect. The problem is that the page was replaced four months ago. The model gives the old price.

Why it matters:

Some knowledge is stable. Some knowledge is temporal.

Pricing changes. API limits change. Policies change. Feature availability changes. Compliance rules change. If your retrieval pipeline treats all documents as equally current, it will confidently produce outdated answers.

Solution:

Model time as part of retrieval.

Each source or chunk should carry temporal metadata:

• effective_at,
• valid_until,
• last_reviewed_at,
• review_interval_days,
• superseded_by.

Then route with an as_of timestamp.

from datetime import datetime, UTC


def temporal_filter(as_of: datetime) -> dict:
    return {
        "effective_at_lte": as_of.isoformat(),
        "valid_until_gt_or_null": as_of.isoformat(),
        "lifecycle": "active",
    }

The exact filter syntax depends on your datastore, but the concept should be consistent: retrieval should respect the time context of the question.

For many queries, as_of is simply “now.” For others, it is not.

Examples:

• “What is the current refund policy?” → use now.
• “What refund policy applied to orders in 2024?” → use a historical window.
• “What changed between the old and new API limits?” → retrieve both versions deliberately.

Why this works:

It prevents old sources from competing directly with current ones unless the query explicitly asks for historical context.

🔍 Why this matters:

Freshness is not just a cleanup task. It is a routing decision. The right source at the wrong time is still the wrong source.

5. Authority should beat similarity

Scenario:

A user asks whether a feature is supported. A community forum post says yes. The official documentation says no. The forum post is more detailed, more conversational, and ranks higher by similarity. The assistant says yes.

Why it matters:

Similarity measures topical closeness. It does not measure authority.

In production RAG, authority matters because different sources have different accountability. An official policy is accountable. A forum post is not. A current API reference is accountable. A deprecated design document is not.

Solution:

Introduce authority tiers and use them during ranking and selection.

Authority tiers should be simple and explicit.

AUTHORITY_TIERS = {
    1: "canonical_policy",
    2: "official_docs",
    3: "internal_notes",
    4: "community_content",
    5: "archived_or_generated",
}

Then combine authority with relevance, but do not let a low-authority source win simply because it is slightly more similar.

@dataclass(frozen=True)
class RetrievedCandidate:
    chunk_id: str
    text: str
    relevance: float
    manifest: SourceManifest


def rank_candidates(candidates: list[RetrievedCandidate]) -> list[RetrievedCandidate]:
    def score(candidate: RetrievedCandidate) -> float:
        if candidate.manifest.lifecycle != "active":
            return -1.0

        # Lower authority tier number means higher authority.
        authority_bonus = {
            1: 0.20,
            2: 0.12,
            3: 0.05,
            4: 0.0,
            5: -0.10,
        }.get(candidate.manifest.authority_tier, 0.0)

        return candidate.relevance + authority_bonus

    return sorted(candidates, key=score, reverse=True)

For high-stakes questions, you may need stronger rules:

• only tier 1 sources may answer policy questions;
• community content may supplement but not override official docs;
• archived sources may be used only for historical queries;
• internal notes may never be shown to external users.

Why this works:

It aligns retrieval with organizational trust. The system does not treat every piece of text as equally valid just because it is vector-searchable.

6. Permissions must route before relevance

Scenario:

An employee asks an internal assistant about a customer account. The system retrieves a mix of public documentation and private internal notes. The model uses the private notes in a response that later gets shared outside the company.

Or the inverse happens: the system retrieves documents the user cannot access, then the model awkwardly refuses after already having seen them.

Why it matters:

Access control is not a generation problem. It is a retrieval-routing problem.

If unauthorized chunks enter the prompt, you have already created risk. Trying to make the model “be careful” is not a security boundary.

Solution:

Filter sources by user, tenant, environment, and audience before retrieval.

@dataclass(frozen=True)
class RequestContext:
    user_id: str
    tenant_id: str
    environment: str
    audience: str
    groups: frozenset[str]


def allowed_manifests(
    manifests: list[SourceManifest],
    context: RequestContext,
) -> list[SourceManifest]:
    return [
        manifest
        for manifest in manifests
        if context.environment in manifest.environments
        and context.audience in manifest.audience
        and (
            not manifest.acl_groups
            or context.groups.intersection(manifest.acl_groups)
        )
    ]

The key ordering matters:

1. Determine request context.
2. Filter allowed source manifests.
3. Route by intent.
4. Retrieve only from permitted sources.
5. Generate.

Why this works:

It prevents unauthorized knowledge from becoming available to the model in the first place.

🚨 Production warning:

Do not rely on the LLM to redact, ignore, or “not mention” restricted content. The enforcement point must be before generation.

7. Conflicts need provenance not vibes

Scenario:

Two sources disagree about a refund window. One says 30 days. Another says 45 days. Both are relevant. Both are readable. The model picks one because the wording sounds more convincing.

Now your assistant is resolving policy conflicts using vibes.

Why it matters:

Multi-source RAG systems do not just retrieve noise. They retrieve contradictions.

If you do not define how conflicts are resolved, the model will resolve them for you. That is not a good default.

Solution:

Use provenance and precedence rules.

Every retrieved candidate should carry enough provenance to answer:

• Where did this come from?
• Who owns it?
• When did it become effective?
• Is it active or superseded?
• What source class is it?
• What authority tier does it have?
• Does it conflict with a higher-authority source?

A simple precedence policy might be:

1. Active policy beats active documentation.
2. Current version beats older version.
3. Official source beats community source.
4. Customer-facing source beats internal speculation when answering customers.
5. If conflict remains unresolved, disclose uncertainty or escalate.

def choose_canonical(
    candidates: list[RetrievedCandidate],
    as_of: datetime,
) -> RetrievedCandidate | None:
    valid = [
        candidate
        for candidate in candidates
        if candidate.manifest.lifecycle == "active"
    ]

    if not valid:
        return None

    valid.sort(
        key=lambda candidate: (
            candidate.manifest.authority_tier,
            candidate.manifest.source_id,
        ),
    )

    return valid[0]

This is intentionally conservative. In many real systems, you do not want the model to silently choose between conflicting policies. You want the pipeline to select the canonical source or surface the conflict.

A good answer when conflict remains might be:

“The current policy source says 30 days, but another active source says 45 days. Please verify with the policy owner.”

That is more trustworthy than a confident wrong answer.

🧠 The important part:

Conflict resolution is a product and governance decision, not just a prompt trick.

8. Know when not to retrieve

Scenario:

A user asks, “Has my refund been processed?” The system searches documentation and finds a helpful article about refund timelines. The model answers with general policy, but the user wanted account-specific status.

The source was not wrong because it was inaccurate. It was wrong because it was the wrong kind of source.

Why it matters:

Not every question should go to a knowledge base.

Some questions need:

• live APIs,
• account data,
• order status,
• telemetry,
• incident dashboards,
• human support,
• or no answer at all.

A RAG system that treats every query as a document search problem will produce polished answers to questions it should not answer.

Solution:

Route by capability, not just topic.

Define what your sources can actually provide.

from enum import Enum


class Capability(Enum):
    STATIC_KNOWLEDGE = "static_knowledge"
    ACCOUNT_LOOKUP = "account_lookup"
    LIVE_METRICS = "live_metrics"
    HUMAN_SUPPORT = "human_support"


@dataclass(frozen=True)
class CapabilityRoute:
    capability: Capability
    reason: str


def route_capability(query: str) -> CapabilityRoute:
    q = query.lower()

    if any(word in q for word in ("my refund", "my order", "my invoice", "my account")):
        return CapabilityRoute(
            capability=Capability.ACCOUNT_LOOKUP,
            reason="user_specific_state",
        )

    if any(word in q for word in ("current latency", "error rate", "status")):
        return CapabilityRoute(
            capability=Capability.LIVE_METRICS,
            reason="live_system_state",
        )

    if any(word in q for word in ("talk to a person", "human", "agent")):
        return CapabilityRoute(
            capability=Capability.HUMAN_SUPPORT,
            reason="explicit_human_request",
        )

    return CapabilityRoute(
        capability=Capability.STATIC_KNOWLEDGE,
        reason="general_question",
    )

This keeps RAG in its lane.

If the route is ACCOUNT_LOOKUP, the system should call an authorized account API. If the route is HUMAN_SUPPORT, it should hand off. If the route is STATIC_KNOWLEDGE, then document retrieval makes sense.

Why this works:

Trust is not only about selecting the right document. It is also about recognizing when a document is not the right source of truth.

9. Evaluate routing separately

Scenario:

The assistant gives a wrong answer. The team argues about whether the prompt is bad, the model hallucinated, or retrieval failed. Nobody knows because the only metric being tracked is final answer quality.

Why it matters:

Final-answer evaluation hides routing failures.

If the answer is wrong, you need to know whether:

• the router selected the wrong source class,
• the retriever searched the right source but missed the right chunk,
• the wrong source was allowed despite being outdated,
• a low-authority source outranked a canonical source,
• permissions leaked an internal document,
• or the model ignored the retrieved evidence.

Those are different failures with different fixes.

Solution:

Evaluate routing as its own subsystem.

Create test cases with:

• query,
• user context,
• environment,
• expected source classes,
• expected source IDs,
• forbidden sources,
• expected freshness constraints,
• and expected capability route.

Then measure routing quality directly.

def source_class_recall(
    expected: set[SourceClass],
    selected: set[SourceClass],
) -> float:
    if not expected:
        return 1.0

    return len(expected & selected) / len(expected)


def forbidden_source_violations(
    selected_source_ids: set[str],
    forbidden_source_ids: set[str],
) -> int:
    return len(selected_source_ids & forbidden_source_ids)

Useful routing metrics include:

Metric	What it tells you
Source class recall	Did the router include the right source types?
Source class precision	Did it include too many irrelevant source types?
Forbidden source rate	Did it search sources it should not have?
Freshness violation rate	Did it retrieve outdated sources for current questions?
Authority violation rate	Did it prefer low-authority sources for high-stakes queries?
Capability misroute rate	Did it use docs when it should have used live data or human support?

Why this works:

You can improve routing without accidentally blaming the model or the vector store.

A good routing eval dataset is worth more than a vague vibe check over a few dozen prompts.

A production routing checklist

Before trusting a multi-source RAG system in production, I would want clear answers to these questions.

Source modeling

• Do we know what each source is for?
• Do we know who owns each source?
• Do we know whether each source is active, deprecated, archived, or draft?
• Do we know which audiences each source is appropriate for?
• Do we know which sources are canonical for high-stakes answers?

Query routing

• Do we classify or route queries before retrieval?
• Can the router distinguish billing, API, operational, account-specific, and general product questions?
• Do we log the routing decision?
• Can routing decisions be replayed and evaluated?

Trust controls

• Are authority tiers defined?
• Are freshness constraints enforced?
• Are outdated sources excluded unless the query is historical?
• Are low-authority sources prevented from overriding canonical sources?
• Are conflicts detected and handled explicitly?

Access control

• Are permissions enforced before retrieval?
• Are tenant boundaries enforced?
• Are internal sources separated from customer-facing answers?
• Are environment-specific sources filtered correctly?

Capability boundaries

• Can the system tell when a question needs live data instead of documents?
• Can it route to human support when needed?
• Can it refuse when no appropriate source exists?
• Does it avoid answering account-specific questions with generic policy text?

Evaluation

• Do we evaluate routing separately from retrieval and generation?
• Do we test forbidden-source behavior?
• Do we test outdated-source behavior?
• Do we test authority conflicts?
• Do we test user-context routing?

The deeper issue is this:

RAG is not just about giving the model more context. It is about giving the model the right context from the right source under the right conditions.

If you only optimize similarity, you are building a search engine. If you want a trustworthy assistant, you need a routing layer that understands intent, authority, freshness, permissions, and capability.

That is the part that makes RAG feel reliable instead of merely impressive.