The RAG Pattern I Keep Rebuilding: Dynamic Routing Instead of One-Shot Retrieval

Most RAG tutorials end exactly where production begins.

They show you chunking a document, embedding it, stuffing it into a prompt. It works beautifully on the demo. Then you drop it into an agent that needs to answer multi-hop questions, compare across documents, or decide whether it needs to retrieve at all — and the whole thing falls apart.

The problem isn't retrieval accuracy. It's that most RAG implementations treat it as a one-shot preprocessing step, when agents need it to be a runtime decision.

Here's the pattern I've landed on after building and rebuilding this a few times: dynamic retrieval routing.

The Gap Between Tutorial RAG and Agentic RAG

In a standard RAG setup, you retrieve once at the start of the conversation. You embed the user's query, find the top-k chunks, and hand them to the model. Done.

In an agentic system, that breaks down for three reasons:

1. The agent doesn't know what it doesn't know — it can't decide to retrieve more if it already answered from stale context.
2. Different queries need different strategies — some need dense vector search, others need BM25 keyword matching, others need a web search.
3. Retrieval timing matters — calling retrieval too early wastes tokens; too late and the agent is reasoning on guesswork.

The fix is to push retrieval into the agent's decision loop, not just its preamble.

A Retrieval Router in Practice

Here's the architecture I've been running. The agent gets a route_and_retrieve tool instead of a raw retrieval tool. Based on the query type, it picks the right retrieval strategy.

from enum import Enum
from typing import Literal

class RetrievalStrategy(Enum):
    VECTOR = "vector"
    KEYWORD = "keyword"
    WEB = "web"
    HYBRID = "hybrid"

def route_query(query: str) -> RetrievalStrategy:
    """Classify the query to pick the right retrieval strategy."""
    classify_prompt = f"""Classify this query by retrieval need:

Query: {query}

Options:
- vector: factual questions about stored documents
- keyword: specific entity lookups (names, dates, numbers)
- web: current events, anything after knowledge cutoff
- hybrid: comparison or multi-hop questions spanning sources

Respond with only the strategy name."""

    result = llm.invoke(classify_prompt).content.strip().lower()

    if result not in [s.value for s in RetrievalStrategy]:
        return RetrievalStrategy.HYBRID
    return RetrievalStrategy(result)

def retrieve(query: str, top_k: int = 5) -> list[str]:
    strategy = route_query(query)

    if strategy == RetrievalStrategy.VECTOR:
        return vector_search(query, top_k)
    elif strategy == RetrievalStrategy.KEYWORD:
        return bm25_search(query, top_k)
    elif strategy == RetrievalStrategy.WEB:
        return web_search(query)
    else:  # HYBRID
        vector_results = vector_search(query, top_k)
        keyword_results = bm25_search(query, top_k)
        return merge_and_rerank(vector_results, keyword_results, query)

The agent calls retrieve(question) as a tool, gets context back, and decides whether it has what it needs or should retrieve again with a refined query.

What Changes When Retrieval Is a Loop

Once retrieval is inside the agent's tool loop, a few things become possible that weren't before.

Self-refinement. The agent can look at what it retrieved and decide it wasn't specific enough. It can call retrieve() again with a more targeted query — something you can't do in a one-shot RAG setup.

Strategy switching mid-task. A comparison question might need vector search for one half and keyword search for the other. The agent can call the router twice with different sub-queries.

Conditional retrieval. Some questions don't need retrieval at all — the model's weights already cover it. The agent can skip retrieval entirely for factual recall, or for queries where it already has sufficient context from a previous turn.

Here's what that conditional logic looks like in a minimal agent loop:

def agent_loop(query: str, max_retrievals: int = 3):
    context = []
    recent_queries = []

    for i in range(max_retrievals):
        prompt = build_prompt(query, context)
        response = llm.invoke(prompt)

        if response.tool_calls:
            for call in response.tool_calls:
                if call.name == "retrieve":
                    docs = retrieve(call.arguments["query"])
                    context.append({"role": "user", "content": format_docs(docs)}")
                    recent_queries.append(call.arguments["query"])
                elif call.name == "done":
                    return response.content
        else:
            return response.content

    return "Could not resolve in max retrieval steps"

The done tool is how the agent signals it's satisfied with what it has. That might feel strange to write — you're letting the model decide when to stop — but in practice it works well when the prompt makes the stopping condition explicit.

The Failure Mode That Took Me by Surprise

The thing that bit me wasn't retrieval quality. It was context window pressure from redundant retrieval.

In a loop, it's easy for the agent to call retrieve() on semantically similar queries two turns in a row, each time appending overlapping context. After a few turns, you've burned 60% of your context on near-duplicate chunks.

My fix was a lightweight dedup step before appending new context:

def deduplicate_context(existing: list, new_chunks: list[str], threshold: float = 0.85) -> list[str]:
    """Remove new chunks that are too similar to existing context."""
    existing_text = "\n".join(existing)
    filtered = []

    for chunk in new_chunks:
        similarity = compute_similarity(chunk, existing_text)
        if similarity < threshold:
            filtered.append(chunk)

    return filtered

This isn't sophisticated — it's just cosine similarity on the full texts. But it cut my average token usage per conversation turn by about 35% in testing, and it didn't hurt answer quality.

What I Learned

The jump from "RAG as preprocessing" to "RAG as agent tool" isn't complicated architecturally. It's mostly a shift in where you're making the decision about what to retrieve.

Once I stopped thinking of retrieval as something that happens once at the start of a conversation and started thinking of it as a capability the agent invokes when it needs it, a lot of the edge cases resolved themselves.

The dynamic routing piece is the part I'd protect if I had to cut something. Even if the query classification is wrong 10% of the time, having the agent operate with an intentional retrieval strategy tends to produce better results than blindly embedding and hoping.

The dedup step was the surprise payoff. It wasn't part of the original design — I added it after watching token counts balloon on longer conversations. It's the kind of thing that's easy to skip in a tutorial but matters a lot in practice.

If you're running RAG inside an agent and you're doing it in one shot at the top, it's worth asking what a second retrieval, a different strategy, or a little deduplication would change.