The assumption every RAG demo makes
Standard RAG assumes the user's question maps onto one vector search. One query in, one embedding, one top-k lookup, one answer.
That assumption holds up in demos, because demos ask demo questions. "What's our parental leave policy?" is one document. Retrieve it, stuff it into the prompt, done.
Then you ship, and a real user types: "Did the carrier rate change we approved in Q2 actually reduce our cost per shipment in the Northeast, and does that hold if I exclude the Boston depot?"
That question needs a policy document, a rate table, a transactional aggregate, and a filtered re-computation. Your retriever will embed the whole sentence, find the three chunks nearest to it in vector space, and hand the model text that is topically adjacent and factually useless. The model, being a good sport, will answer anyway.
The problem isn't the embedding model or the chunk size. You hardcoded how many times to retrieve, and where to retrieve from, at design time, for a question you hadn't read yet.
Agentic RAG moves that decision to runtime. Planners, memory, MCP servers, sub-agents: all of it is implementation detail hanging off that one change.
Architecture 1: standard RAG is a straight line
STANDARD RAG — fixed pipeline, one pass
┌──────┐ 1. prompt+query ┌─────────────┐
│ User │ ───────────────────► │ Chat UI │
└──────┘ └──────┬──────┘
▲ │ 2. query
│ 6. response ▼
│ ┌─────────────┐
│ │ Retriever │
│ └──────┬──────┘
│ │ 3. fetch (top-k, one shot)
│ ▼
│ ┌───────────────────────────┐
│ │ Knowledge Sources │
│ │ docs · PDFs · code · DB │
│ │ APIs · web index │
│ └───────────┬───────────────┘
│ │ 4. chunks
│ ┌──────▼──────┐
└──────────────────────────│ LLM │
└─────────────┘
5. prompt + query + enhanced context
The defining property is that the model is never consulted about retrieval. It receives context and produces text, and retrieval already finished by the time it runs.
That's a design choice with real advantages. One embedding call plus one vector query is cheap and predictable, latency sits in a tight distribution, and you can cache aggressively. Failures are legible: a bad answer means bad chunks, and you can go read the chunks. Your eval harness is a fixed input and a fixed output, so it works.
Standard RAG is the right architecture when your corpus is homogeneous and your users mostly want lookup rather than synthesis. Don't let anyone talk you out of it for those workloads.
Architecture 2: agentic RAG makes retrieval a decision
AGENTIC RAG — planner decides retrieval at runtime
┌──────┐ 1. prompt+query ┌─────────────┐
│ User │ ─────────────────► │ Chat UI │
└──────┘ └──────┬──────┘
▲ │ 2. query
│ 6. response ▼
│ ┌──────────────────────┐ ┌──────────────┐
│ │ Aggregator / │◄────►│ Planning │
│ │ Orchestrator Agent │ 3. │ ReAct · CoT │
│ └───────┬──────────────┘ └──────────────┘
│ │ ▲
│ │ └──────┐ ┌──────────────┐
│ │ 4. fan-out └─►│ Memory │
│ ▼ │ short · long │
│ ┌───────────────┼───────────────┐ └──────────────┘
│ ┌────▼────┐ ┌────▼────┐ ┌─────▼───┐
│ │ Agent 1 │ │ Agent 2 │ │ Agent 3 │
│ └────┬────┘ └────┬────┘ └────┬────┘
│ │ MCP servers / tool layer │
│ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ │ SQL DB │ │ Search │ │ Data │
│ │ │ │ index │ │ Explorer│
│ └─────────┘ └─────────┘ └─────────┘
│ │ 5. prompt + query + enhanced context
│ ┌──────▼──────┐
└─────────────────────│ LLM │
└─────────────┘
◄── loop back to (3) if context insufficient
Four things are different.
1. A planner sits between the query and the retriever. The aggregator agent decomposes the question before anything gets embedded. ReAct and chain-of-thought are the mechanism that turns one user sentence into a retrieval plan. The Northeast shipping question decomposes into four steps: fetch the Q2 rate change record, aggregate shipment costs by region and date, re-aggregate with the Boston depot excluded, compare.
2. Retrieval fans out across heterogeneous backends. Standard RAG normalizes everything into one vector index at ingestion time. Agentic RAG queries systems in their native language at query time: SQL against the warehouse, semantic search against the document index, a time-series query against event data. You don't embed a rate table, you query it.
3. MCP is the integration seam. When your retrievers are MCP servers rather than bespoke functions, the tool surface is declarative and swappable. You add a data source by registering a server instead of shipping a new agent build. It's a plugin architecture applied to retrieval.
4. Memory carries retrieval state across turns. Short-term memory holds what this session already fetched, so hop three doesn't re-retrieve what hop one found. Long-term memory holds durable facts about the user and their prior queries. Without it, a multi-hop system re-derives the same context every turn and you pay for it twice.
The structural difference is the loop back to the planner. Standard RAG has one retrieval pass. Agentic RAG has a loop with a termination condition, and that loop is where both the value and the risk live.
What runtime retrieval costs you
| Dimension | Standard RAG | Agentic RAG |
|---|---|---|
| Retrieval decision | Design time | Runtime |
| LLM calls per query | One | One per hop, plus planning |
| Latency | One round trip, tight variance | Accumulates per hop, long tail |
| Cost per query | Flat | Variable, roughly linear in hops |
| Determinism | High | Low, the same query can take a different plan |
| Debugging | Inspect the chunks | Reconstruct the trace |
| Eval strategy | Answer quality on fixed input | Answer quality and trajectory quality |
| Multi-hop questions | Fails silently | Handles |
| Structured data | Poor, embeddings flatten schema | Native, via SQL tools |
| Failure mode | Confident answer from wrong chunks | Loops, fans out, burns budget |
The determinism row is the one that costs most in practice. Standard RAG failures are boring and reproducible. Agentic RAG failures are interesting, which is a bad property for a production system: the same question asked twice can take different paths and produce different answers, and neither is wrong so much as differently sourced. Reproducibility was doing quiet work in your incident response process, and you give it up when you move the decision to runtime.
The sufficiency gate
Before adding another hop, check whether the context you already have is enough to answer, and put a hard ceiling on the case where it isn't. That check does more for a production agentic RAG loop than a better planner does.
MAX_HOPS = 4
TOKEN_BUDGET = 20_000
def agentic_retrieve(query: str, tools: dict) -> RetrievalResult:
plan = planner.decompose(query) # ReAct-style sub-questions
ctx, trace, spent = [], [], 0
for hop in range(MAX_HOPS):
gap = planner.next_gap(query, ctx) # what's still missing?
if gap is None:
break # sufficiency gate: stop early
tool = router.select(gap, tools) # sql | search | timeseries
chunks = tool.invoke(gap)
spent += count_tokens(chunks)
if spent > TOKEN_BUDGET:
trace.append(("halt", "budget_exceeded", hop))
break
ctx.extend(chunks)
trace.append((tool.name, gap, len(chunks)))
return RetrievalResult(
context=dedupe(ctx),
trace=trace, # emit this. always.
exhausted=len(trace) >= MAX_HOPS, # flag for review queue
)
next_gap returning None is the sufficiency gate. It lets a single-hop question cost one hop instead of four. Without it, your agentic system pays multi-hop prices on lookup questions, which are most of your traffic, and that shows up on the bill.
MAX_HOPS and TOKEN_BUDGET are not optional. An unbounded retrieval loop against a live SQL tool is a denial-of-service vector pointed at your own warehouse, and a badly worded user question is enough to trigger it.
trace is what makes the system debuggable. Emit it on every request, store it next to the answer, and make it queryable. When agentic RAG fails, the answer tells you nothing about why.
Where teams get this decision wrong
The most common mistake is adopting agentic RAG for a single-hop corpus. If your knowledge base is a few thousand support articles and users ask article-shaped questions, a planner adds latency and cost to reach the same document standard RAG would have found in one pass. Route on question shape rather than on architectural fashion.
Skipping the router is close behind. A planner handed eleven tools and no routing heuristic will explore all of them. Constrain the tool set per query class before the planner ever sees it.
Then there's treating the two architectures as either/or. A classifier in front of both paths usually works better: the cheap single-hop route for lookup questions, the agentic route for questions that decompose. If most of your traffic is lookup, most of your traffic shouldn't be paying for a planner.
Evaluation is the expensive mistake. In standard RAG, answer quality is a decent proxy for system health. In agentic RAG it isn't, because a correct answer reached through six wasteful hops is a cost and latency problem waiting for your traffic to grow. Track hops per query, tool selection precision, redundant retrieval rate, and how often the sufficiency gate fires.
Finally, teams forget which argument actually closes the case, and it isn't multi-hop reasoning. Embeddings are a lossy representation of a relational schema. If your answers require joins, aggregations, or filters, no chunking strategy saves you, and you need a tool that speaks SQL.
Key takeaways
- The difference between the two architectures is control flow, and everything else follows from it. Standard RAG decides how to retrieve at design time; agentic RAG defers that to runtime.
- Standard RAG isn't a legacy pattern. On a homogeneous corpus with single-hop questions it's cheaper, faster, more reproducible, and easier to evaluate.
- A sufficiency gate is what keeps the cost model viable, because it stops the loop early on the ordinary questions that make up most of your traffic.
- Always bound the loop.
MAX_HOPSand a token budget give you a worst case you can name. - Structured data is the clearest signal to switch. Embeddings flatten schema, so joins and aggregations need a tool that queries the database directly.
- Evaluate the trajectory as well as the answer. A right answer reached the wrong way is a latent production problem.
Top comments (0)