Naive RAG is dead. Here is what agentic retrieval actually looks like

#ai #rag #software #llm

Retrieval stopped being a fixed pipeline you run once. It became a decision the agent makes, again and again, while it works. Here is how I got there, and what changed.

The pattern that everyone shipped in 2023

If you built a RAG system a couple of years ago, you probably built the same thing I did. Take your documents, chop them into fixed-size chunks, embed each chunk, drop the vectors into a database. At query time, embed the question, grab the top few nearest chunks, paste them into the prompt, and let the model answer.

It demos beautifully. You ask a question, it pulls the right paragraph, the answer looks great, everyone nods. Then it goes to real users and starts getting things quietly wrong.

I have watched this happen more than once. The frustrating part is that nothing throws an error. The retrieval runs, chunks come back, the model answers with total confidence. It is just answering from the wrong context, and you only find out when someone downstream notices the answer is off.

So it is worth being honest about why the 2023 pattern breaks, because the failures are not random.

Why it breaks

The first problem is that vector similarity is not the same as relevance. Two chunks can sit close together in embedding space and mean opposite things in practice. I saw this clearly on a system dealing with policy tiers, where a "Tier 1" rule and a "Tier 2" rule were nearly identical as text but had completely different consequences. The embedding could not tell them apart, so the retriever kept handing back the wrong one. Semantically close, operationally opposite.

The second problem is fragmentation. When you split on a fixed character count, a single fact often gets cut across two chunks. The top-k pulls one half, misses the other, and no reranker on earth can fix a fact that was never retrieved whole. I wrote a whole separate piece on why chunking is a modeling decision, not preprocessing, and this is the heart of it.

The third problem is the one people underestimate. Real user queries are messy, and a surprising share of them fall outside whatever distribution your embedding model was trained on. One analysis put it around 31 percent of real queries landing out of distribution, which lines up with a jump in retrieval failures. Your embeddings were tuned on clean text. Your users type fragments, typos, and half-formed questions. The gap shows up as bad retrieval.

None of these are exotic. They are the default behavior of the naive pattern, which is exactly why "just add RAG" so often disappoints.

The first real fix: stop trusting one signal

The first upgrade is to stop relying on a single retrieval signal. Dense vector search is good at meaning but bad at exact terms like names, codes, and identifiers. Keyword search is the opposite. So you run both, dense and sparse, and combine them. Then you add a reranker, usually a cross-encoder, that looks at the query and each candidate together and reorders them by actual relevance instead of raw vector distance.

That combination, hybrid retrieval plus reranking, fixes a large chunk of the "semantically close but wrong" failures. On my own systems it was the single biggest quality jump.

But here is the part I want to stress, because it is the part most people skip. You cannot improve retrieval you are not measuring. Before I trust any retrieval change, I build a small set of real questions with the chunks that should come back for each one, and I score retrieval on that set. Did the right chunk make the top-k or not. Without that, you are tuning by vibes, and every change feels like progress whether or not it is.

Where it becomes agentic

Hybrid retrieval is still a fixed pipeline though. Same steps, every query, in the same order. The real shift, the thing that actually killed naive RAG for me, is treating retrieval as a decision the agent makes rather than a step you always run.

Think about what a careful researcher does. They do not embed your exact question and grab three paragraphs. They rephrase the question into something searchable. They decide whether they even need to look something up. If the first search is thin, they try a different angle. For a complex question they break it into parts and chase each one.

Agentic retrieval is that behavior, wired into the loop. A few concrete pieces of it:

Query rewriting comes first. The user's raw question is often a bad search query, so the agent rewrites it into one or more cleaner queries before anything hits the index. This alone handles a lot of that out-of-distribution problem.

Then there is the decision of whether to retrieve at all. Not every step needs a lookup. An agent that retrieves on every turn wastes tokens and drags in noise. A better one asks whether it already has what it needs.

For hard questions, the agent plans. It splits a multi-part question into sub-questions, retrieves for each, and combines what comes back. This is where single-shot RAG falls apart and an agent loop shines, because the answer to one sub-question shapes the next search.

And when a search comes back weak, the agent can notice and try again with a different query or a different source, instead of confidently answering from thin evidence.

The mental model that stuck with me is that retrieval turned into a policy, something the agent decides based on the state it is in, rather than a fixed function you call once and trust.

Knowing when not to retrieve at all

A quick but important detour, because the industry swung to a new extreme. With million-token context windows, plenty of people now say retrieval is pointless, just put everything in the prompt.

Sometimes that is right. If the whole relevant input is small and the model needs to reason across all of it at once, long context is simpler and often better. But every token in that window is paid for on every call, so at scale it gets expensive fast, and quality can drop as key details get buried in a huge prompt. Measured comparisons have put retrieval anywhere from roughly 8 to 82 times cheaper than stuffing an equivalent corpus into context, depending on the workload.

So it is not RAG or long context. Long context for small, nuanced inputs. Retrieval for large or fast-changing knowledge. Both together for multi-hop agent reasoning, where you retrieve to narrow things down and use context to reason over what came back.

Making the answer auditable

One more piece that separates a toy from something you can put in front of users. The agent should not just answer, it should show its work. Every claim in the output should point back to the chunk it came from.

This does two things. It lets a human verify the answer instead of trusting it, and it gives you a signal when retrieval was weak, because an answer with no solid source behind it is a red flag you can actually catch. Grounding and citation enforcement is not a nice-to-have. On a system where being wrong has a real cost, it is the difference between a demo and a tool.

What this looked like in practice

When I built my Research Investigation Agent, this is roughly the shape it took. It is a small graph of steps rather than a single call. The question first gets interpreted and planned. Retrieval is hybrid, pulling from both web search and a vector store, so it is not leaning on one signal. There is a self-reflection step where the agent checks whether the evidence it gathered is actually enough, and if it is not, it goes back and retrieves again before writing anything. And the whole run is traced, so I can see which step made which decision.

None of the individual ideas there are mine. What changed for me was seeing them fit together into one thing: retrieval as an ongoing decision inside a loop, checked and repeated, not a single hopeful lookup at the start.

The short version

Naive RAG is not dead because retrieval stopped mattering. It is dead because "embed, search top-k, paste, hope" was always the weakest possible version of retrieval. What replaced it is closer to how a person actually researches. Rephrase the question. Decide whether to look. Search more than one way. Check whether the evidence holds up. Try again if it does not. Show where the answer came from.

If your RAG system feels like it is guessing, it probably is. The fix is usually not a bigger model or a longer prompt. It is giving retrieval the ability to think.

What does your retrieval stack look like right now, a fixed pipeline or a loop that can decide?