A few years ago, RAG seemed like an almost perfect way to make LLMs useful in real products.
The idea was simple: the model does not have to know everything by itself. If it lacks up-to-date or internal data, we can retrieve the relevant pieces from external sources and pass them into the context before generating an answer.
In early prototypes, this looked very convincing. You connect a knowledge base, ask a question, and the model answers not just “from memory”, but with support from the retrieved materials. For corporate assistants, documentation search, and internal AI tools, this felt like a huge step forward.
But once RAG moved from prototypes to real systems, its limitations became much more visible.
On a small and clean set of documents, RAG often works almost magically. On real data, things are messier: sources are duplicated, outdated, contradictory, poorly structured, and full of tables, links, nested sections, and different versions of the same process.
In that environment, the simple pattern of “split documents into chunks, put them into a vector database, retrieve a few similar pieces of text, and send them to the prompt” starts to work worse than expected.
That is where the skepticism came from. In 2025–2026, it has become increasingly common to hear that RAG is no longer needed: long context will replace it, agents will solve everything, and vector search was just a temporary compromise that helped models work around context window limits.
But that does not mean RAG is no longer useful. The real problem is that we treated it for too long as a simple add-on to an LLM: find similar documents, put them into the prompt, and expect a reliable answer.
In real systems, that is not enough. It is not enough to simply find text. We need to understand what context the model needs for a specific task: which sources to use, which data is still relevant, how to handle contradictions, and how to verify that there is enough information.
So the question is gradually changing. It is no longer just “How do we find relevant documents and put them into the prompt?” The better question is: “What context does the model need to solve the task reliably?”
That is where a more mature approach to RAG is being built in 2026.
Why RAG Became So Popular in the First Place
To understand why RAG started receiving criticism, it is useful to remember why it became so important in the first place.
An LLM does not have direct access to your data. It can use the knowledge it learned during training and the information you pass into the current request. Everything else — fresh documentation, internal policies, logs, project decisions, or private code — remains outside its field of view.
RAG offered a clear idea: instead of trying to bake all knowledge into the model, we can pass the right external information to it at answer time. First, the system searches connected sources for relevant materials. Then it adds them to the request context. Only after that does the model generate an answer based on the retrieved data.
This idea was strong not only technically, but also from a product perspective.
For businesses, it sounded very convincing: you do not need to train a separate model from scratch, and you do not need to move all corporate knowledge into a black box. You can connect an LLM to existing sources and get an assistant that answers based on internal materials.
For early scenarios, this was often enough. FAQ bots, documentation search, and answers over small knowledge bases could be built fairly quickly, demonstrated clearly, and deliver understandable value.
But real systems quickly exposed the limitation of this approach: “finding similar text” and “giving the correct answer” are not the same thing.
Where Naive RAG Starts to Break
When people talk about RAG problems, they usually do not mean the general idea of retrieving external context. They mean its simplest implementation. This is usually called naive RAG: documents are split into fragments, or chunks, stored in a vector database, and when the user asks a question, the system retrieves several of the most similar fragments and passes them to the model.
For a prototype, this is often enough. For a real system, not always.
The first problem is that a similar fragment is not necessarily the right one. Vector search is good at capturing semantic similarity, but semantic similarity does not guarantee accuracy. This becomes especially important in engineering and business tasks: API versions, configuration parameters, error codes, service names, tables, logs, SQL schemas, financial values, or legal wording.
For example, a user asks: “What is the timeout for this endpoint in the production environment?” The system may find a fragment that also contains the words “timeout”, “endpoint”, and “production”, but it may refer to another service, another API version, or an old configuration. Formally, the search worked: it found similar text. But from the user’s perspective, the answer is wrong.
And this is an important point: we should not blame everything on the LLM. The model may process the provided context well. The problem is that the context itself was not suitable.
The second problem is document chunking. Documents are rarely just a set of independent paragraphs. One section may refer to another. An exception may be written later. A table may explain the text above it. A definition at the beginning of the document may change the meaning of an entire later section.
When we split a document into chunks of 500 or 1000 tokens, we are doing something technically convenient, but not necessarily meaningful. For example, one fragment may say: “All clients have a limit of 100 requests per minute.” The next one may say: “Exception: for enterprise clients, the limit is configured individually.” If the system retrieves only the first fragment, the model may honestly give an incomplete or incorrect answer. Not because it is “stupid”, but because we separated the rule from the exception.
The third problem is deeper: naive RAG does not really understand the task. It usually works the same way for every question: a request comes in, the system finds something, and the model generates an answer. But real questions are different. Sometimes search is not needed at all. Sometimes the question needs clarification. Sometimes the system should search logs instead of documentation. Sometimes it first needs to understand which service is being discussed. Sometimes it needs to perform several searches and connect the results.
A question like “Why did latency increase after the latest release?” cannot be solved with one search over a knowledge base. It is already a small investigation: which release are we talking about, which metrics changed, which changes landed during that period, what appeared in the logs, whether dependencies changed, and whether there were nearby incidents.
Naive RAG works well for the task “find an answer in the documentation”. But in real products, we often expect more from it: the behavior of an analyst, developer, or engineer who does not just search for similar text, but understands what data is needed to answer the question.
Why Long Context Did Not Eliminate the Problem
When models with large context windows appeared, it seemed that RAG might actually become unnecessary. If a model can accept hundreds of thousands or even a million tokens, why search for separate fragments? Would it not be easier to pass the entire document, all documentation, or a large part of the codebase directly?
This argument became especially noticeable after the appearance of models with very long context windows. For example, OpenAI stated that GPT-4.1 in the API supports up to 1 million tokens of context.
But a large context does not automatically mean a good context.
There is a difference between a model technically being able to accept a lot of text and a model reliably finding and using the right information inside that text. For example, the study Lost in the Middle showed that answer quality may decrease when the relevant information is located in the middle of a long context, even if it is formally included in the request.
In practice, this is similar to human reading. If you give someone 300 pages of documentation and ask them to quickly answer a specific question, the fact that the answer exists somewhere inside the document does not guarantee a good result.
Long context helps. But it does not remove the need to select, structure, and explain to the model what matters.
So long context did not kill RAG. It changed its role.
Previously, RAG was often a way to work around a small context window: we could not pass everything to the model, so we retrieved a few fragments. Now external context search is needed not only because of token limits, but also because of quality, cost, speed, access control, and manageability. Even if a model can read a lot, it is not always reasonable to send everything to it.
The best context is not the largest one. The best context is the one that helps solve the task.
The Main Shift: The Problem Is Not Search, but Context Assembly
At some point, it becomes clear that RAG is not just “search plus generation”. It is a specific case of a broader task: assembling the right context for the model.
A model never answers in a vacuum. It relies on what it sees at request time: system instructions, the user’s question, chat history, retrieved documents, tool call results, API data, tables, code snippets, security constraints, output format, and source links.
If that context is assembled poorly, even a strong model can make mistakes. If it is assembled well, even a model that is not the newest can give a useful and reasonably accurate answer.
That is why the question “How do we build RAG?” is gradually becoming too narrow. In a real system, it is more important to understand what data the model needs, where to get it from, how to verify it, how to handle access rights, how to compress long documents, and how to detect when there is not enough information to answer.
This is how RAG stops being a single linear pattern and becomes part of a broader approach to context. From this shift came more mature approaches: better search, reranking, routing, agentic scenarios, GraphRAG, and long-context solutions.
First, We Started Fixing Search
The first stage of RAG’s evolution was quite pragmatic: if simple vector search often retrieves the wrong thing, then search itself needs to be improved.
Instead of relying on one search method, systems started combining several. Semantic search is good at finding fragments that are similar in meaning, while keyword search is better at preserving exact names, fields, error codes, versions, and rare terms. In real documents, both are needed: one helps preserve meaning, the other helps preserve precision.
Then came reranking. First, the system retrieves a larger set of potentially relevant fragments. Then a separate model re-evaluates them and keeps the ones that are truly useful for the question. This adds latency, but often noticeably improves answer quality.
At the same time, people started working more carefully with document fragments. Instead of simply splitting text into equal chunks, systems began adding explanations to each fragment: which section it comes from, what the original document is about, and which neighboring blocks are important for understanding it. Anthropic described this approach in Contextual Retrieval: in their experiments, adding context to fragments together with keyword search reduced failed retrievals by 49%, and by 67% when combined with reranking.
This is a good example of the right kind of evolution. We are not throwing RAG away; we are stopping ourselves from treating it like magic. Good RAG is not just a vector database. It is work with documents, metadata, structure, versions, access rights, ranking, and quality evaluation. And this is often where most of the value is.
Then RAG Stopped Being a Single Chain
The next stage of RAG’s evolution was realizing that one and the same scenario cannot answer every question equally well.
Imagine an internal AI assistant inside a company. At one moment, the user asks how a certain process works. A minute later, they ask which tasks are currently open for a project. Then they ask what changed in the latest merge request, why tests failed, which errors appeared most often in logs over the past week, or which policy applies to a specific situation.
Formally, all of these are questions to the same assistant. But behind each of them are different sources and different ways of getting the answer. Some need documentation, some need Jira, some need GitLab, some need CI logs, and some need Graylog or Elasticsearch.
If you try to solve everything through one database of document fragments, the system quickly turns into a dump. Documentation, tasks, logs, code snippets, policies, and everything else get thrown into the same place, with the hope that search will figure it out. In simple cases, this may work. But the more sources and question types you have, the harder the system becomes to manage.
That is why RAG gradually became modular. The system first tries to understand what kind of question it is dealing with, and then chooses the appropriate source and way of working with it. One request may only need documentation search. Another may require an API call. A third may need a SQL query. A fourth may require reading code or analyzing logs. And sometimes the system needs to go through several steps and assemble an answer from multiple places.
This is how RAG stops being a separate search block and becomes part of the assistant’s overall logic: understand the request, choose sources, retrieve data, assemble context, check whether there is enough information, and only then produce an answer.
This is much closer to how real corporate AI systems need to work.
Then Agents Arrived, but Not as a Replacement for RAG
When people say that agents replaced RAG, it sounds impressive, but it is not quite accurate.
An agent does not eliminate external context search. It changes who controls that search.
In classic RAG, the developer defines the sequence of actions in advance: first find relevant materials, then pass them to the model, then get the answer. In an agentic approach, the model can decide what it needs to do: whether to search at all, which tool to call, how to reformulate the query, whether the retrieved data is enough, and whether another iteration is needed.
This is especially useful in tasks where it is not clear in advance how many steps will be required. For example, a user asks: “Why has the quality of AI code review degraded over the last two weeks?” Simple RAG will try to find similar documents. An agentic scenario can behave differently: check model metrics, compare latency and error rate, find prompt changes, inspect merge requests, look at logs, generate several hypotheses, and verify which of them are supported by data.
This is no longer search as a separate operation. It is search as part of reasoning.
But it is important not to fall into the opposite extreme. Agentic RAG is not the “best version of RAG for everything”. It is more expensive, slower, harder to debug, and requires good constraints. If you need an FAQ over documentation, an agent may be unnecessary complexity.
Agents are useful where the task is truly dynamic: investigation, analysis, research, code review, incident response, and working across multiple systems. For a simple question like “how do I configure an environment variable?”, high-quality documentation search may be better, cheaper, and more stable.
Sometimes You Need a Map of Connections, Not a Fragment
There are tasks where ordinary RAG runs into a problem not because of search quality, but because of how knowledge is represented.
So far, we have been talking about how to find the right text fragment: a documentation section, a piece of a policy, an error description, part of an instruction. But not every question can be reduced to one fragment. Sometimes you need not a specific paragraph or section, but a broader picture across many documents.
For example, the question “what is written in this section?” can be solved with ordinary document search. But the question “which causes appeared most often in incidents over the quarter?” is more complex. The answer may be scattered across dozens of documents, tickets, or reports. What matters is not only the text itself, but also the connections between texts: which services were mentioned together, which teams were involved, which errors repeated, and which components depended on each other.
This is where GraphRAG appears. The idea is to stop treating documents only as a set of independent fragments and instead extract entities and relationships from them: services, teams, components, incidents, causes, and solutions. From this, a graph is built that helps answer not only “where is this written?”, but also “how is this connected?”
This is not a replacement for ordinary RAG in all cases. If you need to find a specific rule in documentation, a graph may be unnecessary complexity. But if the task requires seeing recurring patterns, dependencies, or the overall structure of a large set of documents, simple search over similar fragments is no longer enough.
GraphRAG matters not because it is another fashionable version of RAG. It shows a broader point: knowledge is not always convenient to store and retrieve as separate pieces of text. Sometimes knowledge is a network of relationships, and the model needs not only fragments, but also an understanding of how those fragments are connected.
So, Is RAG Still Relevant in 2026?
I do not think “what replaced RAG?” is the right question. It makes it sound like one technology was simply replaced by another. In practice, things are more complicated: naive RAG did not disappear, but it became part of a more mature approach to working with context.
Simple RAG can be described as an attempt to find similar text fragments and pass them to the model. This is enough for early prototypes and some simple scenarios. But in real systems, harder questions quickly appear: where to search, how to verify what was found, how to handle access rights, what to do with contradictory sources, and how to work with tasks that require not one fragment, but several steps of analysis.
In 2026, RAG is not a complete architecture by itself. It is one way to give the model the data it needs. For one task, documentation search may be enough. For another, you may need an agent, a graph of relationships, SQL, APIs, logs, metrics, or long context.
So “let’s build RAG” is too vague as a requirement. It is better to start with a different question: what task should the model solve, what data does it need, and how do we verify that the data is sufficient?
RAG is not dead. What died is the illusion that vector search over documents, by itself, turns an LLM into a reliable system.
The real task now is not just to find text, but to assemble context the model can rely on when answering.
Top comments (2)
Great breakdown! You mentioned that the simple 'chunk-and-stuff' vector search is dead. With 2026 bringing such massive context windows (like 1M+ tokens), do you see the future of enterprise RAG leaning more toward Agentic RAG (where the LLM actively plans and routes its retrieval steps) or GraphRAG (pre-computing those complex relationships during ingestion)? Or is the ultimate fintech stack going to demand a hybrid of both, despite the latency hits
Thanks a lot! Appreciate that!
I’d bet on a controlled hybrid.
GraphRAG makes sense for stable relationships you can precompute: entities, policies, dependencies, risk rules. Agentic RAG makes sense for dynamic tasks: investigations, tool use, logs, APIs, multi-step reasoning.
Long context helps, but it doesn’t replace context selection or verification — especially in fintech.
So the future stack is probably: deterministic retrieval + permissions first, graph where relationships matter, agents only where planning is really needed.