What RAG Actually Is (And Why It’s More Than Search + LLM)

In the previous article, we explored why Large Language Models hallucinate and why RAG exists.

We learned that LLMs don’t magically know everything. Their knowledge is limited to what they learned during training, and that knowledge eventually becomes outdated. RAG helps solve this problem by providing relevant external information at the time of answering a question.

But that raises another question - What actually happens inside a RAG system?

When people talk about RAG, terms like chunking, embeddings, vector databases, hybrid search, reranking, and evaluations often appear together. For someone starting their journey, it can feel overwhelming.

So before diving into each of those topics, let’s first understand the big picture.

What Does RAG Actually Mean?

RAG stands for: Retrieval + Augmentation + Generation

Each word represents a step in the process.

Retrieval — Find information relevant to the user’s question.

Augmentation — Add the retrieved information to the prompt or context sent to the model.

Generation — The model uses that context to generate a response.

At a very high level, the flow looks like this:

The Key Idea Behind RAG

One important thing often gets misunderstood. RAG does not teach new knowledge to the model. The model itself remains unchanged. Instead, RAG changes the information available during generation.

Think about asking a colleague a question. If they don’t have the required information, they may guess. If you provide them with the relevant document before they answer, the quality of the answer improves dramatically.

Their intelligence didn’t change. The information available to them changed.

RAG follows the same principle.

RAG Architecture

At first, the previous diagram made RAG look surprisingly simple. Retrieve some information. Attach it to the prompt. Generate an answer. Done.

But when I started exploring production-grade RAG systems, I realized there was much more happening behind the scenes. Most real-world systems consist of two major pipelines:

1. Ingestion Pipeline
2. Retrieval (Inference) Pipeline

Let’s look at both.

The Ingestion Pipeline

Before users can ask questions, the system must prepare its knowledge base.

A simplified ingestion pipeline looks like this:

Documents -> Chunking -> Embeddings -> Searchable Storage

Documents

These can be PDFs, support articles, product catalogs, internal documentation, knowledge-base articles, release notes, or any other source of information.

Chunking

Large documents are split into smaller pieces so that relevant information can be retrieved efficiently.

Embeddings

Each chunk is converted into a machine-understandable representation that can later be searched based on meaning.

Searchable Storage

The processed chunks are stored in a system optimized for retrieval.

At this point, the knowledge is ready to be used.

The Retrieval Pipeline

Now imagine a user asks a question. The retrieval pipeline comes into action.

User Question -> Retrieve Relevant Context ->Prompt Construction ->LLM -> Response

In production systems, however, several additional steps often exist between these stages. Questions may be transformed. Results may be filtered. Retrieved content may be reranked. Citations may be attached. Responses may be evaluated for quality and faithfulness.

A more realistic view looks something like this.

The Part That Surprised Me

At this point, I felt I finally understood RAG. Then every box in the architecture started raising new questions.

1. Why do we need chunking?
2. What exactly are embeddings?
3. Why are vector databases everywhere?
4. Why isn’t a traditional database enough?
5. Why do production systems use hybrid search?
6. Why do people talk about rerankers?
7. And if all the required information already exists, why do retrieval systems still fail sometimes?

The deeper I went, the more I realized that RAG is not simply “search + AI.” Each component exists because it solves a specific problem.

Next Question

While studying these architectures, one question kept coming back to me. If the goal is to provide context to the model, why don’t we simply give it all the available documents? Why retrieve only a small subset? Why not let the model read everything and figure out what’s relevant on its own? It sounds like a reasonable idea. In fact, that was my first thought as well.

As I learned later, this approach breaks surprisingly fast because of context-window limitations, cost, latency, and relevance challenges. And understanding those limitations is exactly what led me to the next topic in this series.

Why Can’t We Just Give the LLM All the Available Context Instead of Retrieving Only What’s Relevant?

That’s what we’ll explore in the next article.