What Really Happens Inside a RAG Pipeline? (Behind the Scenes)

Most articles about Retrieval Augmented Generation ( RAG ) give you the Hollywood trailer version like User Questions -> Retrieve Documents -> LLM -> Answer.

Cute, But that skips over all the actual magic. The part where a computer sifts through millions of documents in milliseconds and somehow knows which tiny paragraph is the one you need.

This isn't another "RAG 101" article. We' are going backstage. We're pooping the hood. We're going to see what actually happens from the moment your documents gets uploaded to the moment your LLM spits out an answer.

The Pipeline Hollywood Doesn't Show you

Here's what a real production RAG system looks like behind the curtain:

The real brainwork happens before the LLM even wakes up. For unsung heroes do the heavy lifting:

• Embeddings = Turn words into math
• Similarity search - finding "close enough" matches
• ANN ( Approximate Nearest Neighbor ) - cheating (smartly) to go fast
• HNSW - the secret graph sauce

Let's crack each one open.

Step 1: Turning Language Into Numbers ( Because Computers Are Math Nerds)

Computers don't speak human. They speak numbers. So before any document can be searched, every chunk of text gets converted into a list of numbers called an embedding.

For example

"The cat is sleeping."

↓

[-0.42, 0.91, 0.17, ..., 0.54]

But there's the wild part: these aren't just a few numbers. Modern models crank out vectors with hundreds or thousands of dimensions:

Model	Dimensions
Small fry	384
Standard	768
Beefy	1024
Chunky	1536
Absolute unit	3072

Each dimension captures some tiny semantic signal learned during training. But here's the kicker: no single dimension means anything on its own. Dimension #237 isn't "cats". Dimension #918 isn't "animals". Meaning only emerges from how all the dimensions relate to each other. Think of it like a point in the space with 768 dimensions. ( Yes , your brain hurts, That's normal.)

Similar Meaning = Nearby Vectors

Imagine a simple 2D world (because we can't draw 768D):

Dogs
   🐕

Cats
    🐈

Cars                     🚗

Airplanes                          ✈️

Similar stuff clusters together. Real embedding spaces are 768D+, but the same rule applies:

• Dogs are near wolves
• Paris is near France
• Python is near programming
• Apple(fruit) and Apple(compnay) live in totally different neighborhoods depending on context

This geometric clustering is what makes semantic search possible. It's not magic - It's geometry at a scale your brain refuses to visualize.

Step 2: Your Question Gets the Same Treatment

You ask :
"How do I reset my password?"
Your query goes through the exact same embedding model:

❓ User Question

    ↓

🧮 Embedding Model

    ↓

📍 Query Vector

Now we have:

• Millions of document vectors
• One lonely query vector

The million-dollar question:
Which document vectors are closest to this query vector?

Step 3: Measuring Semantic Similarity ( a.k.a "How Close Are We?")

The go-to metric? Cosine similarity.

Wait, Why Not Just Euclidean Distance?
Imagine these vectors:

A = [1, 1]
B = [2, 2]

Euclidean distance says they're different because B is farther from the origin. But semantically? They're pointing in the exact same direction. Same vibe, different magnitude.

That's why embedding systems care more about direction than raw distance.

Cosine Similarity: The Angle Game

Cosine similarity measures the angle between two vectors:

          📍 Query
            ▲
           /
          /
         /
        /
       ●────────► 📄 Document

Angle	What It Means
0°	Basically the same thing
20°	Very similar
90°	Unrelated (orthogonal vibes)
180°	Opposite

The formula:

cos(θ) = (A · B) / (|A| × |B|)

Where A · B is the dot product and |A|,|B| are the magnitudes.

Result ranges from -1 to 1. In practice, after normalization, most similarities land between 0 and 1. higher = closer = more relevant.

Step 4: Finding the Closest Vectors ( The Brute Force Problem)

Now imagine:

• 50 million document vectors
• Each with 1536 dimentions

A naive approach? Compare your query against every single one:

❓ Query

    ↓

📄 Vector 1
📄 Vector 2
📄 Vector 3
...
📄 Vector 50,000,000

Even if each comparison is lightning-fast, 50 million x anything = too slow for real-time use.

This is exact nearest neighbor search. It guarantees the best answer, but the cost grows linearly with your data. For big systems? Unacceptable.

ANN: The Art of Smart Cheating

Enter Approximate Nearest Neighbor ( ANN ). Instead of checking everything, ANN asks a slightly different question:

Can we find vectors that are almost certainly among the nearest neighbors -- without looking at everything?

It's a tradeoff: a tiny bit of accuracy for a massive speed boost. For most RAG system? Totally worth it.

The Parking Lot Analogy

Imagine looking for your parked car.

• Brute force: Check every single parking space. Guaranteed to find it. Takes forever.
• ANN: Head straight to the section where you usually park. You might miss the absolute closet spot in rare cases, but you'll find your car way faster.

ANN works the same way. Instead of searching the entire universe of vectors, it intelligently narrows down where to look.

Different ANN strategies:

• Tree-based structures.
• Inverted file indexes ( IVF )
• Product quantization ( PQ )
• Graph-based indexes like HNSW

HNSW is the rockstar of the bunch. Let's meet it.

HNSW: Navigating a Graph Instead of Scanning Everything

HNSW = Hierarchical Navigable Small World.

Sounds like a prog-rock album title. The idea is actually pretty intuitive.
Instead of storing vectors as a boring list, HNSW organizes them into a graph. Each vector connects to its nearby neighbors:

      ●────●────●
     / \      \
    ●   ●──────●
     \  |
      ●─●────●

Searching becomes a navigation problem instead of a brute-force slog.

Multiple Layers: From Zoomed-Out to Zoomed-In

HNSW gets even fancier by stacking the graph into multiple layers:

🔝 Layer 3 (Sparse, zoomed-out view)

       ●
        \
         ●

──────────────────

🥈 Layer 2 (Medium detail)

   ●────●────●

──────────────────

🥇 Layer 1 (Dense, zoomed-in detail)

●─●─●─●─●─●─●─●

• Top layers: Fewer nodes, coarse overview
• Bottom layers: Denser, more detailed

The Search Dance

1. Start at the sparse top layer
2. Jump toward the region closest to your query
3. Move down one layer
4. Refine the search
5. Repeat until you hit the bottom
6. Return the nearest vectors found

Instead of comparing against millions of vectors, the algorithm follows a relatively small number of graph edges to reach the right neighborhood. Way fewer distance computations = way faster.

Why HNSW Is So Fast

Brute force? O(N) - work grows with every vector you add.
HNSW? More like O(log N) - work grows much, much slower.

For millions of vectors, this the different between:

• Millions of comparisons -> seconds
• Hundreds of comparisons -> milliseconds

That's why HNSW is the default in pretty much every modern vector database worth its salt.

Putting It All Together: The Full Journey

Here's what actually happens when you ask a question:

❓ Question

    ↓

🧮 Embedding Model

    ↓

📍 Query Vector

    ↓

🕸️ HNSW Graph Traversal

    ↓

🔍 Approximate Nearest Neighbors

    ↓

📐 Cosine Similarity Ranking

    ↓

🏆 Top-K Chunks

    ↓

📝 Prompt Construction

    ↓

🤖 LLM

    ↓

💬 Answer

Notice: the LLM doesn't search your data directly. Almost all the heavy lifting happens before the LLM is even invited to the party.

The quality of your retrieval often matters more than using a bigger, fancier language model. A great retriever with a decent LLM beats a bad retriever with GPT-5