RAG from Zero to Hero: Building Naive, Advanced and Modular RAG from Scratch with Python

🎥 Watch the complete video tutorial: https://www.youtube.com/watch?v=aenpTk8blIs

In this hands-on tutorial, we'll build Naive RAG, Advanced RAG and Modular RAG step by step.

I wanted to learn Retrieval-Augmented Generation (RAG) the way Andrej Karpathy teaches machine learning: understanding every concept from first principles, building things from scratch, and knowing what actually happens under the hood.

But I struggled to find a single resource that covered the complete journey, from understanding the fundamentals to implementing more advanced RAG architectures.

After learning it myself, I thought other developers might face the same problem. So I decided to create this walkthrough.

In this article, we'll explore RAG, understand why we need it, and progressively build three types of RAG systems using Python:

• Naive RAG
• Advanced RAG
• Modular RAG

We'll also understand document chunking, vectorization, indexing, retrieval, query rewriting, reranking, routing and query fusion.

The goal isn't just to make RAG work. It's to understand why each component exists.

---

1. Why Do We Need RAG?

Large Language Models (LLMs) are impressive. They can understand questions, generate explanations, write code and perform complex reasoning.

However, they have some important limitations.

1. Hallucinations

LLMs can generate answers that sound convincing but aren't factually correct.

2. Outdated knowledge

An LLM's knowledge comes primarily from its training data. It doesn't automatically know about events, documents or information created after its training cutoff.

Consider a simple example:

If you ask an LLM:

"What happened in the latest news three days ago?"

A model without access to that information cannot reliably answer.

This is where Retrieval-Augmented Generation comes in.

What is RAG?

RAG stands for Retrieval-Augmented Generation.

The central idea is simple:

Instead of asking an LLM to answer a question using only its internal knowledge, we first retrieve relevant information from external documents and provide that information as context.

The LLM then generates an answer based on the retrieved context.

The basic workflow looks like this:

1. Load external documents.
2. Split documents into smaller chunks.
3. Convert chunks into numerical vectors.
4. Store and index those vectors.
5. Convert the user's query into a vector.
6. Retrieve relevant chunks.
7. Combine the retrieved context with the user's question.
8. Send the prompt to an LLM.
9. Generate an answer.

That's the fundamental idea behind RAG.

---

2. Understanding the RAG Architecture

Before writing code, let's understand the two main parts of a RAG system.

Indexing

Indexing happens before the user asks a question.

We load documents, split them into chunks, transform those chunks into vectors and store them in an index.

Retrieval and Generation

When a user asks a question, we transform the query into a vector, search for relevant chunks, construct a prompt and send it to the LLM.

The overall architecture:

Documents → Chunking → Vectorization → Indexing

User Query → Vectorization → Retrieval → Context + Query → LLM → Answer

This is the foundation we'll use throughout the implementation.

---

3. Building Our First RAG System: Naive RAG

Let's implement everything ourselves.

For this walkthrough, I'll use a document containing the story of The Amazing Spider-Man.

Why Spider-Man? Because it's an easy example to understand, and we can ask questions about its characters and story.

Our goal is to answer questions using information from this document.

Step 1: Load the document

First, we load our text document using Python.

We can inspect its contents and calculate its length.

Imagine the document contains around 12,000 characters.

One approach would be to send the entire document directly to an LLM along with every question.

But that's inefficient.

• We waste tokens on irrelevant information.
• Long documents increase the prompt size.
• The model receives more information than it needs.

Instead, we need to divide the document into smaller pieces.

Step 2: Document chunking

Chunking means splitting a document into smaller sections.

For example, we might initially choose a chunk size of 200 characters.

Each chunk becomes a smaller unit that can be independently retrieved.

However, simple character-based chunking creates a problem.

Consider this sentence:

"Peter Parker grows up with questions about his family when they disappear under mysterious circumstances."

If we split this sentence at an arbitrary character position, we might get:

Chunk 1: "Peter Parker grows up with questions about his family when they disappear under myster..."

Chunk 2: "...ious circumstances."

Now the second chunk has lost important context.

This is a problem because retrieval might return incomplete information.

Improving chunking with paragraphs and overlap

Instead of blindly splitting the document every 200 characters, we can first divide it into paragraphs.

Then we combine paragraphs until we approach our desired chunk size.

We also introduce overlapping chunks.

Overlap means carrying a small portion of the previous chunk into the next chunk.

For example:

• Chunk size: 200 characters
• Overlap: 50 characters

This preserves some surrounding context between chunks.

We also need to avoid cutting words in the middle. If a chunk boundary falls inside a word, we can move the boundary to a nearby whitespace character.

This gives us more meaningful chunks.

Our initial implementation produces around 61 chunks, while the improved chunking approach produces around 63 chunks.

The exact number depends on the document and chunking configuration.

The important lesson is that chunking is not just about dividing text. It's about preserving meaning.

---

4. Vectorization: Converting Text into Numbers

Computers cannot directly perform mathematical similarity searches on ordinary text.

We need to represent our chunks numerically.

This is where vectorization comes in.

In this implementation, we'll use TF-IDF.

What is TF-IDF?

TF-IDF stands for Term Frequency-Inverse Document Frequency.

It combines two ideas:

TF: Term Frequency

How frequently does a word appear in a document?

IDF: Inverse Document Frequency

How uncommon is that word across the collection of documents?

Words that appear frequently in a particular document but are relatively uncommon across other documents can receive higher importance.

Common words such as "the", "is" and "and" generally provide less useful information for distinguishing documents.

TF-IDF converts text into numerical feature vectors based on these word statistics.

We'll use Scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(
    stop_words="english"
)

vectors = vectorizer.fit_transform(chunks)

The result is a sparse matrix.

What is a sparse matrix?

A sparse matrix is a matrix in which most elements are zero.

Instead of storing every zero value, sparse representations efficiently store the non-zero values.

This saves memory when working with high-dimensional text vectors.

For our implementation, we'll convert the vectors into dense NumPy arrays before passing them to FAISS.

Important distinction: TF-IDF is a lexical vectorization technique. It represents word importance, not deep semantic meaning. Modern production RAG systems commonly use dense embedding models for semantic retrieval.

---

5. Indexing with FAISS

Now we have vectors representing our document chunks.

But how do we efficiently search through them?

Imagine having thousands or millions of chunks. Searching through every document manually would be inefficient.

We need an index.

We'll use FAISS (Facebook AI Similarity Search).

FAISS is a library designed for efficient similarity search over vectors.

Think of our vectors as points in a high-dimensional mathematical space.

When a query arrives, we convert it into a vector and search for nearby vectors.

The nearest vectors represent our candidate chunks.

Normalization and cosine similarity

We also need to understand cosine similarity.

Cosine similarity measures the angle between two vectors.

A higher cosine similarity generally indicates that two vectors point in a more similar direction.

Mathematically:

[

\text{Cosine Similarity}(A,B)

\frac{A \cdot B}{|A||B|}
]

We normalize our vectors so their lengths become 1.

After normalization, the inner product between two vectors is equivalent to cosine similarity.

We'll use FAISS to create an index and add our vectors to it.

With an inner-product index and normalized vectors, we can retrieve the nearest neighbors efficiently.

---

6. Retrieval: Finding Relevant Chunks

Now comes the interesting part.

Let's ask:

"Who is Spider-Man?"

First, we transform the user's query using the same TF-IDF vectorizer.

Then we search the FAISS index.

We can request the top 3 nearest neighbors by setting:

k = 3

FAISS returns similarity scores and the indices of the nearest chunks.

Using those indices, we retrieve the original text chunks.

These chunks become our context.

Now we have:

• The user's question.
• The relevant document chunks.

We're almost ready to connect our LLM.

---

7. Connecting the LLM

We need an LLM to generate an answer using our retrieved context.

For this implementation, I use Groq through an OpenAI-compatible client.

The important part isn't the provider. It's how we construct the prompt.

A basic prompt might look like this:

Answer using only the provided context.

If the answer is not present in the context,
respond with "I don't know".

Context:
{retrieved_context}

Question:
{user_question}

We combine our retrieved chunks into a single context string and send the prompt to the LLM.

Now the model has access to information from our document.

This completes our first Naive RAG implementation.

Testing our first RAG system

Let's ask:

"Who is Spider-Man?"

Our system retrieves relevant chunks and sends them to the LLM.

The model can now answer using the information provided in the document.

However, our initial implementation also reveals some problems.

• Some chunks contain incomplete sentences.
• Retrieval can return irrelevant information.
• A user's query might not match the wording used in the document.
• The retrieved context may contain unnecessary information.

This is why we need to improve our architecture.

---

8. Advanced RAG: Improving Retrieval

Naive RAG follows a straightforward pipeline:

Query → Retrieval → Context → LLM → Answer

Advanced RAG introduces additional processing before and after retrieval.

These are commonly called:

• Pre-retrieval optimization
• Post-retrieval optimization

The objective is to improve the quality of retrieved information before generating the final answer.

A. Pre-retrieval: Query Rewriting

Imagine a user asks:

"Who is Spider-Man?"

The query is understandable, but it might not contain all the information needed for effective retrieval.

We can send the query to an LLM and ask it to rewrite the query for retrieval.

For example:

Original query:

"Who is Spider-Man?"

Rewritten query:

"Find details about Spider-Man, Peter Parker, including his identity, powers, creators and origin story."

The rewritten query provides additional search terms and context.

We then convert this new query into a vector and search our existing index.

The goal is to retrieve more relevant chunks.

This process is called query rewriting.

B. Post-retrieval: Reranking

Retrieval gives us candidate chunks, but not every retrieved chunk is equally useful.

Some chunks might be relevant, while others might contain unnecessary information.

Reranking helps us improve the order and relevance of the retrieved results.

In our implementation, we send the retrieved chunks to an LLM and ask it to:

• Identify relevant information.
• Rank the chunks according to relevance.
• Ignore irrelevant information.
• Preserve useful facts for answering the question.

For example, if we ask:

"Who is Spider-Man?"

A chunk explaining that Peter Parker is Spider-Man is much more useful than a chunk describing an unrelated event.

The reranked context is then passed to the LLM.

Advanced RAG architecture

Our improved pipeline now looks like this:

User Query → Query Rewriting → Retrieval → Reranking → Context → LLM → Answer

We've introduced two important improvements without changing the fundamental idea of RAG.

Query rewriting improves what we search for.

Reranking improves what we use after searching.

---

9. Modular RAG: Combining Different Retrieval Strategies

Now we move to Modular RAG.

This is where things become more interesting.

Instead of treating RAG as one fixed pipeline, we treat different operations as independent modules.

We can combine them depending on the problem.

In this implementation, we'll explore three important techniques:

1. Routing
2. Query Fusion
3. Reciprocal Rank Fusion

A. Routing

Does every question need document retrieval?

Not necessarily.

Imagine asking:

"Who is Spider-Man?"

An LLM already has general knowledge about Spider-Man.

But what if we ask:

"What happened to Spider-Man yesterday in my document?"

Now we need to search our external document.

Routing allows the system to decide whether it should answer directly or retrieve information from an external knowledge source.

We can ask an LLM to classify a question:

• Chat: The question can be answered using the model's existing knowledge.
• Document: The question requires external document retrieval.

The system then follows the appropriate path.

This avoids unnecessary retrieval operations for questions that don't need them.

B. Query Fusion

Traditional retrieval often searches using one query.

But a single query may not capture every possible interpretation of the user's question.

Consider:

"Who is Spider-Man?"

We can generate multiple related queries:

• Who is Spider-Man?
• What is Spider-Man's real name?
• Which comic book character is known as Spider-Man?
• What is Spider-Man's origin story?

Now we have several search queries instead of one.

We perform retrieval for each query.

Each query can discover different relevant chunks.

However, this introduces another problem: duplicate results and different rankings across queries.

We need a way to combine them.

C. Reciprocal Rank Fusion (RRF)

Reciprocal Rank Fusion is a ranking technique used to combine results from multiple ranked lists.

Instead of relying only on raw similarity scores, it considers the position of each result in different rankings.

The general formula is:

[
RRF(d)=\sum_{r \in R}\frac{1}{k+r(d)}
]

Where:

• (d) is a document or chunk.
• (R) represents the different ranked result lists.
• (r(d)) is the rank of the document in a particular list.
• (k) is a smoothing constant.

Chunks that consistently appear near the top of multiple result lists receive higher combined scores.

We can then sort the chunks and select the most relevant results.

This helps us combine information retrieved through multiple query perspectives.

Modular RAG architecture

Our modular pipeline can now include:

User Query → Routing → Query Generation → Multiple Retrieval Operations → RRF → Context → LLM → Answer

We can also combine the techniques from Advanced RAG:

• Query rewriting
• Reranking
• Retrieval
• Query fusion
• Routing

The important idea is that these components are modules. We can add, remove or combine them depending on the application.

---

10. Putting Everything Together

We've now explored three different RAG architectures.

Feature	Naive RAG	Advanced RAG	Modular RAG
Document chunking	Yes	Yes	Yes
Vectorization	Yes	Yes	Yes
Indexing	Yes	Yes	Yes
Retrieval	Yes	Yes	Yes
Query rewriting	No	Yes	Can be included
Reranking	No	Yes	Can be included
Routing	No	Not in our basic implementation	Yes
Query fusion	No	Not in our basic implementation	Yes
Flexible modules	Limited	Somewhat	Yes

The progression is straightforward:

Naive RAG: Retrieve relevant chunks and generate an answer.

Advanced RAG: Improve retrieval using query optimization and post-retrieval processing.

Modular RAG: Combine multiple independent techniques to create a more flexible retrieval system.

These aren't rigid definitions or mandatory stages. Real-world RAG architectures can combine techniques from different approaches.

---

11. What Comes After Modular RAG?

Building a basic RAG system is only the beginning.

There are several other directions worth exploring.

Some important challenges include:

• Retrieval quality
• Generation quality
• Chunking strategies
• Context relevance
• Query understanding
• Latency and cost
• Evaluation and reliability

We can also explore different optimization strategies.

Iterative RAG

The system retrieves information, generates an answer and repeats retrieval when more information is needed.

Recursive RAG

The system can break complex questions into smaller questions and retrieve information recursively.

Adaptive RAG

The system adapts its retrieval strategy depending on the query and task.

Beyond these approaches, we can explore:

• Dense embeddings
• Hybrid search
• Metadata filtering
• RAG evaluation
• Fine-tuning retrievers
• Context compression
• More advanced retrieval architectures

RAG is not a single algorithm. It's a collection of techniques that can be combined to build better knowledge-grounded applications.

---

Conclusion

We started with a simple question:

Why do we need RAG when we already have powerful LLMs?

Then we built a complete progression:

1. Naive RAG from scratch.
2. Advanced RAG with query rewriting and reranking.
3. Modular RAG with routing, query fusion and Reciprocal Rank Fusion.

Along the way, we explored chunking, TF-IDF, vectorization, sparse matrices, FAISS, normalization, cosine similarity and prompt construction.

My biggest takeaway is that understanding the individual components is much more valuable than simply knowing how to call a RAG framework.

Once you understand the fundamentals, frameworks become tools rather than black boxes.

Keep building, keep experimenting, and understand what's happening under the hood.

---

Watch the Complete Video Walkthrough

I also recorded the complete implementation, explaining the concepts and building the different RAG architectures step by step using Python.

🎥 RAG: Zero to Hero

https://www.youtube.com/watch?v=aenpTk8blIs

If you're learning LLMs, Generative AI or AI engineering, I hope this resource helps you understand RAG from the ground up.

Thanks for reading!