You've Heard of RAG. But What Does It Actually Do?

Hello, I'm Rijul. I'm building git-lrc, a micro AI code reviewer that runs on every commit. It's free and source-available on GitHub. Star git-lrc to help more developers discover the project. Do give it a try and share your feedback

Have you been scrolling through AI content and constantly seeing the term RAG?

Maybe it sounds like some complicated, high-level AI concept that is difficult to understand.

But RAG is actually fairly simple.

And if you're trying to build AI systems, it is a concept worth understanding.

Let's look at some of the different types of AI systems we see today and understand where RAG fits in.

Types of AI Systems We See Today

There are many different ways of building AI systems today, such as:

• Generative AI
• RAG systems
• Agentic systems
• Multi-agent systems

In this article, we'll focus on RAG.

What Is RAG?

RAG stands for Retrieval-Augmented Generation.

The basic idea is simple:

Instead of asking an LLM to answer a question using only what it already knows, we retrieve relevant information and give it to the LLM along with the question.

To understand how this works, we need to look at the main components of a RAG pipeline.

Understanding the RAG Pipeline

A basic RAG pipeline looks like this:

Documents
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector database
    ↓
User question
    ↓
Question embedding
    ↓
Similarity search
    ↓
Relevant chunks
    ↓
Prompt + retrieved chunks
    ↓
LLM
    ↓
Answer

Let's break down each part.

What Is Chunking?

Suppose I have a file called policies.md.

It contains 1,000 lines of information.

I don't want to send the entire document to the LLM every time a user asks a question.

Instead, we break the document into smaller pieces.

This process is called chunking.

Chunking = breaking documents into smaller pieces that can be independently retrieved.

For example:

policies.md
    ↓
Chunk 1
Chunk 2
Chunk 3
...
Chunk 50

Now, instead of searching through or sending the entire document, we can retrieve only the chunks that are relevant to the user's question.

What Are Embeddings?

An embedding represents the semantic meaning of text as numbers.

For example, this sentence:

"What is your refund policy?"

can be converted into an embedding such as:

[0.021, -0.183, 0.442, ...]

The actual embedding contains many more numbers, but the important idea is that text is converted into a numerical representation.

This is useful because we can compare these numerical representations to determine how semantically similar two pieces of text are.

For example, these two questions use different words:

"How can I get my money back?"

and

"What is your refund policy?"

But they have a similar meaning.

Their embeddings should therefore be relatively close to each other.

Understanding the Vector Database

We can store the chunks and their embeddings in a vector database.

We also keep information about where the chunk came from.

Something like:

{
    "text": "Customers can request a refund within 30 days...",
    "embedding": [...],
    "source": "policies.md"
}

So the vector database gives us a way to store our knowledge and later search for information based on semantic similarity.

What Does Retrieval Mean?

Now let's see how retrieval actually works.

Suppose the user asks:

What is the refund policy?

First, we convert the question into an embedding.

Then we search the vector database using that embedding.

The database might return something like:

1. policies.md
   "Customers can request a refund within 30 days..."

2. policies.md
   "Subscriptions can be cancelled at any time..."

3. pricing.md
   "The Pro plan costs $29/month..."

The first result is highly relevant to the question.

Finding these relevant pieces of information is called retrieval.

How RAG Changes the Prompt

Now we have the relevant information.

We can add it to the prompt that we send to the LLM.

Before

User: What is the refund policy?

The model has to answer using only the information it already has.

After

System:
Answer using the provided context.

Context:
Customers can request a refund within 30 days...

Question:
What is the refund policy?

Now the LLM has the relevant information available in its context.

It can generate:

Customers can request a refund within 30 days.

And that's essentially what Retrieval-Augmented Generation means.

Retrieval

Find relevant information.

Augmentation

Put that information into the prompt.

Generation

Ask the LLM to generate an answer using that information.

That's RAG.

Building Your Own RAG: A Simple Example

Now let's take a simple use case and see this in action.

I have a small repository that you can clone and try yourself.

Suppose I have a chatbot that is supposed to answer questions about a company.

The project contains a documents folder with four files:

• Company
• Policies
• Pricing
• Products

The chatbot uses Gemini as the underlying LLM.

First, Let's See the Problem Without RAG

First, activate the virtual environment:

source .venv/bin/activate

Then run the chatbot without RAG:

python chat.py --no-rag

Now I'll ask:

What is the company's refund policy?

As you can see, the response is generic.

The chatbot doesn't have access to the company's actual information.

Let's try another question:

Who is the CEO?

Again, we get a generic response.

The problem is that the LLM doesn't have our company's documents in its context.

Now Let's Run the RAG Version

Now let's run the RAG version of the chatbot:

python chat.py

We'll ask the same question:

What is the company's refund policy?

This time, you can see the retrieved chunks.

The system first searches the documents and finds the pieces of information that are relevant to the question.

It then provides those chunks to the LLM so it can generate an answer based on the company's actual information.

Now let's ask something that isn't present in the knowledge base:

Who is the CEO?

This time, the chatbot can recognize that the information isn't available in the knowledge base instead of simply making up an answer.

That's one of the useful properties of RAG.

Wrapping Up

We have built and demonstrated a basic RAG system.

Along the way, we got familiar with concepts such as:

• Chunking
• Embeddings
• Vector databases
• Retrieval
• Knowledge bases
• Retrieval-Augmented Generation

Once we break the process down, these concepts are much simpler than they might initially seem.

Of course, this is only a basic example of RAG.

There are many deeper topics to explore, such as different chunking strategies, embedding models, retrieval methods, reranking, hybrid search, and evaluation.

But for now, you have a basic understanding of what RAG actually does and how the pieces fit together.

See you in the next article.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github