RAG Explained Like You're Five (With Real Code You Can Actually Run)

Okay so last time I wrote about RAG and a bunch of people messaged me saying "cool but I still don't get what's actually happening behind the scenes." Fair. I threw around words like embeddings and vector search without really slowing down to explain them properly. So this time I'm doing it differently.

No jargon dump. Real life example first. Then a real, working, copy paste code example you can run on your own laptop today, even if you've never touched AI stuff before. By the end of this you should actually understand what's happening, not just be able to repeat the word "RAG" at a job interview.

Grab a coffee, this one's long.

First, a real life example that has nothing to do with computers

Imagine you're taking an exam. Two versions of the exam exist.

In version one, it's closed book. You have to answer everything purely from memory. If you forgot something you studied three months ago, tough luck, you're guessing now. And when you guess, you don't say "I'm not sure." You just write something that sounds confident because that's what gets you marks in a normal exam.

In version two, it's open book. You're allowed to bring your notes. When a question comes up, you flip to the right page, read the relevant paragraph, and write your answer based on what's actually written there.

A regular LLM like plain ChatGPT answering your question is basically doing the closed book exam. It's answering from whatever it memorized during training, which happened at some point in the past and doesn't include your personal documents, your company's internal wiki, or anything private.

RAG is what happens when you hand the model an open book exam instead. Before it answers, you go find the right page from your own documents and put it in front of the model. Then you ask the question. The model reads what's in front of it and answers based on that, instead of guessing from memory.

That's genuinely the whole idea. RAG stands for Retrieval Augmented Generation, which is just a fancy way of saying "look stuff up, then generate an answer using what you looked up."

A second real life example, this time with a librarian

Picture walking into a massive library and asking the librarian a question like "what does our company's return policy say about opened electronics."

A librarian who just answers off the top of their head, without checking anything, might get it roughly right or might completely make it up based on what return policies usually look like at other stores. That's your plain LLM.

A good librarian instead says "let me check" and walks over to the right shelf, pulls out the specific policy document, flips to the relevant page, reads that paragraph, and then answers you using what's actually written there. That "walking over to the shelf and finding the right page" part is retrieval. The "reading it and giving you an answer in plain English" part is generation. Put those two together and you get Retrieval Augmented Generation.

Why can't the model just memorize everything

Good question, and it's the one that trips people up. A few reasons.

The model was trained at some point in the past on a giant pile of public text from the internet. Anything private, like your own notes, your company's internal docs, your codebase, or anything created after training happened, it simply never saw. There's no way for it to know that stuff exists.

Even if you tried to train the model on your own data specifically (this is called fine tuning), that process is slow, expensive, and annoying to redo every single time your documents change. If your company policy gets updated tomorrow, you don't want to retrain a whole model. You just want the system to pick up the new document.

RAG sidesteps all of that. Your documents live wherever they normally live. When someone asks a question, you search those documents on the spot, grab the relevant bit, and hand it to the model fresh every single time. Update a document, and the very next question automatically benefits from the update. No retraining needed.

Okay but how does the computer know which part of the document is "relevant"

This is the part everyone glosses over, so let's actually slow down here because this is the heart of the whole system.

Computers are bad at understanding meaning the way we do. If you search a document for the exact words "cancel my subscription" using basic search, and the document actually says "how to terminate your membership," a simple keyword search will completely miss it, even though to a human those two phrases obviously mean almost the same thing.

To fix this, we use something called an embedding model. Its whole job is to take a piece of text and turn it into a list of numbers, something like [0.12, -0.87, 0.44, ... ] with maybe a few hundred or a thousand numbers in the list. This list of numbers is called a vector.

Here's the important bit. The embedding model is trained in such a way that texts with similar meaning end up with similar lists of numbers, even if the actual words are totally different. So "cancel my subscription" and "terminate my membership" end up with vectors that are close to each other in this numerical space, while something totally unrelated like "recipe for banana bread" ends up far away.

Once everything, both your documents and the user's question, gets converted into these number lists, finding "relevant" text becomes a math problem. You're just measuring which document vectors sit closest to the question's vector. That's it. That's the secret. It's not understanding in the human sense, it's just very clever math on numbers that happen to represent meaning.

The full pipeline, step by step, in plain English before we touch code

Here's the order things actually happen in, and I'll walk through each piece with the librarian example still in your head.

Step one, you take your documents and chop them into smaller pieces. This is called chunking. You wouldn't hand someone an entire 300 page manual to find one paragraph, you'd hand them the specific page. Same idea here.

Step two, you run every chunk through the embedding model and get back a list of numbers for each one. You save all these number lists somewhere, this storage is usually called a vector store or vector database.

Step three, this all happens once, ahead of time, before anyone even asks a question. Think of it as pre organizing the library shelves.

Step four, when someone actually asks a question, you run their question through the same embedding model to get its number list.

Step five, you compare the question's numbers against every chunk's numbers and pull out the closest matches. This comparison usually uses something called cosine similarity, which is just a mathematical way of measuring how similar two lists of numbers are, kind of like measuring the angle between two arrows.

Step six, you take those closest matching chunks and paste them into a prompt along with the original question, then send that whole thing to the LLM.

Step seven, the LLM reads the chunks you gave it and answers the question using that information, instead of guessing from memory.

That's the entire system. Now let's actually build one.

Let's build a tiny working version, for real

I'm going to keep the tools boring on purpose so you can actually follow what's happening instead of getting lost in library documentation. We'll use plain Python, a small free embedding model that runs locally on your own computer (no API key needed for this part), and basic math with numpy.

First, install what we need.

pip install sentence-transformers numpy

The sentence-transformers library gives us a ready made embedding model we can use for free, and it downloads automatically the first time you run it.

Step 1, our tiny "documents"

In a real system these would come from actual files, a database, or a website. To keep things simple and easy to follow, I'm just going to write them directly as text in the code.

documents = [
    "Our return policy allows customers to return unopened electronics within 30 days for a full refund.",
    "Opened electronics can only be exchanged for the same item if there is a manufacturing defect.",
    "Software products and digital downloads are non refundable once purchased.",
    "Shipping costs are non refundable, only the product price is refunded.",
    "To start a return, log into your account and click on Order History, then select Request Return."
]

Step 2, chunking

In this example each sentence is already a nice small chunk, so we don't need to split them further. In a real project, you'd usually break long documents into pieces of a few hundred words each, often with a bit of overlap between chunks so you don't accidentally cut a sentence in half at a chunk boundary. For our tiny example, each line above is already one clean chunk.

Step 3, turning our documents into number lists (embeddings)

from sentence_transformers import SentenceTransformer

# this loads a small, free embedding model onto your computer
model = SentenceTransformer('all-MiniLM-L6-v2')

# this turns every document into a list of numbers
document_embeddings = model.encode(documents)

print(document_embeddings.shape)
# output looks something like: (5, 384)
# 5 documents, each turned into a list of 384 numbers

If you print document_embeddings[0], you'll literally see a long list of decimal numbers. That's the "meaning" of the first sentence, expressed as math instead of words.

Step 4, turning the user's question into a number list too

question = "Can I get money back for a game I bought online?"

question_embedding = model.encode([question])

Notice the question doesn't use the words "refund" or "software" directly, it says "money back" and "game I bought online." A basic keyword search would probably miss the right document entirely. Let's see if our embedding based approach does better.

Step 5, finding the closest matching documents using math

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# compare the question against every document
similarities = []
for i, doc_embedding in enumerate(document_embeddings):
    score = cosine_similarity(question_embedding[0], doc_embedding)
    similarities.append((score, documents[i]))

# sort so the best matches come first
similarities.sort(key=lambda x: x[0], reverse=True)

# grab the top 2 most relevant chunks
top_matches = similarities[:2]

for score, text in top_matches:
    print(round(score, 3), "-", text)

When I run this, the top result is the sentence about software products and digital downloads being non refundable, even though the question never uses the word "software" or "refundable." That's the embedding model catching the meaning of "game bought online" and "money back," not just the literal words. This is genuinely the moment RAG clicked for me the first time I saw it work.

Step 6, building the prompt with the retrieved chunks

Now we take those top matching chunks and hand them to the LLM along with clear instructions.

retrieved_text = "\n".join([text for score, text in top_matches])

prompt = f"""
You are a helpful customer support assistant.
Only answer using the information given in the context below.
If the answer isn't in the context, say you don't know, do not guess.

Context:
{retrieved_text}

Question: {question}
"""

print(prompt)

That instruction telling it not to guess matters a lot more than it looks like at first glance. Without it, the model tends to blend its own general knowledge in with your retrieved context, and you lose the whole point of grounding the answer in your actual documents.

Step 7, sending it to an actual LLM

This last part depends on which AI provider you're using, but the shape of it is always the same, you send the prompt, you get text back. Here's what it looks like using a typical API setup.

import anthropic

client = anthropic.Anthropic(api_key="your-api-key-here")

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=300,
    messages=[
        {"role": "user", "content": prompt}
    ]
)

print(response.content[0].text)

And that's it. That's a full, real, working RAG pipeline from raw text documents all the way to a grounded answer. Everything above this line is the entire concept. Everything real world RAG systems add on top is just making these same seven steps faster, cheaper, and more accurate at a bigger scale.

What changes when you go from this toy example to a real project

The core idea never changes, but a few things get more serious once you're dealing with thousands of documents instead of five sentences.

Instead of storing embeddings in a plain Python list, you'd use an actual vector database like Chroma, Pinecone, or Weaviate, which are built to search through millions of these number lists quickly.

Instead of writing your documents directly in the code, you'd load them from PDFs, websites, or a database, and you'd need a proper chunking strategy, usually a few hundred words per chunk with some overlap between chunks.

You'd also want to handle the case where nothing relevant gets found at all, and make sure the model actually says "I don't know" instead of making something up, which is why that instruction in the prompt matters so much.

And you'd probably want to periodically re run the embedding step whenever your source documents change, otherwise you end up answering questions using outdated information without any warning that something's wrong.

None of that changes the underlying idea though. It's still just search, then answer using what you found. Everything else is refinement.

Where you'll actually run into this in daily life

Once you understand this, you start noticing it everywhere. That chatbot on a company's website that somehow knows details specific to your account and their exact policies, probably running some version of this. Tools that let you upload a PDF and "chat" with it, same idea, your PDF gets chunked and embedded, then your questions get matched against those chunks. Even code editors that can answer questions about your specific codebase are usually doing something similar, searching your actual files for relevant code before answering, instead of relying purely on what the model happened to learn during training.

Once you see the pattern, these products stop feeling like unexplainable magic and start feeling like something you could genuinely build yourself over a weekend, because now you actually have.

A small challenge if you want to actually learn this properly

Reading code is fine but it won't stick the way building something does. Try this, take ten notes or documents about anything you actually care about, your recipes, your class notes, your journal entries, whatever, and run them through the code above. Ask questions and watch what gets retrieved. Try asking things using totally different words than what's in your notes and see if it still finds the right chunk. Then try breaking it on purpose, ask something that isn't covered in your notes at all and see if your prompt instruction actually makes it say "I don't know" instead of guessing.