Micro-RAG

Large language models (LLMs) have a problem: hallucination — from time to time, they state falsehoods with confidence.The reason is simple. Models know only what they read during training, hence nothing about today's weather, or your company's financial report this year.

RAG, short for Retrieval-Augmented Generation, is a technology designed to address exactly this problem.

The idea of RAG is simple: feed the model facts before letting it answer a question.

A typical pipeline has five steps:

1. Split the documents into chunks
2. Turn each chunk into a vector
3. Organize the chunks and vectors as an index
4. Before answering a question, retrieve the top-K most relevant chunks
5. Hand them to the LLM to generate the answer

That may sound like a lot. So we'll just walk you through the whole process with a short piece of Python code.

Note that our code cuts plenty of corners and is very much a toy model. This way, you can see the overall flow without getting bogged down in details.

1. Preparing the Corpus

First, we need a knowledge base. Every question that follows is answered against it.

For simplicity, we'll organize it as a list of strings — one string per document.

docs = [
    "Python is a popular programming language for data processing, machine learning, and AI in general.",
    "Data processing involves cleaning, transforming, and aggregating raw data into a useful format.",
    "Machine learning trains models to find patterns in data and make predictions on new examples.",
    "Deep learning uses neural networks with many layers to learn hierarchical representations.",
    "Neural networks are computing systems inspired by the brain, made of interconnected layers of nodes that learn to recognize patterns and make predictions from data.",
    "Transformers use self-attention to model relationships between tokens in a sequence.",
    "Attention lets a model weigh different parts of the input when producing each output representation.",
    "GPT stands for Generative Pre-trained Transformer, a decoder-only Transformer model trained to predict the next token.",
    "Large language models (LLM) are neural networks trained on massive text corpora to predict the next token.",
    "Pretraining learns general representations from large unlabeled data before task-specific fine-tuning.",
    "Fine-tuning adapts a pretrained model to a specific task or domain using labeled or instruction data.",
    "Post-training adapts a pretrained model with instruction tuning, preference optimization, or safety alignment.",
    "Retrieval-Augmented Generation (RAG) combines retrieval from an external knowledge source with generation from an LLM.",
    "A basic RAG pipeline chunks documents, embeds the chunks, stores them in a vector index, retrieves top-k chunks, and passes them to the LLM.",
    # rest omitted
]

Don't worry about what these sentences say. They were generated by an LLM anyway.

So docs is about the most bare-bones "knowledge base" you can imagine. In real life it might be a several-hundred-page product manual — but the idea is the same.

One more thing: our documents are already short, so we skip the first step of an ordinary RAG pipeline — chunking.

2. Tokenization — Splitting Sentences into Words

Computers don't know sentences or articles. They only know numbers. So the first step is to split a sentence into tokens. Later we'll treat these tokens as the dimensions of a high-dimensional space, and a sentence or an article becomes a vector in that space.

For simplicity, let's treat each token as an English word and write the simplest tokenizer we can:

import re
def tokenize(doc):
    return re.findall(r"[a-z]+", doc.lower())

Code notes

• doc.lower() turns the text into lowercase first.
• The regular expression [a-z]+ means "a run of consecutive lowercase letters" — whatever matches is treated as one token.
• To keep things simple, digits and punctuation are both dropped.
• re.findall() finds every piece that matches the pattern and returns a list.
• And that's the splitting, in effect.

With a different corpus, though, dropping punctuation could change the meaning. Tokenization strategy is itself a trade-off.

A small example

Run it on one sample, say tokenize(docs[1]):

"Data processing involves cleaning, transforming, ..."

and you get

['data', 'processing', 'involves', 'cleaning', 'transforming', ...]

3. The Vocabulary — Giving Every Word an ID

def build_vocab(docs):
    blacklist = ['is', 'are', 'was', 'were', 'a', 'an', 'the',
        'and', 'or', 'of', 'to', 'in', 'on', 'for', 'from', 'with', ]
    vocab = set([w for doc in docs for w in tokenize(doc) if w not in blacklist])
    return sorted(vocab)

vocab = build_vocab(docs)
word_to_id = {w: i for i, w in enumerate(vocab)}

Code notes

This code does two things:

1. Build the vocabulary: walk every document's words, collect them, and toss out the low-information ones (the blacklist); then sort what's left into a word list.
2. Build the ID table: word_to_id is a dict that maps each word to its position in the vocabulary (0, 1, 2, …).

• The blacklist holds articles, the verb "to be", and a few other words of that kind. They turn up constantly yet carry almost no meaning. Keep them, and two completely unrelated sentences will be judged "similar" just for sharing a couple of throwaway words like of and to.

• Once they're filtered out, every remaining word in this corpus has an ID of its own.

4. Word Vectors — Representing a Passage as a Set of Numbers

Now the main event: how do we turn a piece of text into a vector?

The scheme here is the classic one, and the simplest — bag of words.

• Bag of words treats each word as one dimension of a high-dimensional space.
• The vocabulary length is the number of dimensions.
• So each word is, in effect, a unit vector along one axis of that space.

How do we use that to represent a passage?

• Whichever words appear in the passage, we add up their unit vectors, and the sum represents the whole passage.
• Under this scheme, if a word appears N times in the passage, the sum has a magnitude of N in that dimension — not 1.

from collections import Counter

def embed(doc):
    ''' compute the word vector '''
    emb_vec = [0] * len(vocab)          # start with a long all-zero vector
    for w, count in Counter(tokenize(doc)).items():
        if w in word_to_id:
            emb_vec[word_to_id[w]] = count   # write the count into that word's slot
    norm = sum(x**2 for x in emb_vec)**0.5 or 1.0
    return [x / norm for x in emb_vec]       # normalize

Code notes

• First, Counter. It counts things: Counter([3, 1, 4, 1, 5]) hands back something dict-like — {3: 1, 1: 2, 4: 1, 5: 1} — telling you how many times each element appeared.

• So the logic of embed() is plain: the value in dimension i is how many times the i-th word appears in this passage. The more often a word shows up, the more weight it carries — and the more it contributes to the similarity score.

• The or 1.0 at the end of the second-to-last line is a safety net. If none of a text's words are in the vocabulary, the vector is all zeros, its norm is 0, and the division blows up. In Python, 0.0 or 1.0 is 1.0 — one little "or" dodges the divide-by-zero.

• Why bother with the safety net? Words from the knowledge base are in the vocabulary, of course — no need there. But we also run this function on the user's question, and the user's words are not guaranteed to be in the vocabulary.

• One more thing: every component of this vector is non-negative, and it has been normalized (last line), so the dot product of two vectors is exactly the cosine similarity (see below).

• Normalization also keeps the comparison fair between sentences of different lengths: it stops a longer sentence from gaining an edge just by having more words — and hence a larger dot product.

5. Similarity — How Alike Are Two Passages?

With vectors in hand, we can measure how "far apart" two passages are:

def dot(a, b):
    ''' cosine similarity '''
    return sum(x * y for x, y in zip(a, b))

Code notes

The dot product of two vectors (given both are normalized) is the cosine similarity. The intuition is simple: the more words two passages share, the larger the dot product.

• Maximum 1: the two texts point in exactly the same direction — in bag-of-words terms, identical word-count distributions.
• Minimum 0: the two texts share no vocabulary at all. Word counts can't be negative, so the dot product never drops below 0.

An example

Start with the extremes. In this corpus, the most similar pair is #10 and #11 (counting from zero — the Fine-tuning and Post-training entries). They share the content words tuning, pretrained, model, and instruction, for a similarity of 0.44. The least similar pairs share no words at all and sit squarely at 0.

But look only at the most similar pair, and it's easy to believe the system "understood" something. In fact, the whole approach has a problem.

For instance, #5 is about Transformer*s* (plural). Ask about the singular transformer, and it comes up empty just the same.

More word overlap, higher score; words fail to line up, the score collapses — even when both passages are about the same thing.

So this version of vectors and similarity doesn't "understand" meaning. It's just counting word overlap.

6. Building the Index — Precomputing Every Document

index = [(doc, embed(doc)) for doc in docs]

The index is a table of "each document ↔ its vector". With it, comparing two articles costs a single dot product — no re-tokenizing, no re-counting.

Think of it as a library's card catalog: every book's summary is written up in advance, so at lookup time you just flip through the cards.

7. Retrieval — Finding the k Most Relevant Documents

def retrieve(query, k=3, threshold=0.0):
    q_vec = embed(query)
    doc_scores = [(doc, dot(q_vec, vec)) for doc, vec in index]
    doc_scores.sort(key=lambda x: x[1], reverse=True)
    return [(doc, score) for doc, score in doc_scores[:k] if score > threshold]

Code notes

It goes like this:

1. Turn the query into a vector.
2. Score it against every document in the store.
3. Sort by score, highest first.
4. Take the top k, dropping anything below the threshold.

Note that retrieve() does not guarantee k documents. The threshold gate filters out irrelevant ones; if no document scores above it, you get an empty list. That's deliberate — a missing answer beats a forced one.

8. Generation — Feeding What We Found to the LLM

def dumb_llm(prompt):
    m = re.search(r'CONTEXT:\s+(.*?)\s+QUESTION:', prompt, re.DOTALL)
    return m.group(1) if m and m.group(1) else "Sorry, I have no idea."

def ask(query, hits):
    context = '\n'.join(f"- {doc}" for doc, score in hits)
    prompt = f"""Answer according to the following context:

CONTEXT:
{context}

QUESTION:{query}
"""
    return dumb_llm(prompt)

Code notes

• ask() does the one thing every RAG system does: stuff the retrieved results into the prompt as "context" and hand it to the model.

• dumb_llm() is a stand-in toy. It generates nothing at all; it just uses a regular expression to copy the CONTEXT block back out verbatim. In a real system, this is where GPT or DeepSeek sits — it reads the context and puts an answer together in its own words.

An example

Try it:

ask('What is a neural network?', [(docs[4], 0.9), (docs[3], 0.5)])

It will dutifully splice those two entries into the context, and dumb_llm() spits them straight back out.

This toy actually points at something essential: retrieval sets the quality ceiling of RAG. Hand the LLM the wrong material, and even a strong model can only answer wrong — or refuse with that Sorry, I have no idea.

9. The Complete RAG

def rag(query):
    hits = retrieve(query, k=5)
    ans = ask(query, hits)
    return ans, hits

Driver code:

import sys

if __name__ == '__main__':
    if len(sys.argv) > 1:
        query = ' '.join(sys.argv[1:])
        ans, hits = rag(query)
        print(f'Q: {query}\nA: {ans}')
        print('\n=============')
        for doc, score in hits:
            print(f'[{score:.2f}] {doc}')

Four steps fold into one: retrieve → assemble → generate → return. Ask it What is RAG? and it returns the two RAG entries from the corpus, scored 0.26 and 0.20 — not high, since the question is short and shares few words with the documents, but enough to push the right material to the top.

Now ask about something the corpus has nothing to say — say, Why do cats chase rats? There is no cat and no rat in the vocabulary, so the query vector is nearly all zeros; no document scores above the threshold, retrieval returns an empty list, and what you finally see is:

Sorry, I have no idea.

Empty retrieval, an honest refusal — this is exactly where RAG beats letting the model wing it.

10. From Toy to the Real World

The RAG here is a teaching version. A production system upgrades every stage, but the skeleton stays the same:

Stage	Teaching version	Real version
Tokenization	regex word splitting	subword tokenization (BPE / SentencePiece)
Vectorization	bag-of-words counts	neural embeddings (OpenAI Embeddings, BERT, and so on)
Similarity	dot product	cosine / inner product + approximate nearest-neighbor index (FAISS, HNSW)
Retrieval	full scan of the store	vector databases (Pinecone, Milvus, pgvector)
Generation	a toy regex	an actual LLM, plus citations and reranking

The design choices planted along the way — stop-word filtering, normalization, threshold filtering, refusal — all have counterparts in industry. A few dozen lines, and the backbone of RAG is clear.

Closing

The idea behind RAG is simple: don't expect a model to know everything. Teach it to look things up.

1. Retrieval finds the right information.

2. Generation puts it into good prose.

Next time you ask an AI assistant about something that happened after its training cutoff, and it gives you an accurate answer, there's a good chance a pipeline like this is running behind it — from a list of strings to a program that knows how to look things up.