RAG: How to Give an LLM Access to Your Own Documents

In the previous post, we gave the LLM tools so it could call functions and take actions. But there's another limitation we haven't dealt with yet — the LLM has no idea about your private data. It's never seen your documents, your codebase, your company wiki, or anything that wasn't in its training data.

The obvious fix is to dump everything into the context. But that breaks down quickly — context windows have limits, and cramming too much in actually degrades the quality of responses. The model starts losing track of things buried in the middle.

That's where RAG comes in. Instead of sending everything, you send only what's relevant.

How RAG works

The idea is straightforward. You store your documents in a searchable index. When a question comes in, you find the most relevant pieces and inject just those into the prompt. The model answers using that focused context instead of trying to remember everything.

Without RAG:  [entire knowledge base] + question → LLM
With RAG:     [3 relevant chunks]     + question → LLM

The tricky part is "finding the most relevant pieces." You can't just do a keyword search — what if the document says "canine" but the question says "dog"? That's where embeddings come in.

What embeddings are

An embedding is a list of numbers that represents the meaning of a piece of text. Similar meanings produce similar lists of numbers, so you can measure how close two pieces of text are just by comparing their vectors.

"The dog chased the cat"      → [0.21, -0.54, 0.87, ...]
"A puppy ran after a kitten"  → [0.23, -0.51, 0.85, ...]   ← very close
"Stock market crashed today"  → [-0.67, 0.12, -0.34, ...]  ← far away

The first two sentences are semantically similar even though they share no words. The third is unrelated, so its vector points in a completely different direction. This is what makes it possible to search by meaning rather than by keyword.

The full pipeline

RAG has two phases. The first you do once — index your documents. The second happens every time a question comes in:

Indexing (done once):

Documents → split into chunks → embed each chunk → store in vector database

Query (per question):

Question → embed it → find closest chunks → inject into prompt → generate answer

One thing that trips people up: you must use the same embedding model for both phases. If you index with one model and query with another, the vectors won't be comparable.

Chunking

When you index documents, you split them into smaller pieces first. The chunk is what gets retrieved, so the size matters. Too large and you inject a lot of irrelevant text. Too small and you lose the surrounding context that makes the answer make sense.

A good starting point is chunks of around 400 characters with a 50-character overlap between consecutive chunks. The overlap is important — if a useful sentence happens to sit right at the boundary between two chunks, the overlap makes sure it shows up in at least one of them.

Setup

pip install ollama chromadb

You'll also need an embedding model — this is different from a chat model. It only produces vectors, not text:

ollama pull nomic-embed-text

Quick check:

import ollama

r = ollama.embeddings(model="nomic-embed-text", prompt="Hello world")
print(f"Vector dimensions: {len(r.embedding)}")   # should print 768

Exercise 1 — Understanding embeddings

Before building RAG, let's see what embeddings actually look like in practice:

import ollama
import math

def embed(text: str) -> list[float]:
    r = ollama.embeddings(model="nomic-embed-text", prompt=text)
    return r.embedding

def cosine_similarity(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x ** 2 for x in a))
    mag_b = math.sqrt(sum(x ** 2 for x in b))
    return dot / (mag_a * mag_b)

sentences = [
    "The dog chased the cat",
    "A puppy ran after a kitten",
    "Stock markets fell sharply today",
    "Python is a programming language",
    "Cats and dogs are common household pets",
]

embeddings = {s: embed(s) for s in sentences}

print("Similarity scores:\n")
for i, s1 in enumerate(sentences):
    for s2 in sentences[i+1:]:
        score = cosine_similarity(embeddings[s1], embeddings[s2])
        print(f"{score:.3f}  |  '{s1[:35]}' vs '{s2[:35]}'")

The scores between "dog/cat" and "puppy/kitten" will be high even though they share no words. That's the whole point — embeddings capture meaning, not spelling.

Exercise 2 — Building a vector store

Now let's index some documents and search them:

import ollama
import chromadb

client = chromadb.Client()
collection = client.create_collection("my_docs")

documents = [
    "Python is a high-level programming language known for its simplicity.",
    "Kotlin is a modern language that runs on the JVM, developed by JetBrains.",
    "PostgreSQL is a powerful open-source relational database system.",
    "Kafka is a distributed event streaming platform for high-throughput messaging.",
    "Docker is a platform for containerising applications and their dependencies.",
    "Spring Boot is a framework for building Java and Kotlin web applications.",
    "ChromaDB is an open-source vector database for storing and querying embeddings.",
    "RAG stands for Retrieval-Augmented Generation — gives LLMs access to external data.",
]

print("Indexing documents...")
for i, doc in enumerate(documents):
    embedding = ollama.embeddings(model="nomic-embed-text", prompt=doc).embedding
    collection.add(ids=[str(i)], embeddings=[embedding], documents=[doc])

def search(query: str, top_k: int = 3) -> list[str]:
    query_embedding = ollama.embeddings(model="nomic-embed-text", prompt=query).embedding
    results = collection.query(query_embeddings=[query_embedding], n_results=top_k)
    return results["documents"][0]

queries = [
    "What database should I use?",
    "How do I build a web service?",
    "Tell me about messaging systems",
    "What is a container?",
]

for q in queries:
    print(f"\nQuery: {q}")
    for doc in search(q):
        print(f"  → {doc}")

Try "How do I build a web service?" — it retrieves the Spring Boot entry despite sharing no keywords. That's semantic search in action.

Exercise 3 — Full RAG pipeline

Let's connect retrieval to an LLM and build a proper Q&A system:

import ollama
import chromadb

client = chromadb.Client()
collection = client.create_collection("knowledge_base")

documents = [
    "Python was created by Guido van Rossum and first released in 1991.",
    "Kotlin was developed by JetBrains and officially released in 2016.",
    "PostgreSQL supports ACID transactions, foreign keys, joins, and views.",
    "Kafka was originally developed at LinkedIn and open-sourced in 2011.",
    "Docker uses Linux namespaces and cgroups to isolate containers.",
    "Spring Boot auto-configures your application based on dependencies you add.",
    "ChromaDB stores embeddings alongside metadata and original document text.",
    "RAG improves LLM accuracy by injecting relevant retrieved context into the prompt.",
    "Flyway is a database migration tool that applies versioned SQL scripts in order.",
    "Gradle is a build automation tool using Groovy or Kotlin DSL for build scripts.",
]

for i, doc in enumerate(documents):
    emb = ollama.embeddings(model="nomic-embed-text", prompt=doc).embedding
    collection.add(ids=[str(i)], embeddings=[emb], documents=[doc])

def rag_answer(question: str) -> str:
    # embed the question
    question_emb = ollama.embeddings(model="nomic-embed-text", prompt=question).embedding

    # find the most relevant chunks
    results = collection.query(query_embeddings=[question_emb], n_results=3)
    retrieved = results["documents"][0]

    print("Retrieved:")
    for chunk in retrieved:
        print(f"  → {chunk}")
    print()

    # inject into prompt
    context = "\n".join(f"- {chunk}" for chunk in retrieved)
    prompt = f"""Answer the question using ONLY the context below.
If the answer is not in the context, say "I don't know based on the provided context."

Context:
{context}

Question: {question}
Answer:"""

    response = ollama.chat(
        model="llama3.2",
        options={"temperature": 0},
        messages=[{"role": "user", "content": prompt}]
    )
    return response.message.content


questions = [
    "Who created Python?",
    "What is Flyway used for?",
    "When was Kafka open-sourced?",
    "What is the capital of France?",   # not in the knowledge base
]

for q in questions:
    print(f"Q: {q}")
    print(f"A: {rag_answer(q)}")
    print("-" * 60)

Pay close attention to the last question — "What is the capital of France?" is not in the knowledge base, so the model should say "I don't know based on the provided context." This is called grounding. The model is constrained to only use what was retrieved. That's how RAG prevents hallucination.

Exercise 4 — RAG over a real file

Now let's index an actual file from your project and query it:

import ollama
import chromadb

def chunk_text(text: str, chunk_size: int = 400, overlap: int = 50) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

with open("README.md", "r") as f:
    content = f.read()

chunks = chunk_text(content)
print(f"Split into {len(chunks)} chunks\n")

client = chromadb.Client()
collection = client.create_collection("readme")

for i, chunk in enumerate(chunks):
    emb = ollama.embeddings(model="nomic-embed-text", prompt=chunk).embedding
    collection.add(ids=[str(i)], embeddings=[emb], documents=[chunk])

def ask(question: str) -> str:
    q_emb = ollama.embeddings(model="nomic-embed-text", prompt=question).embedding
    results = collection.query(query_embeddings=[q_emb], n_results=3)
    retrieved = results["documents"][0]

    context = "\n---\n".join(retrieved)
    prompt = f"""Answer based ONLY on the context below.
If the answer is not present, say "Not found in the document."

Context:
{context}

Question: {question}
Answer:"""

    r = ollama.chat(
        model="llama3.2",
        options={"temperature": 0},
        messages=[{"role": "user", "content": prompt}]
    )
    return r.message.content

print(ask("How do I run the tests?"))
print(ask("What database does this project use?"))
print(ask("How do I set up Kafka locally?"))

Try changing chunk_size to 200 and then to 800 and compare the answers. Smaller chunks give more precise retrieval but can lose context. Larger chunks capture more context but can bring in noise. There's no perfect setting — it depends on your documents.

Wrapping up

RAG is how you give an LLM access to knowledge it was never trained on. The key pieces are embeddings (which let you search by meaning), a vector store (which holds all the embedded chunks), and a retrieval step that finds what's relevant before the LLM ever sees the question.

The grounding constraint — telling the model to only answer from the retrieved context — is what keeps it honest. Without it, the model will fill in gaps from its training data, which is exactly what you don't want when you're trying to answer questions about your own documents.

In Post #4, we put tools and retrieval together and build our first real agent — one that decides on its own what to do and keeps going until the task is done. See you there. 🚀