RAG from Scratch: Building a Document Q&A System with LangChain, ChromaDB & Free AI Models

In this blog, we'll build a RAG pipeline from scratch using LangChain, and understand what happens at every step: from loading a document to generating the final answer.

I want you to keep this diagram in mind.

📄 DOCUMENT
     ↓
1️⃣ Document Loader
     ↓
📃 Documents
     ↓
2️⃣ Text Splitter ✂️
     ↓
🧩 Chunks
     ↓
🧠 Embedding Model
     ↓
🔢 Vectors
     ↓
3️⃣ ChromaDB 🗄️
     ↓
4️⃣ Retriever 🔎
     ↓
Relevant Chunks
     ↓
🤖 LLM
     ↓
💬 Final Answer

Now, we'll go through each component one by one and understand both the theory behind it and how it works practically.

1. Component 1 — Document Loader

Reads your files (PDF, DOCX, TXT, etc.) and extracts their content.

Imagine you have:

employee_handbook.pdf

Your Python application can't simply reason over the PDF itself.

The Document Loader reads the file and converts it into a format that LangChain can work with.

employee_handbook.pdf
        ↓
   PDF Loader
        ↓
LangChain Document objects

A LangChain Document generally contains:

Document(
    page_content="Employees are entitled to 20 days...",
    metadata={
        "source": "employee_handbook.pdf",
        "page": 10
    }
)

So you get two important things:

• page_content - The actual text.
• metadata - Information about the text.

For RAG, metadata is incredibly useful because later you can tell the user:

"This answer came from Employee Handbook, page 10."

Practical example:

For a PDF, you can use a PDF loader such as PyPDFLoader or PyMuPDF.

For this blog, I'll use PyMuPDFLoader because it generally offers better performance and text extraction for real-world RAG applications. If you're just getting started or working with simple PDFs, PyPDFLoader is also an excellent choice.

from langchain_community.document_loaders import PyMuPDFLoader

loader = PyMuPDFLoader("employee_handbook.pdf")

documents = loader.load()

Now:

PDF
 ↓
PyMuPDFLoader
 ↓
documents

You can inspect:

• print(documents[0].page_content)
• print(documents[0].metadata)

There are many others document loaders in the LangChain ecosystem. Some of them are:

• PDF → PyMuPDFLoader
• TXT → TextLoader
• DOCX → Docx2txtLoader
• CSV → CSVLoader
• Web page → WebBaseLoader

2. Component 2 — Text Splitter ✂️

Breaks large documents into smaller, meaningful chunks that are easier for the AI to process.

Now suppose your PDF has 100 pages.

You don't want to treat it as one giant block.

Instead:

100-page PDF
     ↓
     ✂️
     ↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
...
Chunk 500

This process is called chunking.

A chunk is simply a smaller piece of the original document.

Why do we need chunks?

You don't necessarily want to embed the entire 100-page document.

We split large documents into smaller chunks so the AI can process and understand information more effectively.

It also helps the retriever find only the most relevant pieces instead of searching through the entire document.

Which splitter should we use?

RecursiveCharacterTextSplitter is a great starting point.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)

chunks = text_splitter.split_documents(documents)

Conceptually:

Document
↓
Try paragraphs
↓
Too large?
↓
Try smaller boundaries
↓
Eventually split further

It tries to keep related text together rather than blindly chopping at arbitrary positions.

What is chunk overlap?

Imagine:

Chunk 1:
A B C D E F

Chunk 2:
E F G H I J

E F is the overlap.

Why? Because important information might sit around a boundary.

Without overlap, you could separate related information. A small overlap helps preserve continuity.

Few other text splitters :

• CharacterTextSplitter- Simple splitting based on a character separator.
• TokenTextSplitter- Splits based on tokens, useful when working with token limits.
• MarkdownHeaderTextSplitter- Useful for Markdown documents while preserving headings/structure.
• SemanticChunker- Splits text based on semantic meaning rather than just character count.

3. Component 3 — Embedding Model

Stores the chunks as numerical embeddings so they can be searched by meaning.

"Employees receive 20 days of annual leave."
                  ↓
          Embedding Model
                  ↓
[0.023, -0.421, 0.812, ...]

That list of numbers is an embedding vector.

What Embedding model should we use?

Since I want free local open models, I can use a Hugging Face/Sentence Transformers embedding model locally. Hugging Face models

Good candidates:

• BAAI/bge-small-en-v1.5
• intfloat/e5-base-v2
• all-MiniLM-L6-v2

To start with, I'd recommend: BAAI/bge-small-en-v1.5

Why?

• Open model
• Can run locally
• No per-request API fee
• Good for semantic retrieval
• Lightweight enough for experimentation
• Well suited to RAG/semantic search

You can use it through LangChain's Hugging Face integration.

What about Gemini?

Here's an important distinction.

Gemini is an LLM/model family from Google, but your RAG pipeline needs two different model capabilities:

Embedding model
        +
Generation model

• You need an embedding model to turn chunks/questions into vectors.

• You need a generative LLM to produce the final answer.

So your architecture could be:

PDF
 ↓
LangChain (RecursiveCharacterTextSplitter)
 ↓
Chunks
 ↓
Embedding Model (BGE ← embedding model)
 ↓
Vector DB (ChromaDB)
 ↓
Retriever
 ↓
LLM (Gemini ← generation model)
 ↓
Answer

4. Component 4 — Vector Database 🗄️

Stores the chunks as numerical embeddings so they can be searched by meaning.

Now we have:

Chunk
 ↓
Embedding model
 ↓
Vector

We need somewhere to store those vectors. That's where Vector Database comes in.

I'll go with ChromaDB because it's simple, local, free to experiment with, and integrates nicely with LangChain.

Think of ChromaDB as a specialized database for your AI application's semantic search.

Chunk 1 → Vector 1
Chunk 2 → Vector 2
Chunk 3 → Vector 3
Chunk 4 → Vector 4

Chroma stores the vectors along with associated information such as document content and metadata.

Why not just use a normal database?

Traditional databases are great for exact keyword or structured searches, but RAG needs to find information based on meaning.

Vector search helps find that semantic relationship.

For example:

Question: “How many vacation days do employees get?”
Document: “Employees receive 20 days of annual leave.”

A vector database can recognize that “vacation days” and “annual leave” have similar meanings by comparing their embeddings, even though the exact words are different.

Practical ChromaDB implementation

LangChain provides a Chroma integration.

Conceptually:

from langchain_chroma import Chroma

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model,
    persist_directory="./chroma_db"
)

Some other Vector database examples:

• FAISS — Free, open-source, excellent for local projects and learning.
• Qdrant — Open-source vector database with strong filtering and search capabilities.
• Weaviate — Open-source vector database with a free self-hosted option.
• Milvus— Open-source and designed for large-scale vector search.

5. Component 5 — Retriever 🔎

Finds and returns the most relevant chunks from the vector database based on the user's question.

The retriever is the component that says:

"Given this question, which chunks should I bring back?"

Suppose the user asks:

How many annual leaves do employees get?

The process becomes:

Question
   ↓
Embedding Model
   ↓
Question Vector
   ↓
ChromaDB
   ↓
Similarity Search
   ↓
Top relevant chunks

For example:

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 3}
)

Here, k = 3. Means return the 3 most relevant chunks.

Therefore,
🥇 Chunk 32 → 0.92 similarity
🥈 Chunk 31 → 0.84 similarity
🥉 Chunk 89 → 0.71 similarity

The top results are sent to the LLM.

These retrieved chunks become the context provided to the LLM to generate the answer.

But how is this "similarity" actually calculated?

One of the most common methods is cosine similarity.

Look at the direction of two vectors and see how similar they are.

If two vectors point in similar directions: Similarity → HIGH

If they point in very different directions: Similarity → LOW

Formula for cosine sinilarity is:

              A · B
cosine = ─────────────
         ||A|| × ||B||

• A = question vector
• B = chunk vector
• A · B = dot product
• ||A|| = magnitude of vector A
• ||B|| = magnitude of vector B

(This is just for knowledge purposes, you don't have to worry about the formula as Retriever is taking care of all the calculations bts.)

So conceptually:

Similar meaning
      ↓
Similar vector direction
      ↓
High similarity score

Look at these two sentences:

"How many vacation days do employees get?"
"Employees receive 20 days of annual leave."

They don't share many exact words. But their meaning is related.

The embedding model has learned semantic relationships, so their vectors can end up relatively close in the embedding space.

That's why vector search can find:

annual leave

when the user asks about:

vacation days


• ChromaDB doesn't generate the answer.
• ChromaDB doesn't understand the question like an LLM.
• It performs vector similarity search.

Question Vector
       ↓
[0.11, -0.43, 0.76, 0.24...]

       ↕
       ↕ similarity
       ↕

Chunk 1 Vector
[0.12, -0.45, 0.78, 0.21...]

Chunk 2 Vector
[0.15, -0.40, 0.72, 0.18...]

Chunk 3 Vector
[-0.72, 0.31, -0.15, 0.84...]

The system calculates how similar the vectors are.

6. Prompt — Giving the LLM the Right Context 📝

Combines the user's question with the retrieved context and tells the LLM how to answer.

For example, the user asks:

"How many annual leave days do employees get?"

The retriever might find:

Context:

Employees are entitled to 20 days of annual leave
per calendar year. Employees must submit their
leave requests through the HR portal.

Instead of simply sending the question How many annual leave days do employees get? to the LLM, we provide the LLM with both:

📚 Context
+
❓ User Question
+
📝 Instructions

And now the LLM has the information it needs.

The LLM combines the retrieved context with its own language understanding to generate a relevant, natural-language answer to the user's question.

For example:

User Question
"How many vacation days do employees get?"
          ↓

📚 Retrieved Context
"Employees receive 20 days of annual leave....."
          +
🧠 LLM's language understanding
          ↓

💬 Answer
"Employees receive 20 days of annual leave per year."

Prompt Template in LangChain

Instead of manually creating strings every time, LangChain lets us create a reusable Prompt Template.

For example:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_template("""
You are a helpful assistant.

Answer the question using only the provided context.
If the answer is not present in the context,
say that you don't have enough information.

Context:
{context}

Question:
{question}

Answer:
""")

Now every time a user asks a question, we can insert:

context
+
question

into the template.

Which LLM to use?

Gemini Flash

For this project, we'll use Gemini through its free API tier. If you specifically need an open-weight model that can run locally, Gemma is a suitable alternative.

Why Gemini?
✅ Free tier available
✅ Very capable for RAG question answering
✅ Easy to integrate with LangChain
✅ Works nicely in Google Colab
✅ No need to run a large LLM locally

(FYI: Gemini is not an open-source model. It is Google's proprietary model family that currently offers a free API tier for certain models/usage.)

Your code sends the request to Google's API:

Your Python code
      ↓
Gemini API
      ↓
Google's servers run Gemini
      ↓
Answer returned

Few other LLMs:

• Llama— Meta's open-weight models, widely used for local AI and RAG applications.
• Mistral— Efficient open-weight models that work well for RAG and text-generation tasks.
• Qwen— Alibaba's open-weight models, offering strong performance across many AI tasks.
• GPT-OSS — OpenAI's open-weight models designed to run on your own infrastructure.

Grounding

The prompt can also instruct the LLM to ground its answer in the retrieved context.

For example:

❌ Don't make up information.

✅ Answer using the provided context.

❓ If the context doesn't contain the answer,
   say that you don't know.

This is important because LLMs can sometimes generate plausible-sounding information that isn't actually present in your documents.

RAG helps reduce this problem by providing relevant external context.

RAG doesn't magically eliminate hallucinations, but good retrieval + good prompting can significantly improve grounded responses.

6. Generation — The LLM Generates the Answer 🤖

Generation is the final step where the LLM uses the retrieved context along with its language understanding to generate a relevant, natural-language answer.

Example:

Context: "Employees receive 20 days of annual leave."
Question: "How many vacation days do employees get?"
                    ↓
                  🤖 LLM
                    ↓
Answer: "Employees receive 20 days of annual leave per year."

In simple words: Retrieve the information → Give it to the LLM → LLM generates the answer.

Conclusion 🚀

And that's RAG from scratch! We've walked through the complete journey of a document, from loading and chunking the content to creating embeddings, storing them in ChromaDB, retrieving relevant context, and finally using an LLM to generate an answer.

The key takeaway is that RAG isn't just about using an LLM, it's about giving the LLM the right information at the right time. By combining retrieval with generation, we can build AI applications that can work with our own documents and knowledge sources.

I hope this breakdown made the RAG pipeline a little less intimidating and a lot more understandable. 😊
Now it's your turn, try it with your own documents and see what you can build!

💻 Try It Yourself

Want to experiment with the complete pipeline? I've put together a practical notebook that walks through the implementation step by step. You can run it, experiment with different documents, models, chunk sizes, and retrieval settings, and see how each component affects the final answer.

🔗 Open the Google Colab Notebook