Build a RAG Chatbot with LangChain and Chroma

Retrieval-Augmented Generation, or RAG, is a pattern that lets a language model answer questions from your own documents instead of only from its training data. We turn our documents into searchable vectors, then hand the most relevant passages to the model with each question. In this tutorial we build one with LangChain and Chroma, using free models that run locally.

Key takeaways

• RAG grounds a language model in our own documents, so answers come from facts we control instead of from training memory.
• Chroma stores each chunk of text next to its embedding vector, and a similarity search finds the chunks that match a question.
• All-MiniLM-L6-v2 turns sentences into 384 number vectors, and it runs on a normal laptop CPU with no API key, according to its Hugging Face model card.
• A small local model can answer well and still misread a question, so we print the retrieved sources with every answer and check them.
• The whole chatbot runs offline once the models are downloaded, and it costs nothing to run.

What we build

We build a support chatbot for a fictional outdoor shop called Trailhead Outdoors. It answers questions about returns, shipping and discounts from three short documents we write ourselves. The chatbot has two scripts. The first script, ingest.py, loads the documents, splits them into chunks, turns each chunk into an embedding vector and stores everything in Chroma. The second script, chat.py, takes a question, finds the three most relevant chunks, and asks a small local language model to answer using only those chunks. We also print the source file of each chunk, so we can check every answer against the documents it came from.

Prerequisites

We need Python 3.12 and about 500 MB of free disk space for the two models. We also need a terminal and a text editor. We do not need an OpenAI account, an API key or a GPU. Everything in this tutorial runs on a CPU. The versions we used are LangChain 1.4.10, ChromaDB 1.5.9, sentence-transformers 6.2.0, transformers 5.22.2 and torch 2.15.0. Newer versions should work the same way, but the transformers 5 release removed one pipeline we would normally reach for, and we show the replacement in Step 4.

Step 1: Set up the project and install the stack

We start with a fresh folder and a virtual environment, then install the stack in one command.

mkdir rag-chatbot && cd rag-chatbot
python3 -m venv .venv
source .venv/bin/activate
pip install langchain langchain-community langchain-chroma langchain-text-splitters chromadb sentence-transformers transformers

The install takes a few minutes. LangChain is the orchestration layer that connects our loaders, splitters, embeddings and vector store. The langchain-chroma package holds the Chroma integration, and langchain-community holds the Hugging Face embeddings wrapper, as the LangChain Chroma docs explain. ChromaDB is the vector database itself. Sentence-transformers and transformers provide the two local models.

Step 2: Write three short documents for the knowledge base

We create a docs folder with three Markdown files about the fictional shop. Short files keep the demo easy to follow, and the same code works on a folder of real text files later. Here is the returns file in full, and the other two follow the same pattern.

# Returns Policy

Trailhead Outdoors offers a 30-day return window for unused items in their original packaging. Customers can start a return from the Orders page in their account, print a prepaid label, and drop the parcel at any courier point. Refunds go back to the original payment method within 5 business days of the warehouse receiving the item. Sale items marked final sale cannot be returned. Items bought with a gift card are refunded as store credit.

The shipping file states that standard delivery takes 3 to 5 business days and costs 4.99 pounds, express delivery takes 1 to 2 business days and costs 9.99 pounds, and orders over 75 pounds ship free with standard delivery. The pricing file states that students get 10 percent off with a valid student ID, first orders get 15 percent off with the code WELCOME15, and only one code can be used per order.

Step 3: Load, chunk and embed the documents

Our ingest.py script does four jobs: load the files, split them into chunks, embed the chunks, and store them in Chroma.

"""ingest.py: load docs, split into chunks, embed, store in Chroma."""
import shutil
from pathlib import Path

from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

BASE = Path(__file__).parent
CHROMA_DIR = BASE / "chroma_db"

if CHROMA_DIR.exists():
    shutil.rmtree(CHROMA_DIR)

loader = DirectoryLoader(str(BASE / "docs"), glob="**/*.md", loader_cls=TextLoader)
documents = loader.load()
print(f"Loaded {len(documents)} documents")

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks")

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    collection_name="trailhead_docs",
    persist_directory=str(CHROMA_DIR),
)
print(f"Stored {vectorstore._collection.count()} vectors in {CHROMA_DIR}")
print(f"Embedding dimension: {len(embeddings.embed_query('test'))}")

We run it with python ingest.py. The first run downloads the embedding model, which is about 80 MB. Our run printed this output.

Loaded 3 documents
Split into 4 chunks
Stored 4 vectors in /home/hatch/workspace/rag-chatbot/chroma_db
Embedding dimension: 384

Three files became four chunks because one file was longer than 500 characters. The 50 character overlap keeps sentences that cross a chunk boundary readable in both chunks. Chroma saves the collection to the chroma_db folder, so the next script can load it without re-embedding anything.

How does retrieval pick the right chunk?

Each chunk is turned into a list of 384 numbers, called an embedding vector, by the all-MiniLM-L6-v2 model. The model card on Hugging Face states that the model maps sentences and short paragraphs to a 384 dimensional dense vector space, and that it has about 22.7 million parameters. Sentences with similar meaning get vectors that point in a similar direction. When we ask a question, the question is embedded the same way, and Chroma measures the distance between the question vector and every chunk vector. The closest chunks win. This is why the search finds meaning instead of matching keywords. A question about the cost of express delivery finds the shipping paragraph even though it never uses the word price.

Step 4: Build the chat loop with a small local model

Our chat.py script loads the stored collection, turns it into a retriever, and answers each question with flan-t5-small. This model has 80 million parameters, according to its Hugging Face model card, and it runs on a CPU in a few seconds per question.

"""chat.py: retrieve relevant chunks from Chroma, answer with flan-t5-small."""
import sys
from pathlib import Path

from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_chroma import Chroma
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch

BASE = Path(__file__).parent

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vectorstore = Chroma(
    collection_name="trailhead_docs",
    embedding_function=embeddings,
    persist_directory=str(BASE / "chroma_db"),
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")
gen_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")

def generate(prompt: str) -> str:
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True)
    with torch.no_grad():
        out = gen_model.generate(**inputs, max_new_tokens=120)
    return tokenizer.decode(out[0], skip_special_tokens=True).strip()

One detail caught us out. Our first version used the text2text-generation pipeline from transformers, but transformers 5.22.2 has removed that pipeline, and the error message points to the version 5 migration guide. Loading the tokenizer and the sequence to sequence model directly, as above, is the replacement that works. The ask function then builds a prompt from the retrieved chunks and the question.

PROMPT = (
    "Answer the question using only the context below. "
    "If the answer is not in the context, say you do not know.\n\n"
    "Context:\n{context}\n\nQuestion: {question}\nAnswer:"
)

def ask(question: str) -> str:
    docs = retriever.invoke(question)
    context = "\n\n".join(d.page_content for d in docs)
    sources = sorted({Path(d.metadata["source"]).name for d in docs})
    prompt = PROMPT.format(context=context, question=question)
    answer = generate(prompt)
    return answer, sources

The prompt tells the model to answer only from the context. That single instruction is the cheapest guard against invented answers in a small RAG app. The retriever returns three chunks, and we print their file names next to each answer so we can trace every claim back to a document.

Step 5: Ask it questions and check the answers

We ran four questions through the chatbot. Here is the real output.

Q: How long do I have to return an item?
A: 30 days
Sources: returns-policy.md, shipping.md

Q: How much does express delivery cost?
A: 9.99 pounds
Sources: returns-policy.md, shipping.md

Q: What discount do students get?
A: 10 percent off
Sources: pricing.md, returns-policy.md, shipping.md

Q: How many days does standard delivery take?
A: 3 to 5 business days
Sources: returns-policy.md, shipping.md

All four answers match the documents. Each answer also lists more source files than it strictly needs. That is expected here: we have only four chunks and we ask for the top three, so nearly every query pulls in most of the collection. On a real knowledge base with hundreds of chunks, the same setting returns a much smaller share of the total.

Why did the chatbot get one answer wrong?

Before those four, we asked a fifth question in a different form: "Do students get a discount?" The model answered "No", even though the pricing document says students get 10 percent off. We asked the same thing as "What discount do students get?" and it answered correctly. The retrieval was fine in both cases. The pricing file was in the context both times. The small model simply misread a yes or no question. This is the honest limit of an 80 million parameter model: it can find the facts and still fumble the phrasing. Two habits protect us. First, ask "what" questions instead of yes or no questions when we test a small model. Second, always print the sources, and read the retrieved chunk before trusting the answer.

What we learned from the small model

Local RAG works end to end with no paid service. The pipeline is simple: chunk, embed, store, retrieve, generate. The quality of the answer depends on two separate things, and they fail separately. Retrieval fails when chunks are too big or the question uses words the documents never use. Generation fails when the model is too small to follow the instruction, even with the right context in front of it. When an answer looks wrong, we check the retrieved chunks first. If the chunks are right and the answer is wrong, the generator is the problem, and a bigger model or a clearer prompt is the fix.

What to try next

First, swap flan-t5-small for flan-t5-base and run the same five questions to see which answers change. Second, add ten more documents and lower the retriever to k equals 2, then check whether the answers stay correct with less context. Third, wrap chat.py in a small Streamlit app so the chatbot has a text box and a send button instead of a terminal. Each change is one small edit, and each one teaches us which part of the pipeline matters most.

---

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.