RAG from Scratch with ChromaDB (No LangChain Required)

Frameworks like LangChain are great for moving fast, but they also hide a lot of what's actually happening under the hood. If you want to understand RAG at a deeper level — or just want a lighter-weight stack without extra abstraction layers — you can build a fully functional Retrieval-Augmented Generation pipeline using just ChromaDB and a handful of standard Python libraries.

This article walks through building RAG from first principles: chunking documents, generating embeddings, storing and querying vectors in ChromaDB, and passing retrieved context to an LLM — all without a single LangChain import.

LangChain is useful, but going framework-free has real advantages in certain situations:

• Transparency — You see exactly what's happening at each step: chunking, embedding, retrieval, prompt construction. Nothing is hidden behind an abstraction.
• Fewer dependencies — Your project stays lightweight, with fewer version-compatibility headaches.
• Full control — You can customize chunking logic, retrieval scoring, or prompt formatting exactly the way you want, without working around a framework's opinions.
• Easier debugging — When something goes wrong, you're debugging your own code, not tracing through several layers of framework abstraction.

What Is ChromaDB?

Chroma is an open-source, embedding-native vector database. It can run fully in-process (embedded, like SQLite) or as a standalone client-server service. It handles storing vectors, associated metadata, and performing similarity search — exactly the retrieval half of a RAG pipeline — without requiring any other orchestration library.

Building the Pipeline Step by Step

1. Install Dependencies

pip install chromadb openai

We'll use OpenAI for both embeddings and generation, but you can swap in any embedding model or LLM provider — the pipeline logic stays the same.

1. Set Up ChromaDB

Chroma can persist data to disk, so your index survives across script runs.

import chromadb

client = chromadb.PersistentClient(path="./chroma_db")

collection = client.get_or_create_collection(
    name="rag_demo",
    metadata={"hnsw:space": "cosine"}
)

A collection in Chroma is roughly equivalent to a table or index — it's where your document chunks and their embeddings live.

1. Load and Chunk Your Documents

Without LangChain's text splitters, you write your own simple chunking function. A straightforward sliding-window approach works well for most text:

import os

def load_documents(folder_path):
    docs = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            with open(os.path.join(folder_path, filename), "r", encoding="utf-8") as f:
                docs.append({"id": filename, "text": f.read()})
    return docs

def chunk_text(text, chunk_size=1000, overlap=200):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

documents = load_documents("./docs")

all_chunks = []
for doc in documents:
    for i, chunk in enumerate(chunk_text(doc["text"])):
        all_chunks.append({
            "id": f"{doc['id']}_chunk_{i}",
            "text": chunk,
            "source": doc["id"]
        })

This mirrors what LangChain's RecursiveCharacterTextSplitter does internally — split into fixed-size windows with overlap — just written explicitly so you can tune it however you like.

1. Generate Embeddings

Call the embedding model directly, batching requests where possible to keep things fast.

from openai import OpenAI

openai_client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

def get_embeddings(texts, model="text-embedding-3-small"):
    response = openai_client.embeddings.create(
        input=texts,
        model=model
    )
    return [item.embedding for item in response.data]

texts = [chunk["text"] for chunk in all_chunks]
embeddings = get_embeddings(texts)

1. Store Chunks in ChromaDB

collection.add(
    ids=[chunk["id"] for chunk in all_chunks],
    embeddings=embeddings,
    documents=[chunk["text"] for chunk in all_chunks],
    metadatas=[{"source": chunk["source"]} for chunk in all_chunks]
)

Chroma stores the raw text, the vector, and any metadata together, so retrieval results come back ready to use — no separate lookup needed.

1. Retrieve Relevant Chunks for a Query

def retrieve(query, top_k=4):
    query_embedding = get_embeddings([query])[0]
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=top_k
    )
    return results["documents"][0], results["metadatas"][0]

query = "What does our refund policy cover?"
retrieved_docs, retrieved_metadata = retrieve(query)

1. Construct the Prompt and Call the LLM

This is the "augment and generate" step — you build the prompt manually, giving you full control over formatting.

def build_prompt(query, context_chunks):
    context = "\n\n".join(context_chunks)
    return f"""Answer the question based only on the following context.
If the answer isn't in the context, say you don't know.

Context:
{context}

Question: {query}
"""

def generate_answer(query, context_chunks, model="gpt-4o-mini"):
    prompt = build_prompt(query, context_chunks)
    response = openai_client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    return response.choices[0].message.content

answer = generate_answer(query, retrieved_docs)
print(answer)

** Putting It All Together**

def rag_query(question, top_k=4):
    docs, metadata = retrieve(question, top_k=top_k)
    return generate_answer(question, docs)

print(rag_query("What does our refund policy cover?"))

That's the entire pipeline: load, chunk, embed, store, retrieve, prompt, generate — with no orchestration framework in between.

Design Considerations for Production RAG

Once you move past a prototype, a few decisions matter more:

• Chunking strategy — A fixed-size sliding window is simple but not always ideal. Splitting on sentence or paragraph boundaries often produces more coherent chunks than a raw character count.
• Metadata filtering — Chroma's query method accepts a where filter, letting you narrow search by metadata (e.g., only search chunks from a specific source or date range) before similarity search runs.
• Batch embedding calls — Embedding APIs are usually billed and rate-limited per request; batch your texts list instead of calling the embedding endpoint once per chunk.
• Persistence and backups — PersistentClient writes to local disk by default; for production, consider running Chroma as a client-server deployment so multiple processes can share the same index safely.
• Re-ranking — After the initial vector search, you can re-score the top results with a cross-encoder model before passing only the best few chunks to the LLM, improving answer quality.
• Evaluation — Since there's no framework enforcing structure, it's worth writing your own lightweight evaluation harness (e.g., checking retrieval precision or answer faithfulness) as your document set grows.

You don't need a heavyweight framework to build a working RAG system. ChromaDB alone — paired with a plain embedding call and a plain LLM call — gives you a transparent, fully controllable pipeline: chunk your documents, embed them, store and search them in Chroma, then feed the retrieved context into your model of choice. This approach trades a bit of convenience for a much clearer picture of exactly what your RAG system is doing at every step, which pays off the moment you need to debug, customize, or optimize it.