DEV Community

Cover image for I Built a Local RAG Assistant with Ollama, ChromaDB and LangChain. Here's What I Learned
TAGBA G-Josaphat E.
TAGBA G-Josaphat E.

Posted on

I Built a Local RAG Assistant with Ollama, ChromaDB and LangChain. Here's What I Learned

#rag #langchain #python #ollama

During my internship at a software services company, I noticed a recurring problem: technicians spent hours searching through hundreds of pages of PDF manuals to answer client questions. The company deploys a suite of management tools accounting, HR, logistics mainly to public institutions managing projects funded by international donors.

I kept thinking: this is exactly the kind of problem LLMs were made for. So for my Master's project in AI & Big Data, I built an intelligent assistant to handle it.

The constraint that shaped every technical decision: no data could leave the local infrastructure. These institutions handle sensitive financial data it cannot be sent to OpenAI, Anthropic, or any cloud provider.

This forced me into the world of fully local LLMs. Here's what I built, what broke, and what I learned.

The Problem

The company had accumulated massive technical documentation PDF manuals, configuration guides, FAQ files but no efficient way to query it. When a technician faced an issue with the accounting module or the HR module, they searched manually. Slow, inconsistent, and dependent on individual expertise.

Classic LLMs can't help here: they don't know the private internal documentation. And without grounding in official docs, they hallucinate procedures dangerous when dealing with accounting operations for donor-funded projects.

The solution: RAG (Retrieval-Augmented Generation) make the LLM answer only from the official documentation, injected at query time.

What RAG Actually Is

Before the code, the concept in one sentence: instead of asking the LLM to "know" everything, you retrieve the relevant passages from your documents and inject them into the prompt as context.

The pipeline has two distinct phases:

Phase 1 : Ingestion (done once)

PDF documents
    ↓
Chunking (split into 300-character blocks)
    ↓
Embedding (convert each block to a vector)
    ↓
ChromaDB (store all vectors)
Enter fullscreen mode Exit fullscreen mode

Phase 2 : Query (at each question)

User question
    ↓
Embed the question (same model)
    ↓
ChromaDB similarity search → top 3 chunks
    ↓
Inject chunks + question into LLM prompt
    ↓
Llama 3 generates a grounded answer
Enter fullscreen mode Exit fullscreen mode

The key insight: the LLM never "knows" your documents. It reads them fresh at each query, from the context you provide. No fine-tuning needed. No retraining when docs are updated.

The Architecture: 4 Docker Services

Everything runs locally via Docker Compose. Four services, each with a clear role:

Service Role Port
Ollama Runs Llama 3 locally 11434
ChromaDB Vector database 8001
FastAPI RAG pipeline + REST API 8000
Streamlit User interface 8501 
# docker-compose.yml (simplified)
services:
  ollama:
    image: ollama/ollama:latest
    ports: ["11434:11434"]
    volumes: ["./ollama_models:/root/.ollama"]
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  chromadb:
    image: chromadb/chroma:latest
    ports: ["8001:8000"]
    volumes: ["./chroma_db:/chroma/chroma"]

  python-api:
    build: ./src
    ports: ["8000:8000"]
    depends_on: [ollama, chromadb]
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - CHROMA_HOST=chromadb
      - CHROMA_PORT=8000
Enter fullscreen mode Exit fullscreen mode

Step 1 : Ingesting the Documentation

I ingested 10 documents accounting references (SYSCOHADA, SYCEBNL), a general accounting code, and internal user manuals. That's 2,111 pages split into 9,669 chunks.

# src/ingestion.py

import os
import chromadb
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

DATA_DIR = "/app/data/documents"
COLLECTION_NAME = "documentation"

def load_documents():
    documents = []
    for fichier in os.listdir(DATA_DIR):
        if fichier.endswith(".pdf"):
            try:
                loader = PyPDFLoader(os.path.join(DATA_DIR, fichier))
                docs = loader.load()
                # Tag each chunk with its source document
                for doc in docs:
                    doc.metadata["module"] = fichier.replace(".pdf", "")
                documents.extend(docs)
            except Exception as e:
                print(f"Skipped {fichier}: {e}")  # Don't crash on encrypted PDFs
    return documents

def store_embeddings(chunks):
    embeddings = HuggingFaceEmbeddings(
        model_name="all-MiniLM-L6-v2",
        model_kwargs={"device": "cpu"}
    )
    # Explicit HTTP connection — not localhost, the Docker service name
    client = chromadb.HttpClient(host="chromadb", port=8000)
    Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        collection_name=COLLECTION_NAME,
        client=client
    )
    print(f"Done — {len(chunks)} vectors stored")
Enter fullscreen mode Exit fullscreen mode

Two things worth noting:

  • I tag each chunk with its source document name as metadata this allows the LLM to cite its source in the answer
  • The try/except is not optional one encrypted PDF (it happened) would have crashed the entire ingestion without it. Install cryptography if you hit an AES error: pip install cryptography

Step 2 : Building the RAG Chain 

# src/rag_chain.py

import chromadb
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.llms import Ollama
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

def create_rag_chain():
    # Must be the same model used during ingestion
    embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

    client = chromadb.HttpClient(host="chromadb", port=8000)
    vectorstore = Chroma(
        client=client,
        collection_name="documentation",
        embedding_function=embeddings
    )

    # Retrieve top 3 most relevant chunks
    retriever = vectorstore.as_retriever(
        search_type="similarity",
        search_kwargs={"k": 3}
    )

    llm = Ollama(
        base_url="http://ollama:11434",
        model="llama3",
        temperature=0.3,
        num_predict=250   # Cap response length — critical on CPU
    )

    template = """You are a technical support assistant.
Answer ONLY from the provided context. Cite your source.
If the answer is not in the context, say so clearly.

Context: {context}
Question: {question}
Answer:"""

    prompt = ChatPromptTemplate.from_template(template)

    def format_docs(docs):
        return "\n\n".join([
            f"[Source: {d.metadata.get('module', '?')}]\n{d.page_content}"
            for d in docs
        ])

    # The full pipeline in 4 steps
    return (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )
Enter fullscreen mode Exit fullscreen mode

The chain is a pipeline: retriever → prompt → llm → parser. LangChain's | operator makes this clean and readable. Each step passes its output to the next.

Step 3 : API and Interface

FastAPI exposes the chain as a REST endpoint:

# src/main.py

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from rag_chain import create_rag_chain

app = FastAPI(title="Local RAG Assistant")
chain = create_rag_chain()

class QueryRequest(BaseModel):
    question: str

@app.post("/query")
def query(request: QueryRequest):
    try:
        answer = chain.invoke(request.question)
        return {"question": request.question, "answer": answer}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
Enter fullscreen mode Exit fullscreen mode

Streamlit makes it usable in a browser in 10 lines:

# src/app.py

import streamlit as st
import requests

st.title("Technical Support Assistant")
question = st.text_input("Your question:")

if st.button("Search") and question:
    with st.spinner("Searching documentation... (1-2 min on CPU)"):
        r = requests.post(
            "http://python-api:8000/query",
            json={"question": question},
            timeout=600
        )
        st.write(r.json()["answer"])
Enter fullscreen mode Exit fullscreen mode

What Broke (and How I Fixed It)

This is the part nobody writes about. Three real problems I hit.

Problem 1 : ChromaDB v2 silently ignoring my data

I was connecting with the old client_settings syntax:

# WRONG silently stores data locally instead of sending to the server
Chroma.from_documents(
    documents=chunks,
    client_settings=Chroma.Settings(chroma_server_host="chromadb")
)
Enter fullscreen mode Exit fullscreen mode

The ingestion ran without errors, showed 9669 chunks stored but ChromaDB had 0 vectors. Querying returned nothing.

The fix explicit HttpClient:

# CORRECT explicit HTTP connection to the ChromaDB Docker service
client = chromadb.HttpClient(host="chromadb", port=8000)
Chroma.from_documents(documents=chunks, client=client, ...)
Enter fullscreen mode Exit fullscreen mode

Also: the v1 heartbeat endpoint (/api/v1/heartbeat) is deprecated. Use /api/v2/heartbeat to check if ChromaDB is alive.

Problem 2 : LangChain moved its modules 

# Broke silently after a LangChain update
from langchain.prompts import ChatPromptTemplate           # ❌ ModuleNotFoundError
from langchain.schema.runnable import RunnablePassthrough  # ❌ ModuleNotFoundError

# Everything stable lives in langchain_core now
from langchain_core.prompts import ChatPromptTemplate      # ✅
from langchain_core.runnables import RunnablePassthrough   # ✅
Enter fullscreen mode Exit fullscreen mode

LangChain has been refactoring aggressively across versions. If you hit ModuleNotFoundError, check langchain_core first it's the stable base layer they're committing to not break.

Problem 3 : Llama 3 timing out on CPU

Llama 3 (8B parameters) on CPU generates roughly 2-5 tokens per second. A 200-word answer takes 2+ minutes. My 120-second timeout was too short.

Three things helped significantly:

  • Dropping k from 5 to 3 chunks shorter prompt = faster time-to-first-token
  • Adding num_predict=250 to cap response length without it, the model generates forever
  • Allocating more RAM to Docker via .wslconfig on Windows to avoid swap

Results

After setup, the assistant correctly answers questions grounded in the official documentation:

  • "How to close monthly payroll?" → Step-by-step procedure from the HR module manual
  • "How to create an account in the chart of accounts?" → Exact navigation path, source cited
  • "What does SYSCOHADA class 4 cover?" → Accurate definition from the accounting reference

And critically when the answer isn't in the documentation, it says so explicitly instead of hallucinating. That was the whole point.

Key Takeaways

On RAG:

The embedding model used during ingestion must be identical to the one used at query time. A different model produces incompatible vector spaces your similarity search returns garbage without any error message.

Chunk size matters more than expected. 500 characters worked less well than 300 for technical documents shorter chunks produce more precise retrieval.

Metadata on chunks is essential for source citation. Tag every chunk with its document name at ingestion time.

On local LLMs:

Running on CPU is slow but viable for non-real-time use cases like internal support tools.

num_predict is your most important parameter for performance. Without it, the model generates until it decides to stop sometimes never.

Docker container networking does not use localhost. Use the service name defined in docker-compose.yml (e.g., chromadb, ollama) as the hostname when connecting between containers.

This project is available on my github : https://github.com/josaphatstar/Assistant-Intelligent-RAG

What's Next

This RAG pipeline handles straightforward questions well. But it has a core limitation: it always does the same thing regardless of the question type. Whether you ask a procedural question or report a system crash, it searches the docs and answers. No reasoning about what strategy fits best.

In the next article, I'll show how I evolved this into an Agentic AI architecture with LangGraph where the system first decides which agent to call (documentation search, error diagnosis, or human escalation ticket), then acts accordingly.

I'm a Master's student in AI & Big Data, and this project came out of a real internship constraint no cloud, no API keys, just local infrastructure and open-source tools. I'm curious: have you built a local RAG pipeline under similar constraints? What stack did you use, and what broke first? Drop it in the comments I'd genuinely like to compare notes.

Top comments (0)