Episode 47 – Local RAG with Ollama: Build a Private AI Retrieval System
Hey, it’s Nick from Build Log. In the last 15 minutes of the episode I walked you through getting a Retrieval‑Augmented Generation (RAG) pipeline running entirely on a laptop – no cloud keys, no API fees, and zero data leakage. Below is the companion blog post that expands every “quick tip” into a full, actionable guide you can copy‑and‑paste straight into your terminal.
Why Go Local? The Real‑World Drivers
- Data sovereignty. Contracts, HIPAA, GDPR – the fine print that says “your data never leaves the premises.”
- Cost predictability. A $0‑API‑bill means you can budget hardware once and forget it.
- Latency. On‑device inference is sub‑second even on a modest MacBook Air, which beats round‑trip network calls by a wide margin.
- Control. You decide which model version you run, how it’s quantized, and whether you expose it to the internet.
Those reasons line up nicely with the scenario I described on the show: 13 client sites, proprietary PDFs, and a three‑year‑old laptop that still has a fan that sounds like a jet engine. If it works there, it works for you.
Hardware Reality Check – What You Really Need
Don’t be fooled by the hype that “any GPU will do.” For a production‑grade RAG stack on a laptop you need:
- CPU: 8 GB+ RAM, at least a quad‑core Intel i5 or Apple M1/M2. The model I use (Mistral‑7B‑instruct‑v0.1) fits comfortably in 6 GB of VRAM when quantized to 4‑bit.
- Disk: 20 GB free for the model weights and the ollama runtime. An SSD is a must; HDDs make the embedding step painfully slow.
- OS: macOS 12+, Ubuntu 20.04+, or Windows 10 (WSL2) – anything that can run Docker or a native binary.
If you’re on a machine that doesn’t meet these specs, consider a cheap external GPU (e.g., an NVIDIA RTX 3060 in an eGPU enclosure) or a small cloud‑burst VM for the heavy‑lifting stages.
Step‑by‑Step: The 15‑Minute Stack
Below is the exact command sequence I ran on a 2020 MacBook Air (8 GB RAM, Apple Silicon). Adjust paths as needed.
1️⃣ Install Ollama (the runtime that serves LLMs locally)
curl -L https://ollama.com/install.sh | sh
2️⃣ Pull a 7‑B instruction‑tuned model (Mistral‑7B‑instruct)
ollama pull mistral:7b-instruct-q4_0
3️⃣ Verify the model boots
ollama run mistral:7b-instruct-q4_0 "What is RAG?"
The first two commands take about 8–10 minutes on a 10 Gbps connection. The third should return a concise answer in under 2 seconds – that’s your sanity check.
3️⃣ Set Up the Vector Store
I’m using ChromaDB because it’s pure Python, works offline, and plays nicely with ollama. Install it in a virtual environment:
Create venv
python3 -m venv rag-env
source rag-env/bin/activate
Install dependencies
pip install chromadb sentence‑transformers tqdm
Now we need an embedding model. The all‑mpnet‑base‑v2 encoder from Sentence‑Transformers gives a sweet spot between speed and quality.
python -
4️⃣ Ingest Your Documents
For the demo I used a folder of PDFs and Markdown files. The script below walks the folder, extracts plain text (via pdfminer.six for PDFs), chunks it into 500‑token pieces, embeds each chunk, and stores it in Chroma.
pip install pdfminer.six tiktoken
python -
That script runs in ~3 minutes for a 200‑page PDF on my laptop. The key takeaway: you only need one‑off embedding; the rest of the pipeline stays offline.
5️⃣ Wire Up the Retrieval + Generation Loop
Now that we have a vector store, the final piece is a thin Flask API that takes a user query, pulls the top‑k relevant chunks, and feeds them to the local LLM.
pip install flask
python -
Start the service with python app.py and hit it with:
curl -X POST http://localhost:8080/rag -H "Content-Type: application/json" -d '{"question":"What is the retention policy for client data?"}'
You should see a JSON payload containing the AI’s answer and the exact chunks it used – perfect for auditability.
Tuning Your Ollama Model for Laptop‑Scale Performance
Out‑of‑the‑box the mistral:7b-instruct-q4_0 model runs at ~1.2 tokens/second on an 8‑GB MacBook Air. That’s fine for short answers but not for long form. Here are three knobs you can turn without re‑training:
- Quantization level. Ollama ships with 4‑bit (q4_0) and 8‑bit (q8_0) variants. If you have a bit more RAM, switch to q8_0 for a ~30 % speed boost.
- Context window. Pass --max-context 2048 when you start the server to cap the prompt length and avoid OOM errors.
- Temperature & top‑p. Lowering temperature to 0.2 and top‑p to 0.9 reduces hallucination on retrieval‑heavy tasks.
Example launch command with all three tweaks:
ollama serve mistral:7b-instruct-q8_0 --max-context 2048 --temperature 0.2 --top-p 0.9
Testing & Troubleshooting – A Mini‑Checklist
- Embedding mismatch. If you get “no results” from Chroma, double‑check that the tokenizer used for chunking matches the one used when you built the index. The tiktoken encoder in the script must be the same version you use at query time.
- Ollama hangs. On macOS, the first inference after a fresh launch can stall due to sandbox permissions. Run chmod +x $(which ollama) and restart the binary.
- GPU fallback. If you happen to have an external GPU, set the environment variable OLLAMA_GPU=1 before launching the server; Ollama will automatically offload matrix math.
- Memory spikes. Use ps -o %mem,rss,command -p $(pgrep -f ollama) to monitor RAM. If you see >80 % usage, switch to the 4‑bit model or reduce max-context.
Putting It Into Production – What I Do at My Agency
Once the Flask API is stable, I containerize it with Docker so the whole stack can be version‑controlled and rolled out to any laptop in seconds.
Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 8080
CMD ["python", "app.py"]
Build and run:
docker build -t local‑rag .
docker run -d -p 8080:8080 --restart unless-stopped local‑rag
In production I add a tiny reverse‑proxy (Caddy or Nginx) with HTTP basic auth to keep the endpoint private. I also schedule a nightly cron job that re‑runs the ingestion script on any new PDFs that land in the /knowledge_base folder – zero manual steps.
Key Takeaways
- Running a RAG pipeline locally is no longer a research‑only activity; a modern 8‑GB laptop can host a 7‑B instruction‑tuned model with a vector store.
- All the heavy lifting (model download, embedding, and vector indexing) is a one‑time cost; day‑to‑day queries are sub‑second.
- Ollama abstracts away the GPU/CPU specifics, letting you focus on prompt engineering and data hygiene.
- Auditability comes for free – the API returns the exact retrieved chunks alongside the answer.
- Containerizing the Flask service makes scaling across a team of 10‑plus laptops trivial.
Subscribe to Build Log
If you found this guide useful, hit the Subscribe button on the Build Log podcast page and drop me a line at nick@buildlog.fm. I’m always looking for real‑world edge cases to explore in upcoming episodes – from on‑device fine‑tuning to multi‑modal retrieval. Let’s keep building, keep hacking, and keep the data where it belongs: in your hands.
Adapted from an episode of Signal Notes. Listen on your favorite podcast app.
Top comments (0)