DEV Community

Cover image for How I Built a 100% Local PDF RAG Assistant on CPU Using Python and Ollama
Sujith Ravikumar
Sujith Ravikumar

Posted on AI-assisted

How I Built a 100% Local PDF RAG Assistant on CPU Using Python and Ollama

The moment this project felt real wasn't when the model generated its first paragraph. It was when I asked it a question that wasn't covered in its documents and instead of making something up, it checked its similarity score, paused, and politely refused to guess.

As a cloud engineer who spends most days building backend systems, designing cloud architectures, and working on production deployment strategies, AI always felt like a separate world dominated by massive GPU clusters and expensive cloud APIs. Recently, I decided to take my first hands-on step into AI engineering.

This is the story of my first end-to-end AI application—a small accomplishment, built from scratch on a standard Linux server, that taught me more than hours of youtube tutorials.


The Spark: A Real Problem Worth Solving

Every engineering team deals with the same quiet productivity drain: dense, multi-page technical PDFs. Finding a single configuration parameter, protocol rule, or specification table often means opening multiple manuals and hitting Ctrl + F until something vaguely matches.

I didn't want to build a flashy general-purpose chatbot that writes poetry or answers trivia. I wanted a focused, dependable technical assistant that does one job well:

  • Answer questions strictly from local reference manuals.
  • Cite the exact source file and page number so engineers can verify the answer immediately.
  • Say "I don't know" whenever the answer isn't actually in the documents.

Why "100% Local" on CPU Mattered

From day one, I set two strict architectural constraints:

  1. Zero external cloud APIs: Technical documents and queries stay completely inside the server.
  2. CPU-only execution: No dedicated GPUs—just standard Linux compute.

Coming from a cloud and infrastructure background, running everything locally eliminates per-token API costs, removes external network latency, and keeps data privacy straightforward. More importantly, it proves you don't need expensive GPU hardware to build a fast, practical Retrieval-Augmented Generation (RAG) system.


The Tech Stack & Why Each Piece Was Chosen

Instead of reaching for heavy abstraction frameworks that hide how data moves, I chose lightweight, transparent tools where every stage could be inspected and debugged:

Layer Tool / Model Chosen Why It Was Considered & Chosen
PDF Parsing & OCR PyMuPDF + pdfplumber + RapidOCR Plain text extractors scramble tables and ignore diagrams. Combining native text extraction, Markdown table conversion, and lightweight CPU ONNX OCR captures the full page accurately.
Embeddings FastEmbed (BAAI/bge-small-en-v1.5) Runs via ONNX Runtime on CPU with zero heavy deep-learning bloat, turning 800-character chunks into compact 384-dimensional vectors in milliseconds.
Vector Store ChromaDB (Local Persistent) Running a distributed vector cluster for a single-server app would be over-engineering. ChromaDB persists metadata in SQLite and the HNSW similarity graph right on disk.
Local LLM Ollama (qwen2.5:3b) Fits comfortably in CPU RAM while following strict grounding prompts. Locked at temperature = 0.0 so responses stay factual and deterministic rather than creative.
API & Serving FastAPI + Uvicorn + systemd Familiar backend tools to expose clean REST endpoints, keep models warm in RAM across requests, and run cleanly behind a reverse proxy.

Three Real Engineering Hurdles I Solved

1. PDFs Are Much Harder Than Plain Text

In my first prototype, I dumped raw text from a PDF and sliced it every 800 characters. The answers were rough because specification tables turned into scrambled words and sentences got cut in half mid-thought.

What fixed it:

  • Added a 120-character overlap between chunks so boundary sentences stay intact.
  • Converted grid tables into structured Markdown pipe tables (| Column | Value |) before chunking.
  • Preserved exact page_number metadata on every chunk so citations never drift.

2. Teaching the AI When to Say "I Don't Know"

A confident hallucination destroys trust in a technical tool immediately. Even with a strict system prompt, compact models will still try to be "helpful" and guess if a question shares keywords with an indexed manual.

What fixed it: I implemented a two-layer safety net before returning any response:

  1. Layer 1 — Cosine Similarity Gate (>= 0.70): Before calling the LLM, the retriever checks the top vector match score. If the closest chunk scores below 0.70, the pipeline stops immediately—saving CPU cycles and returning a clean fallback message.

Grounded Answer + Citations

  1. Layer 2 — Strict Context-Only Prompting: Chunks that pass Layer 1 are wrapped inside explicit <context> boundaries with instructions to emit a fixed abstention signal if the exact answer isn't present:

Irrelevant question - anti-hallucination gate

Answer the <question> using ONLY the facts inside <context>.
- Do not use outside knowledge or guess missing values.
- If the context does not contain enough information, reply ONLY with: INSUFFICIENT_EVIDENCE
Enter fullscreen mode Exit fullscreen mode

3. Turning a CLI Script into a Real Web Service

Getting a Python script to answer questions in a terminal is only half the job. Making it usable as a real application meant applying standard production backend habits:

  • Warm Memory Residency: Loading the embedding model and vector index once at FastAPI startup (lifespan) so user queries don't pay cold-load penalties.
  • Live /health Diagnostics: Building a lightweight telemetry dashboard showing service health, total indexed chunks, active models, and real-time retrieval vs. generation latency.

Live System Diagnostics Dashboard


Honest Reflection & What’s Next

Right now, this is an honest v1 pilot. It handles single-turn, cited technical questions reliably and serves concurrent queries in roughly 1.5 to 4 seconds on CPU.

At the same time, shipping v1 made the next set of improvements clear:

  • Hybrid Search: Combining semantic vector search with exact keyword matching so specific error codes and part numbers rank even higher.
  • Multi-Turn Memory: Supporting natural follow-up questions within a conversation session.
  • Token Streaming: Streaming words to the UI as the CPU generates them so responses feel instantaneous.

Coming from a cloud and backend background, building my first AI application showed me that practical AI isn't magic—it's good data pipelines, sensible guardrails, and solid systems engineering.


Why I Wrote This & Let's Discuss!

I wrote this post because when I first started exploring AI, most guides assumed you had cloud API keys or a dedicated GPU. If you're a backend or cloud engineer curious about building your first AI project, starting small and local is one of the best ways to understand every moving piece under the hood.

Have you built a local RAG pipeline or deployed small LLMs on CPU?
I'd love to hear what worked best for you!

Top comments (0)