<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sujith Ravikumar</title>
    <description>The latest articles on DEV Community by Sujith Ravikumar (@sujith_ravikumar_8b51f6bb).</description>
    <link>https://dev.to/sujith_ravikumar_8b51f6bb</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4033940%2Fc0c47997-3d9a-4c63-885e-03588f7f87b9.jpg</url>
      <title>DEV Community: Sujith Ravikumar</title>
      <link>https://dev.to/sujith_ravikumar_8b51f6bb</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/sujith_ravikumar_8b51f6bb"/>
    <language>en</language>
    <item>
      <title>How I Built a 100% Local PDF RAG Assistant on CPU Using Python and Ollama</title>
      <dc:creator>Sujith Ravikumar</dc:creator>
      <pubDate>Thu, 19 Nov 2026 19:18:22 +0000</pubDate>
      <link>https://dev.to/sujith_ravikumar_8b51f6bb/how-i-built-a-100-local-pdf-rag-assistant-on-cpu-using-python-and-ollama-keg</link>
      <guid>https://dev.to/sujith_ravikumar_8b51f6bb/how-i-built-a-100-local-pdf-rag-assistant-on-cpu-using-python-and-ollama-keg</guid>
      <description>&lt;p&gt;The moment this project felt real wasn't when the model generated its first paragraph. It was when I asked it a question that wasn't covered in its documents and instead of making something up, it checked its similarity score, paused, and politely refused to guess.&lt;/p&gt;

&lt;p&gt;As a cloud engineer who spends most days building backend systems, designing cloud architectures, and working on production deployment strategies, AI always felt like a separate world dominated by massive GPU clusters and expensive cloud APIs. Recently, I decided to take my first hands-on step into AI engineering.&lt;/p&gt;

&lt;p&gt;This is the story of my first end-to-end AI application—a small accomplishment, built from scratch on a standard Linux server, that taught me more than hours of youtube tutorials.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Spark: A Real Problem Worth Solving
&lt;/h2&gt;

&lt;p&gt;Every engineering team deals with the same quiet productivity drain: dense, multi-page technical PDFs. Finding a single configuration parameter, protocol rule, or specification table often means opening multiple manuals and hitting &lt;code&gt;Ctrl + F&lt;/code&gt; until something vaguely matches.&lt;/p&gt;

&lt;p&gt;I didn't want to build a flashy general-purpose chatbot that writes poetry or answers trivia. I wanted a &lt;strong&gt;focused, dependable technical assistant&lt;/strong&gt; that does one job well:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Answer questions strictly from local reference manuals.&lt;/li&gt;
&lt;li&gt;Cite the exact source file and page number so engineers can verify the answer immediately.&lt;/li&gt;
&lt;li&gt;Say &lt;em&gt;"I don't know"&lt;/em&gt; whenever the answer isn't actually in the documents.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Why "100% Local" on CPU Mattered
&lt;/h2&gt;

&lt;p&gt;From day one, I set two strict architectural constraints:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Zero external cloud APIs:&lt;/strong&gt; Technical documents and queries stay completely inside the server.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CPU-only execution:&lt;/strong&gt; No dedicated GPUs—just standard Linux compute.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Coming from a cloud and infrastructure background, running everything locally eliminates per-token API costs, removes external network latency, and keeps data privacy straightforward. More importantly, it proves you don't need expensive GPU hardware to build a fast, practical Retrieval-Augmented Generation (RAG) system.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Tech Stack &amp;amp; Why Each Piece Was Chosen
&lt;/h2&gt;

&lt;p&gt;Instead of reaching for heavy abstraction frameworks that hide how data moves, I chose lightweight, transparent tools where every stage could be inspected and debugged:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Tool / Model Chosen&lt;/th&gt;
&lt;th&gt;Why It Was Considered &amp;amp; Chosen&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;PDF Parsing &amp;amp; OCR&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;PyMuPDF&lt;/code&gt; + &lt;code&gt;pdfplumber&lt;/code&gt; + &lt;code&gt;RapidOCR&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Plain text extractors scramble tables and ignore diagrams. Combining native text extraction, Markdown table conversion, and lightweight CPU ONNX OCR captures the full page accurately.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Embeddings&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;FastEmbed&lt;/code&gt; (&lt;code&gt;BAAI/bge-small-en-v1.5&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Runs via ONNX Runtime on CPU with zero heavy deep-learning bloat, turning 800-character chunks into compact &lt;strong&gt;384-dimensional vectors&lt;/strong&gt; in milliseconds.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Vector Store&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ChromaDB&lt;/code&gt; (Local Persistent)&lt;/td&gt;
&lt;td&gt;Running a distributed vector cluster for a single-server app would be over-engineering. ChromaDB persists metadata in SQLite and the HNSW similarity graph right on disk.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Local LLM&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Ollama&lt;/code&gt; (&lt;code&gt;qwen2.5:3b&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Fits comfortably in CPU RAM while following strict grounding prompts. Locked at &lt;strong&gt;&lt;code&gt;temperature = 0.0&lt;/code&gt;&lt;/strong&gt; so responses stay factual and deterministic rather than creative.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;API &amp;amp; Serving&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;FastAPI&lt;/code&gt; + &lt;code&gt;Uvicorn&lt;/code&gt; + &lt;code&gt;systemd&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Familiar backend tools to expose clean REST endpoints, keep models warm in RAM across requests, and run cleanly behind a reverse proxy.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  Three Real Engineering Hurdles I Solved
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. PDFs Are Much Harder Than Plain Text
&lt;/h3&gt;

&lt;p&gt;In my first prototype, I dumped raw text from a PDF and sliced it every 800 characters. The answers were rough because specification tables turned into scrambled words and sentences got cut in half mid-thought.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What fixed it:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Added a &lt;strong&gt;120-character overlap&lt;/strong&gt; between chunks so boundary sentences stay intact.&lt;/li&gt;
&lt;li&gt;Converted grid tables into structured Markdown pipe tables (&lt;code&gt;| Column | Value |&lt;/code&gt;) before chunking.&lt;/li&gt;
&lt;li&gt;Preserved exact &lt;code&gt;page_number&lt;/code&gt; metadata on every chunk so citations never drift.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  2. Teaching the AI When to Say "I Don't Know"
&lt;/h3&gt;

&lt;p&gt;A confident hallucination destroys trust in a technical tool immediately. Even with a strict system prompt, compact models will still try to be "helpful" and guess if a question shares keywords with an indexed manual.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What fixed it:&lt;/strong&gt; I implemented a &lt;strong&gt;two-layer safety net&lt;/strong&gt; before returning any response:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Layer 1 — Cosine Similarity Gate (&lt;code&gt;&amp;gt;= 0.70&lt;/code&gt;):&lt;/strong&gt; Before calling the LLM, the retriever checks the top vector match score. If the closest chunk scores below &lt;code&gt;0.70&lt;/code&gt;, the pipeline stops immediately—saving CPU cycles and returning a clean fallback message.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhfgcn7fmwzrgpr2qv9pv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhfgcn7fmwzrgpr2qv9pv.png" alt="Grounded Answer + Citations" width="800" height="267"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Layer 2 — Strict Context-Only Prompting:&lt;/strong&gt; Chunks that pass Layer 1 are wrapped inside explicit &lt;code&gt;&amp;lt;context&amp;gt;&lt;/code&gt; boundaries with instructions to emit a fixed abstention signal if the exact answer isn't present:&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyy17diojivzy7zcu9i5e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyy17diojivzy7zcu9i5e.png" alt="Irrelevant question - anti-hallucination gate" width="800" height="175"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Answer the &amp;lt;question&amp;gt; using ONLY the facts inside &amp;lt;context&amp;gt;.
- Do not use outside knowledge or guess missing values.
- If the context does not contain enough information, reply ONLY with: INSUFFICIENT_EVIDENCE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  3. Turning a CLI Script into a Real Web Service
&lt;/h3&gt;

&lt;p&gt;Getting a Python script to answer questions in a terminal is only half the job. Making it usable as a real application meant applying standard production backend habits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Warm Memory Residency:&lt;/strong&gt; Loading the embedding model and vector index once at FastAPI startup (&lt;code&gt;lifespan&lt;/code&gt;) so user queries don't pay cold-load penalties.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Live &lt;code&gt;/health&lt;/code&gt; Diagnostics:&lt;/strong&gt; Building a lightweight telemetry dashboard showing service health, total indexed chunks, active models, and real-time retrieval vs. generation latency.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhgvtjj1celo6ujlfq4kr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhgvtjj1celo6ujlfq4kr.png" alt="Live System Diagnostics Dashboard" width="800" height="192"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Honest Reflection &amp;amp; What’s Next
&lt;/h2&gt;

&lt;p&gt;Right now, this is an honest &lt;strong&gt;v1 pilot&lt;/strong&gt;. It handles single-turn, cited technical questions reliably and serves concurrent queries in roughly &lt;strong&gt;1.5 to 4 seconds on CPU&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;At the same time, shipping v1 made the next set of improvements clear:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Hybrid Search:&lt;/strong&gt; Combining semantic vector search with exact keyword matching so specific error codes and part numbers rank even higher.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Multi-Turn Memory:&lt;/strong&gt; Supporting natural follow-up questions within a conversation session.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Token Streaming:&lt;/strong&gt; Streaming words to the UI as the CPU generates them so responses feel instantaneous.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;Coming from a cloud and backend background, building my first AI application showed me that practical AI isn't magic—it's good data pipelines, sensible guardrails, and solid systems engineering.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h3&gt;
  
  
  Why I Wrote This &amp;amp; Let's Discuss!
&lt;/h3&gt;

&lt;p&gt;I wrote this post because when I first started exploring AI, most guides assumed you had cloud API keys or a dedicated GPU. If you're a backend or cloud engineer curious about building your first AI project, starting small and local is one of the best ways to understand every moving piece under the hood.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Have you built a local RAG pipeline or deployed small LLMs on CPU?&lt;/strong&gt; &lt;br&gt;
I'd love to hear what worked best for you!&lt;/p&gt;

</description>
      <category>ai</category>
      <category>rag</category>
      <category>python</category>
      <category>backend</category>
    </item>
  </channel>
</rss>
