AI-RAG-Bot — chat with your documents. 100% local, 100% private, $0 to run

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

AI-RAG-Bot is a local "chat with your documents" assistant: drop files into a folder, ask questions in your terminal, and get answers grounded strictly in those files. It reads .txt, .pdf, and images — and for scanned PDFs and photos of documents, it runs OCR (Tesseract + OpenCV), so even a photographed receipt becomes searchable knowledge.

Who would pay for this? Anyone who handles documents they're not allowed to paste into a cloud chatbot:

• A freelancer whose client contracts are under NDA
• An accountant handling clients' tax documents
• A law student working through case files
• A small clinic with scanned intake forms

For all of them, the "easy" option today — pasting a client PDF into a hosted AI service — means shipping sensitive data to a third-party server. AI-RAG-Bot's entire value proposition is that nothing ever leaves the laptop: embeddings live in a local ChromaDB, the model runs through Ollama, and the only network call is the initial model download.

And because there's no per-query API bill, the unit economics work: set it up once for a client and it costs $0/month to run, forever. Try building that margin on top of a closed API.

Code

sanyamkansal / AI-RAG-Bot

Local RAG system for intelligent document question answering using OCR, vector search, and a local LLM

AI RAG Bot

AI RAG Bot is a simple, locally-hosted Retrieval-Augmented Generation (RAG) system built with Python. It allows you to query a knowledge base of your own documents, including text files, PDFs, and images (using OCR). The bot processes these documents, creates vector embeddings, stores them in a local ChromaDB instance, and leverages an LLM via Ollama to generate answers grounded in the retrieved document context.

Key Features

• Multi-format Document Support: Process .txt, .pdf, .png, .jpg, and .jpeg files.
• Optical Character Recognition (OCR): Extracts text from images and PDFs using Tesseract OCR, PyMuPDF, and OpenCV.
• Smart Chunking: Cleans and chunks document text to fit context windows efficiently.
• Local Vector Database: Uses chromadb for persistent, local storage of vector embeddings.
• Sentence Embeddings: Utilizes the lightweight and fast all-MiniLM-L6-v2 model from sentence-transformers to generate embeddings.
• State Tracking: Keeps track of document modification times and sizes (document_state.json…

View on GitHub

How I Built It

Every layer of the stack is open source, and the open pieces are exactly what make the project work:

• Local inference: Ollama serves Google's open-weight gemma3:1b — the smallest Gemma variant, about 1 GB, so it runs on modest laptops with no GPU. RAG keeps prompts short and grounded in retrieved chunks, which is exactly why a 1B model is enough here: it answers from the provided context and says "information not found" when the answer isn't in the documents.
• Embeddings: all-MiniLM-L6-v2 from sentence-transformers — small, fast, CPU-friendly.
• Vector store: ChromaDB in persistent local mode. The entire "database" is a folder on disk.
• Extraction: PyMuPDF for text PDFs; OpenCV + pytesseract (Tesseract OCR) for scans and images.
• State tracking: document_state.json remembers each file's size and modification time, so unchanged documents aren't re-processed on startup.

The pipeline: scan documents/ → extract text (OCR where needed) → clean → chunk (~500 characters) → embed → store. At query time, the question is embedded too, ChromaDB returns the most similar chunks, and gemma3:1b is prompted to answer only from those chunks. That "I couldn't find that" behavior matters more than raw cleverness when someone is asking about a contract.

Why Does Open Innovation Matter?

1. Privacy a closed API structurally can't offer. This isn't a policy promise ("we don't train on your data, trust us") — it's architecture. There is no server to send anything to.
2. Cost. Embeddings and generation run on hardware the user already owns. For the freelancer use case, that's the difference between a tool free to run and one with a monthly bill per seat — the reason this side hustle works at all.
3. Freedom to swap. Want a different model? ollama pull anything and change one string. Different embedder? One line. No vendor decides what this tool can become.
4. Offline by default. It works on a plane, in a basement archive, anywhere with no internet. Only a local stack can promise that.

Prize Categories

• Best Use of Gemma — AI-RAG-Bot runs Google's open-weight gemma3:1b as its generation model, served entirely locally through Ollama. No API key, no cloud calls: the model lives on the user's own machine.