Every time we query a cloud-based LLM with proprietary documents, there’s a quiet compromise on data privacy. What if you could run a fully private, offline Document Q&A system directly on your local machine—powered by open-source models—with zero API costs?
In this guide, we’ll build a lightweight, offline Retrieval-Augmented Generation (RAG) pipeline using Ollama, LangChain, and a clean Streamlit UI.
Prerequisites & Tech Stack
To run this setup locally, you’ll need:
Python 3.10+
Ollama installed locally (running models like Llama 3 or Qwen 2.5)
A decent GPU (such as an RTX 4060 or higher) for fast local vector embedding and token generation.
Step 1: Install Dependencies
Create a new project folder, set up a virtual environment, and install the required core packages:
Bash
pip install streamlit langchain langchain-community chromadb sentence-transformers ollama
Step 2: Initialize the Local Ollama Embedding & LLM
Instead of calling OpenAI, we hook directly into our local Ollama instance running offline:
Python
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.llms import Ollama
Connect to local Ollama instance
embeddings = OllamaEmbeddings(model="qwen2.5-coder") # or llama3
llm = Ollama(model="qwen2.5-coder")
Step 3: Process Documents and Build the Vector Store
We use DirectoryLoader to ingest local text or PDF files, chunk them using RecursiveCharacterTextSplitter, and store them in a local Chroma vector database:
Python
from langchain_community.document_loaders import DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
Load documents from a local folder
loader = DirectoryLoader("./docs", glob="*/.txt")
documents = loader.load()
Split text into manageable chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = text_splitter.split_documents(documents)
Create local vector store
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
Step 4: Build the Streamlit Frontend
Create an interactive UI (app.py) allowing users to upload documents and query them instantly:
Python
import streamlit as st
from langchain.chains import RetrievalQA
st.title("🔒 Private Local RAG Chatbot (100% Offline)")
st.sidebar.header("Configuration")
st.sidebar.success("Running locally via Ollama")
query = st.text_input("Ask a question about your local documents:")
if query:
qa_chain = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
with st.spinner("Generating answer locally..."):
response = qa_chain.run(query)
st.write("### Answer:")
st.write(response)
Wrap Up & Production-Ready Starter Kit
Running this locally gives you total data sovereignty and zero reliance on external APIs.
💡 Want to skip the boilerplate setup?
If you want the complete, production-ready source code featuring advanced document chunking, PDF/DOCX multi-format parsers, clean chat memory, and an optimized setup.bat script, you can grab the turn-key Private Local RAG Starter Kit directly from my store here:
👉 https://ko-fi.com/s/fd0a0250da
Happy coding! Let me know in the comments if you run into any local VRAM or embedding bottlenecks.
Top comments (0)