I Built a RAG App on My Laptop Without Paying OpenAI a Single Rupee Here's How

A few weeks ago I got tired of watching my API usage bill creep up every time I tested a new idea for a document Q&A tool. Every experiment, every "let me just try this real quick" moment, was costing me money and, honestly, killing my momentum. So I decided to rip the whole thing out and rebuild it to run completely locally no API keys, no rate limits, no internet dependency once the models were downloaded.

What I ended up with was a fully working Retrieval Augmented Generation (RAG) app running on my own machine, chatting with my own PDFs, at 2 AM, with my WiFi turned off just to prove a point to myself.

If you've been wanting to understand RAG beyond the buzzword and actually build something with it instead of just reading about it this is the guide I wish existed when I started. I originally wrote a deeper technical breakdown of this build over on my blog at nileshblog.tech, but I wanted to give the dev.to community the practical, no fluff version here.

Wait, What Even Is RAG?

Let's kill the jargon first. A plain LLM is like a brilliant friend who read a huge chunk of the internet years ago and then got locked in a room with no news, no updates, and no idea what happened after that. Ask it about your company's internal docs, your personal notes, or anything published after its training cutoff, and it'll either shrug or worse confidently make something up.

RAG fixes that by giving your model a library card. Instead of relying purely on what it memorized, the app:

1. Takes your question
2. Goes and searches your own documents for relevant chunks of text
3. Hands those chunks to the LLM along with your question
4. The LLM answers using that fresh, specific context

That's it. That's the whole trick. No magic, just a smart plumbing system connecting a search engine to a language model.

Why Build It Locally Instead of Using an API?

I'm not anti API OpenAI, Anthropic, and others make genuinely great hosted models. But building locally first taught me more about how RAG actually works than any hosted tutorial ever did. A few real reasons to go local:

• Privacy your documents never leave your machine
• Cost zero token bills while you're iterating and breaking things
• Offline capability genuinely useful for travel, poor connectivity, or paranoid system architecture
• Understanding you stop treating the LLM as a black box

The Stack I Used

Here's the combination that got me a working local RAG app in an afternoon:

• Ollama to run open source LLMs like Llama 3 or Mistral locally with almost zero setup pain
• LangChain (or LlamaIndex, pick your poison) to handle the document loading, chunking, and retrieval pipeline
• ChromaDB a lightweight local vector database to store your document embeddings
• Python obviously, the glue holding everything together

None of these need a GPU cluster. A decent laptop with 16GB RAM handles smaller models comfortably.

Step 1: Get Ollama Running

Download Ollama from its official site, install it, then pull a model:

ollama pull llama3

This downloads the model weights locally. Once done, you can chat with it directly from the terminal but we want it wired into our RAG pipeline instead.

Step 2: Set Up Your Python Environment

python -m venv rag-env
source rag-env/bin/activate
pip install langchain langchain-community chromadb pypdf

Step 3: Load and Chunk Your Documents

This is the step everyone underestimates. Bad chunking = bad retrieval = bad answers, no matter how good your model is.

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("my_notes.pdf")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100
)
chunks = splitter.split_documents(docs)

Keep chunks small enough to be precise, but with enough overlap that you don't slice a sentence in half and lose meaning.

Step 4: Create Embeddings and Store Them

from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OllamaEmbeddings(model="llama3")
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

This converts every chunk into a vector a mathematical fingerprint of its meaning and stores it so we can search by meaning, not just keywords.

Step 5: Build the Retrieval + Generation Chain

from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA

llm = Ollama(model="llama3")
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True
)

response = qa_chain.invoke({"query": "Summarize the key points about deployment"})
print(response["result"])

That's a working local RAG pipeline. Ask it something specific to your documents and watch it pull real, grounded answers instead of hallucinated guesses.

Where People Usually Get Stuck

A few honest lessons from my own trial and error:

• Chunk size matters more than model choice. I spent hours blaming the LLM when the real problem was chunks that were too large and diluted with irrelevant text.
• Retrieval count (k) is a balancing act. Too few chunks and the model lacks context; too many and it gets distracted or hits token limits.
• Not every open model is equally good at following instructions. Llama 3 and Mistral both work well, but test a couple before settling.
• Persist your vector store. Don't rebuild embeddings every single run it's slow and pointless once your documents are indexed.

What's Next

Once this basic pipeline works, the fun part begins adding a simple Streamlit or Gradio front end, supporting multiple file types, adding conversation memory, or swapping ChromaDB for something like Qdrant if you're scaling up. I'm actively documenting these next steps and sharing deeper walkthroughs, benchmarks, and real project breakdowns over at nileshblog.tech — if you want to go further down the local AI rabbit hole, that's where I'm posting the rest of this journey.

Final Thoughts

Building RAG locally isn't just a cost saving trick it genuinely changes how you think about LLM applications. You stop treating the model as a mysterious oracle and start seeing it as one component in a system you fully control. And honestly, there's something deeply satisfying about asking your laptop a question about your own notes, offline, and watching it answer correctly.

If you build your own version of this, I'd love to hear what stack you used and what broke along the way that's usually where the real learning happens.