I Built a "Chat With Your Docs" Tool and Finally Understood RAG

There's a specific kind of frustration that comes from asking an LLM a question about your own codebase, or your own company's internal docs, and watching it confidently make something up. Not because it's broken because it genuinely has no way of knowing what's in a folder on your laptop, or a Notion page behind your company's login wall. It was trained on a giant snapshot of public text a while ago, and that's the entire universe it has access to unless you feed it something else.

The first time I tried to fix this, my instinct was "fine tuning must be the answer I'll just train it on my data." That turned out to be the wrong tool for the job, and expensive, and slow to iterate on. What I actually wanted has a name: RetrievalAugmented Generation, or RAG. And once I built a small version of it myself, the concept stopped feeling like a buzzword and started feeling almost embarrassingly simple.

Here's what I learned building one, including the parts that tripped me up.

The Idea, Stripped of Jargon

RAG boils down to one sentence: before you ask the model a question, go find the relevant text yourself, and hand it to the model along with the question.

That's the whole trick. Instead of hoping the model "knows" the answer from training, you do the lookup search your documents, pull out the parts that seem relevant and paste those chunks into the prompt. The model then answers based on what's sitting right in front of it, not what it vaguely remembers from training data that might be a year out of date or might not include your content at all.

It's the AI equivalent of an openbook exam. You're not testing what the model memorized. You're handing it the book, open to the right page, and asking it to read and answer.

The reason this works so well in practice is that LLMs are genuinely excellent at synthesizing and summarizing text that's directly in front of them that's much closer to their core strength than recalling obscure facts from training. RAG plays to that strength instead of fighting it.

Why Not Just Paste Everything Into the Prompt?

My first "implementation" of RAG, before I knew what I was doing, was literally copypasting an entire document into the chat window before asking my question. That works fine for a fivepage PDF. It falls apart fast once you're dealing with hundreds of documents, a large codebase, or a knowledge base that doesn't fit in any context window no matter how generous.

Even when things technically fit, there's a cost problem you're paying (in tokens, in latency) to feed the model a bunch of irrelevant text alongside the two paragraphs that actually matter. So the real engineering problem RAG solves isn't "how do I give the model my data," it's "how do I automatically find and hand it only the small slice of my data that's actually relevant to this specific question."

That's where the interesting part starts.

The Pieces That Actually Make It Work

Chunking

You can't search a giant document as one blob effectively, so the first step is breaking it into smaller pieces a paragraph, a few sentences, a section. I underestimated how much this step matters. Chunk too large, and you dilute the relevant bit with a bunch of noise. Chunk too small, and you lose context a sentence pulled out of its surrounding paragraph can be almost meaningless on its own.

I ended up landing on chunks of a few hundred words with a bit of overlap between consecutive chunks, so a sentence that happens to sit right at a chunk boundary doesn't get orphaned from its context. This isn't some universal magic number it took a fair amount of trial and error looking at what actually got retrieved for real questions before it felt right.

Embeddings

This was the part that felt like magic until I sat down and actually understood it. An embedding model takes a piece of text and converts it into a list of numbers a vector positioned in a highdimensional space such that texts with similar meaning end up close together in that space, and unrelated texts end up far apart.

So "the cat sat on the mat" and "a feline rested on the rug" very different words, similar meaning would land near each other in this space. That's the property that makes semantic search possible: you're not matching keywords, you're matching meaning.

You run every chunk of your documents through this embedding model once, ahead of time, and store the resulting vectors. When a user asks a question, you embed the question the same way, then look for the stored chunks whose vectors are closest to it.

Vector Storage and Search

Those embeddings need to live somewhere you can search efficiently. For a side project, something lightweight is genuinely enough I started with a simple local vector store and only reached for something like Pinecone or a managed vector database once I actually had a scale problem, not before. A common mistake I see (and made myself) is reaching for heavy infrastructure before confirming the basic approach even works on a small scale.

The search step itself is just "find the k chunks whose vectors are closest to the question's vector" usually using cosine similarity. In practice, retrieving somewhere around three to eight chunks worked well for me; too few and you risk missing the answer, too many and you're back to diluting the prompt with noise.

Assembling the Prompt

This is the step that's easy to gloss over but matters a lot. Once you have your retrieved chunks, you don't just dump them in you construct a prompt that makes it explicit what the model should do with them. Something along the lines of:

You are answering questions using only the context provided below.
If the answer isn't in the context, say you don't know  don't guess.

Context:
[retrieved chunk 1]
[retrieved chunk 2]
[retrieved chunk 3]

Question: {user's question}

That instruction to say "I don't know" rather than guess turned out to be one of the highest-leverage lines in the whole system. Without it, the model will happily blend its own background knowledge with your retrieved context, and you lose the whole point of grounding it in your actual data. With it, you get much more honest "the docs don't cover this" responses instead of confident nonsense.

Where Mine Broke (And What That Taught Me)

The first version I built worked shockingly well on obvious test questions and then fell over on real ones. A few lessons from debugging it:

Retrieval quality matters more than prompt cleverness. I spent an embarrassing amount of time tweaking my prompt template before realizing the actual problem was upstream the wrong chunks were being retrieved in the first place. No prompt engineering fixes bad retrieval. If the right information never makes it into the context, the model has no chance.

Questions phrased differently than the source text can miss. Semantic search is good, not psychic. A question asking "how do I cancel my plan" might not retrieve a doc chunk titled "Subscription Termination Process" as strongly as you'd hope, depending on the embedding model. This pushed me toward also trying a hybrid approach combining semantic search with old-fashioned keyword search which noticeably improved recall on edge cases.

Stale data is a silent failure mode. If your source documents change and you don't re-run the embedding step on the updated content, you'll keep serving confidently wrong answers based on outdated chunks, and nothing in the system will warn you. This is an unglamorous but very real operational concern once you're running one of these for real.

Evaluating "is this a good answer" is genuinely hard. Unlike a typical software bug, there's no clean pass/fail. I ended up building a small set of test questions with known correct answers and periodically eyeballing whether changes to chunking or retrieval helped or hurt, which is a pretty manual process but better than flying blind.

Where This Fits Into the Bigger Picture

Once I understood RAG at this level, a lot of AI products I'd used casually suddenly made more sense. "Chat with your PDF" tools, internal company knowledge bots, customer support assistants that somehow know your specific order history most of these aren't relying on some enormous finetuned model that magically knows everything. They're running some version of the retrievethengenerate loop described above, often with a fair amount of extra plumbing around chunking strategy, reranking, and prompt construction.

It also reframed how I think about "does the model know X." The more useful question, most of the time, isn't whether the model was trained on something it's whether you can retrieve the right information and hand it over at the moment it's needed. That shift in thinking has been more useful to me day to day than any specific prompting trick.

If you're looking for a genuinely good weekend project to understand this stuff at a level deeper than reading about it, building a small RAG pipeline over your own notes or a folder of PDFs is a great one. It's small enough to finish in a weekend, and it touches almost every practical concept that shows up in production AI systems embeddings, search, prompt construction, and the very real gap between "works on my test question" and "works reliably."