A large language model is frozen in time and confidently wrong about anything it wasn't trained on. Retrieval-Augmented Generation is how you fix both problems at once — without retraining the model.
RAG has become one of the most important patterns in applied AI, and also one of the most hand-waved. Let me explain what it actually is, how the pipeline works end to end, and where it quietly breaks.
The problem RAG solves
An LLM's knowledge is baked in at training time. It doesn't know your internal documents, last week's data, or the specific bank statement a user just uploaded. Worse, when you ask it something outside its knowledge, it rarely says "I don't know" — it fills the gap with a fluent, plausible, and often wrong answer.
You have two ways to give a model new knowledge. You can fine-tune it — expensive, slow, and a poor fit for facts that change. Or you can retrieve the relevant information at question time and hand it to the model as context. That second approach is RAG, and for factual, changing, or private data it wins almost every time.
The pipeline, step by step
RAG has two phases: an offline indexing phase and an online query phase.
Indexing (done once, ahead of time):
- Chunk your documents into passages — a few hundred tokens each. Chunk too big and retrieval gets noisy; too small and you lose context. This boundary matters more than people expect.
- Embed each chunk into a vector — a list of numbers that captures its meaning, so that passages about similar topics land near each other in vector space.
- Store those vectors in a vector database.
Query (every time a user asks something):
- Embed the question with the same model, then retrieve the top-k chunks whose vectors are closest to it. This is semantic search — it matches on meaning, not keywords.
- Augment the prompt: stitch the retrieved chunks into the context alongside the user's question, with an instruction like "answer using only the context below."
- Generate. The model now answers grounded in real, retrieved text instead of its frozen memory — and you can ask it to cite which chunk each claim came from.
That's the whole idea: retrieve, then generate. The model stops being a memory and becomes a reasoning engine over facts you supply.
Why this beats fine-tuning for facts
Fine-tuning teaches a model new behavior — a tone, a format, a skill. It's a clumsy way to teach it facts, because facts change and retraining is costly. RAG separates the two cleanly: the model handles language and reasoning; your knowledge base handles truth. Update a document and the system's answers update instantly — no training run required.
It's an architecture I lean on across the AI systems I build, because it turns an LLM from an unreliable oracle into something you can actually put in front of users.
Where RAG breaks
RAG is not magic, and most failures are retrieval failures, not generation failures:
- Bad chunking splits a fact across two passages so neither is retrievable whole.
- Retrieval misses — the right chunk exists but doesn't rank in the top-k, so the model never sees it and falls back to guessing.
- Context stuffing — cramming in twenty marginally relevant chunks buries the useful one and confuses the model.
The lesson: in a RAG system, most of your engineering effort belongs in retrieval quality, not prompt wording.
Seeing it in a real system
The most convincing version of RAG is one where the model is never allowed to state a fact directly. In my write-up "Making an LLM trustworthy over bank statements" — a platform that answers plain-English questions over financial documents — the LLM's only job is to generate a SQL query against a strict schema. The database produces the numbers; the model never invents one. It's RAG taken to its logical conclusion: retrieval isn't just context, it's the guardrail.
If you take one thing from this: don't ask an LLM to remember. Give it the right information at the right moment, and let it do what it's actually good at — reasoning over what's in front of it. More of how I apply this is at www.divyakush.com.
Divyakush Punjabi · Full-Stack & AI Engineer
Portfolio · GitHub · LinkedIn
Top comments (0)