DEV Community

Cover image for How RAG Actually Works in Production (The Model Is the Easy Part)
RONI DAS
RONI DAS

Posted on • Originally published at systemdesign.academy

How RAG Actually Works in Production (The Model Is the Easy Part)

Ask most people how a retrieval-augmented generation system works and you get an answer that is really about the language model. You put your documents somewhere, the model reads them, and it answers your question. The model is the part everyone talks about, because the model is the part that feels like magic. But if you have ever shipped one of these systems and watched it confidently answer the wrong question, or miss an answer that was sitting right there in your data, you already know the uncomfortable truth. The model is the easy part. RAG is a retrieval problem wearing a language model's clothes, and it lives or dies on retrieval quality long before the model ever gets a turn.

This piece is the long version. We will start with why a raw model is not enough and what RAG actually fixes, then walk the ingestion pipeline that runs offline, chunking and embedding and indexing. Then we will follow a single query through retrieval, reranking, and prompt assembly, and see how the model generates an answer that is grounded in what we found. After that we get to the parts that decide whether the thing survives contact with real users: the failure modes, how you evaluate a system whose two halves fail differently, and how you keep the index from going stale. At the end we pull out the patterns that transfer to systems that have nothing to do with language models. I am not going to quote any vendor's internal numbers. Where a figure matters, chunk sizes, how many passages to retrieve, embedding dimensions, I give the industry-typical range and say so, because the reasoning is what transfers, not a precise number I cannot stand behind.

Why a raw model is not enough

A language model knows what it was trained on, and nothing else. Its knowledge is frozen at a training cutoff, so it has never seen anything that happened after that date. It has never seen your company's internal wiki, your customer's support tickets, your product's current pricing, or the contract a user uploaded thirty seconds ago. None of that was in the training data, and none of it can be, because that data is private, or new, or both.

how-rag-works diagram
A raw model answers from frozen training knowledge and invents plausible detail when asked beyond it; RAG grounds the answer in documents retrieved at query time.

Now watch what happens when you ask such a model a question about data it has never seen. It does not stop and say it does not know. That is not how these models work. They are trained to produce fluent, plausible continuations of text, and a confident wrong answer is more fluent than an admission of ignorance. So it fills the gap with something that sounds right, invented citations, a policy that does not exist, a number in the correct shape but wrong. We call this hallucination, and it is not a bug you can patch out. It is the model doing exactly what it was built to do, applied to a question it has no grounds to answer.

RAG addresses this directly. Instead of asking the model to answer from memory, you first retrieve the relevant documents from your own data, then hand those documents to the model along with the question, and instruct it to answer only from what you gave it. The knowledge no longer lives in the model's frozen weights. It lives in a corpus you control, that you can update the moment a document changes, and the model's job shrinks to reading and summarizing text that is right in front of it. That is a much easier job, and a much safer one. The catch, and the whole rest of this article, is that the answer can only ever be as good as the documents you retrieved. If retrieval hands the model the wrong passages, the model will faithfully summarize the wrong passages, and now you have a fluent, well-cited, confidently wrong answer that is harder to catch than the original hallucination.

A fair question at this point is why not fine-tune the knowledge into the model instead. Fine-tuning is for teaching behavior, format, tone, and task, but it is a poor and costly way to inject facts that change, and it cannot cite a source. RAG owns changing knowledge, fine-tuning owns behavior, and mature systems use both.

The pipeline at a glance

Before we go part by part, here is the whole shape, because it is hard to reason about any single stage without knowing where it sits.

how-rag-works diagram
Two pipelines: offline ingestion loads, chunks, embeds, and indexes the corpus; at query time the system embeds the query, retrieves, reranks, assembles a prompt, and generates.

There are really two systems here, and they run at different times. The first is ingestion, and it runs offline, ahead of any user. You load your source documents, split them into passages, convert each passage into a vector (a list of numbers) using an embedding model (a network that maps text to such a vector), and store those vectors in an index built for fast similarity search. This is a batch job, or a streaming pipeline, but either way the user is not waiting on it. Its output is a searchable index.

The second system runs at query time, while a user waits, and every millisecond counts. You take the user's question, embed it with the same model you used on the documents, search the index for the passages closest to it, optionally rerank those candidates for precision, assemble the best of them into a prompt that fits the model's context window (the fixed amount of text it can read at once), and finally send that prompt to the model to generate an answer. Retrieval quality is decided almost entirely in the offline pipeline, but it is felt entirely at query time. Most teams pour their attention into the query-time model call, the visible part, and underinvest in ingestion, the invisible part, which is exactly backwards. Get ingestion wrong and there is no prompt clever enough to save you.

Chunking: the decision that quietly sets your ceiling

The first real decision in ingestion is how to cut your documents into pieces. You cannot embed a whole hundred-page manual as one vector, and you would not want to. So you split documents into chunks, passages of a few sentences to a few paragraphs, and each chunk becomes a unit of retrieval. This sounds like plumbing. It is actually one of the highest-leverage decisions in the entire system, and it is where a lot of RAG systems are silently broken from day one.

how-rag-works diagram
Chunks too large dilute the embedding and waste context; too small lose meaning; a middle size with overlapping edges keeps each passage coherent at its boundaries.

The tension is easy to state and hard to resolve. Make chunks too large and two bad things happen. The embedding of a big chunk is an average of many ideas, so it points nowhere in particular and matches queries weakly, the signal for any one fact diluted by everything around it. And a large chunk eats your context budget at query time, so you can afford to include fewer of them. Make chunks too small and you get the opposite problem. A single sentence pulled out of its surroundings often loses the meaning that made it useful. "It supports up to 200 connections" is useless if the chunk does not carry what "it" refers to. You retrieve a fragment that matches the words of the query but cannot actually answer it.

The practical middle ground for prose is a chunk on the order of a few hundred tokens, but the exact number matters less than two techniques that address the boundary problem directly. The first is overlap. You let consecutive chunks share some text at their edges, so a sentence that would otherwise be orphaned at a boundary appears in full in at least one chunk. The second, and more important, is structure-aware splitting. Instead of cutting blindly every N characters, you split on the document's own structure, on headings, paragraphs, list items, code blocks, table rows. A chunk that respects a section boundary carries a complete thought; a chunk that cuts a sentence in half carries neither half's meaning. For structured content like Markdown or HTML or source code, splitting on the structure rather than on raw length is often the single biggest quality improvement available to you, and it costs nothing but a better parser.

One more technique decouples the unit you retrieve from the unit you send. You index small precise chunks for matching, but at retrieval time you expand each hit to its surrounding window or parent section, so the model reads the fragment in context rather than alone. This is often called small-to-big or parent-document retrieval.

Embeddings: turning meaning into geometry

Once you have chunks, you need a way to find the ones relevant to a question. Keyword matching alone is not enough, because a user who asks "how do I reset my password" should find a document titled "recovering account access" even though they share almost no words. You need to match on meaning, not on spelling, and that is what embeddings give you.

how-rag-works diagram
An embedding model maps text to a vector so that semantic similarity becomes geometric closeness; query and chunks share one vector space.

An embedding model is a neural network trained to map a piece of text to a vector, a list of numbers, typically several hundred to a couple of thousand of them. The training objective is what makes this useful. The model is trained so that texts with similar meaning land close together in this high-dimensional space, and texts with different meaning land far apart. "Reset my password" and "recover account access" end up as nearby points even though the surface words differ, because the model learned that they mean nearly the same thing. Semantic similarity, a fuzzy human notion, becomes geometric distance, a number you can compute.

The crucial discipline here is that you must embed your chunks and your queries with the same model, so that they live in the same space and their distances are comparable. Mixing embedding models, or comparing vectors from different model versions, produces distances that mean nothing. Embedding quality is also domain-specific, so a model strong on general text can be weak on specialized jargon, and many retrieval models encode a short query and a long passage differently, which is part of why query and chunks must go through the same model in the same way. Closeness is commonly measured with cosine similarity, which compares the direction two vectors point rather than their magnitude, so it cares about what a text is about rather than how long it is; note this is a similarity rather than a distance, since cosine distance is one minus it. Dot product and Euclidean distance are also widely used. Once every chunk is a vector and every query is a vector in that same space, "find the relevant passages" reduces to a purely geometric question: which chunk vectors are closest to the query vector? That is a question we know how to answer fast, which is the subject of the next section.

The vector index: why you cannot just compare everything

Finding the closest chunk vectors to a query vector sounds simple, and for a small corpus it is. Compute the distance from the query to every chunk, keep the closest handful, done. This is exact nearest-neighbor search, and its cost grows in direct proportion to the number of chunks. That is fine for a thousand chunks and hopeless for ten million, because you would be doing ten million distance computations on every single query, written O(n), while a user waits.

how-rag-works diagram
Exact search compares the query to every vector, O(n) per query; ANN structures like HNSW and IVF trade a little recall for large speedups.

The escape is to give up on finding the exact closest vectors and settle for finding almost-certainly the closest ones, much faster. This is approximate nearest neighbor search, ANN, and it is what every production vector index actually runs. The trade is explicit and tunable: you accept that occasionally the index will miss a true nearest neighbor, in exchange for answering in sublinear time instead of linear. Graph indexes like HNSW get roughly logarithmic behavior; cluster indexes like IVF are sublinear but not that aggressive. The fraction of true nearest neighbors the index actually returns is its recall, and you trade recall against speed: higher recall costs more time, lower recall buys speed.

Two families of structure dominate. HNSW, hierarchical navigable small world graphs, builds a layered graph where each vector links to its neighbors, with sparse long-range links up top for fast traversal and dense links at the bottom for precision. A search enters at the top, greedily hops toward the query, and descends layer by layer, reaching the neighborhood of the answer in a small number of hops rather than scanning everything. IVF, the inverted file approach, first clusters all the vectors into buckets, then at query time only searches the few buckets nearest the query, skipping the rest entirely. Both approaches, and the quantization tricks often layered on top to shrink the vectors in memory, are answering the same question: how do I avoid comparing the query to everything? The answer is always the same in spirit, spend some memory and some recall to buy a large cut in the number of comparisons, and it is the same bargain a database makes when it builds an index instead of scanning a table.

The two families also differ in operations, not just search. IVF must be trained on a representative sample to build its clusters and needs retraining as the data drifts, while HNSW needs no training but is costly to update, with deletes usually tombstoned and the graph degrading under heavy churn until a rebuild. That update cost often decides which index a team can live with.

Query-time retrieval

Now we are at query time, with a built index waiting. A user asks a question, and the first thing that happens is the same thing that happened to every chunk during ingestion: the question is embedded, with the same model, into a vector in the same space.

how-rag-works diagram
At query time the question is embedded with the same model, and ANN search returns the top-k chunks by vector similarity.

With the query vector in hand, you ask the index for the top-k nearest chunk vectors, where k is a number you choose, commonly somewhere between five and a few dozen depending on how much context you can afford downstream. The index runs its ANN search and hands back that many candidate chunks, each with a similarity score, ranked closest first. That is the entire first-stage retrieval, and it is fast, typically on the order of milliseconds to low tens of milliseconds even over millions of vectors, depending on how the index is tuned, which is the whole point of the index. In practice the vector search is usually constrained by metadata filters (date, source, language, version, tenant), which both improves relevance and is the hook where access control is enforced. That access control is not optional: in any multi-tenant or permissioned corpus, retrieval must filter by the caller's access rights before the model ever sees a chunk, because a vector index has no notion of who may read what, and a leaked passage becomes a cited, fluent answer.

But be honest about what this score is and is not. The similarity score tells you how close two embeddings are, and embeddings compress a passage down to a few hundred to a few thousand numbers, which necessarily throws information away. Two chunks can score nearly identically against a query while only one actually answers it, because the compression blurred the distinction that mattered. First-stage vector retrieval is very good at pulling a relevant neighborhood out of millions of chunks, and only roughly good at ordering within that neighborhood. That gap between "in the right neighborhood" and "the actual best answer, ranked first" is exactly what the next two sections exist to close. You choose k larger than the number of chunks you ultimately want, precisely so that the real answer is somewhere in the candidate set even if it is not yet ranked first.

Hybrid retrieval: dense search has blind spots

Dense vector search, which matches on meaning, is strong but weak in a specific, predictable way. Because it matches on semantic similarity, it can miss the cases where the exact token is what matters. A part number like "X-4021-B", an error code, a person's surname, a rare technical term the embedding model barely saw in training, these are precisely the queries where "close in meaning" fails you, because the thing you need is an exact string, not a paraphrase.

how-rag-works diagram
Dense vector search misses exact keywords, IDs, and rare terms; combining it with sparse keyword search like BM25 and fusing the rankings covers both.

The fix is not to abandon vector search but to pair it with the older technology it was supposed to replace. Sparse keyword search, most commonly BM25, ranks documents by exact term matches weighted so that rare terms count for more and common ones count for less. BM25 is excellent at exactly what dense search is bad at: finding the document that contains this specific string. So you run both. The query goes to the vector index and to a keyword index at the same time, and each returns its own ranked list of candidates. This is hybrid retrieval, and in practice it beats either method alone across a wide range of real queries, because real query logs contain both "explain our refund philosophy," which wants semantic matching, and "what is fee code 30983," which wants exact matching.

The remaining question is how to combine two ranked lists whose scores are not comparable, a cosine similarity and a BM25 score living on different scales. The common answer is reciprocal rank fusion, which throws away the raw scores and combines the lists by rank position instead, rewarding documents that rank highly in either list. It is simple, it needs no score calibration, and it is hard to beat. The lesson underneath is worth keeping: when a single retrieval method has a known blind spot, the cheapest fix is often a second method with the opposite blind spot, fused together, rather than an heroic effort to make the first method perfect.

Reranking: precision on the shortlist

First-stage retrieval, dense or hybrid, is built for one job, to cut millions of chunks down to a few dozen candidates cheaply. It is deliberately not built for precision in ordering, because the thing that would give you precision is too expensive to run over millions of chunks. So you run it in two stages. Cheap and approximate first to get the shortlist, then expensive and precise on just that shortlist. This retrieve-then-rerank pattern is one of the most important structures in production RAG.

how-rag-works diagram
A cross-encoder rescores each candidate together with the query for far better precision than the first-stage score, at higher cost, so it runs only on the shortlist.

The reason first-stage retrieval cannot be precise is architectural. To make the index fast, you embed each chunk independently, ahead of time, with no knowledge of the query. The query and the chunk never actually meet; you only compare their two pre-computed vectors. A reranker throws out that separation. It is a cross-encoder, a model that takes the query and one candidate chunk together, as a pair, and reads them jointly to produce a single relevance score. Because it sees both texts at once, it can judge whether this chunk actually answers this query, catching relevance that the blurred, independent embeddings missed. It is far more accurate.

It is also far more expensive, and that is why it cannot run over your whole corpus. A cross-encoder has to run a full model forward pass for every query-chunk pair, so its cost scales with the number of candidates. Running it on ten million chunks per query is out of the question; running it on the fifty candidates that first-stage retrieval already shortlisted is entirely affordable. So the two stages divide the labor by their nature. The fast approximate stage handles the impossible scale, taking millions down to dozens. The slow precise stage handles the ordering, taking dozens down to the handful you will actually show the model, correctly ranked. You get most of the accuracy of comparing the query against everything, at a tiny fraction of the cost, which is the entire art of two-stage retrieval. It is worth holding a latency and cost budget for the whole path, since the reranker and the generation call, not the vector search, usually dominate both.

Prompt assembly and the context budget

Now you have a small set of high-quality, well-ranked chunks. The next job is to build the actual prompt you send to the model, and this is less obvious than it looks. The model has a context window, a hard limit on how many tokens it can read at once, and that budget has to hold the system instructions, the retrieved chunks, the user's question, and enough headroom for the answer the model still has to write. You cannot just dump everything in.

how-rag-works diagram
Fit the best-ranked chunks into the context window alongside instructions and question, keep the strongest chunks, leave room for the answer, and stop well short of the limit, because more context is not better.

The instinct, once context windows got large, was to stop worrying and stuff in as many chunks as fit. That instinct is wrong, and for reasons that go beyond cost and latency, though those are real too, since every token you send is money and time. The deeper problem is that more context often makes the answer worse. Models attend unevenly across a long context, and irrelevant passages act as distractors that pull the answer off course. Ten sharp, relevant chunks produce a better answer than fifty chunks where forty are noise. So prompt assembly is an editing job, not a dumping job. You take the reranked list and include the best chunks until you hit a sensible budget, well short of the physical maximum, and you stop.

Two details in assembly pay off later. First, tag each chunk with its source, a document title, a URL, an ID, so the model can cite where each claim came from and so you can trace any answer back to the passage that produced it. This is not decoration; it is how you debug the system and how a user learns to trust it. Second, order matters. Because models weight the beginning and end of a context more heavily than the middle, a phenomenon we will come back to under failure modes, it is worth placing the strongest chunks where the model attends best rather than burying them in the center. The prompt you assemble is the model's entire world for this one question. Everything it will say comes from what you chose to put in front of it, which is another way of saying the quality ceiling was set upstream, in retrieval, and prompt assembly only decides how much of that ceiling you actually reach.

Generation, grounded and honest

Finally the model gets its turn, and after all the retrieval machinery, its job is deliberately narrow. It is not being asked what it knows. It is being asked to read the passages you retrieved and answer the question from them, and only from them.

how-rag-works diagram
The model answers strictly from the retrieved passages, is instructed to say it does not know when the context lacks the answer, and cites the sources it used.

The instructions you give it are what make this trustworthy, and they run against the model's natural tendencies. You tell it to answer using only the provided context. You tell it, explicitly, that if the context does not contain the answer, it should say so plainly rather than reaching into its training knowledge or inventing something. This matters because a model's default, as we saw at the start, is to produce a fluent answer whether or not it has grounds for one, and "I do not know based on the provided documents" is a far more useful response than a confident fabrication. Getting the model to abstain when the context is insufficient is one of the highest-value behaviors you can instruct, and one of the hardest to enforce completely. You also ask it to cite, to point each claim back to the source chunk it came from, which turns the answer from an assertion into something a user can verify.

Grounding is the word for tying the answer to the retrieved evidence, and it is the whole reason RAG reduces hallucination. But notice the limit precisely. Grounding constrains the model to the context; it does not make the context correct. If retrieval delivered the wrong passages, the model will ground its answer firmly in the wrong passages and cite them neatly. The model has no way to know that the retrieval upstream failed. This is why, no matter how good your prompt and your instructions, you keep coming back to the same sentence: retrieval quality caps everything. The model is the last stage and the least of your worries. What you feed it is the whole game.

Where RAG breaks

A demo RAG system works on the first ten questions you try. A production one has to work on the ten thousandth question asked by someone who phrases things strangely, about a document that changed last week. The gap between those two is a set of specific, recurring failure modes, and knowing them by name is most of what separates a system that degrades gracefully from one that fails silently.

how-rag-works diagram
Four common failures: retrieval misses the relevant chunk, the right chunk is retrieved but buried, the index is stale, and the model ignores the context and hallucinates anyway.

The first and most common failure is a retrieval miss. The chunk that answers the question exists in your corpus, but retrieval did not return it, so it never reached the model. This is a recall failure, and it is invisible unless you measure for it, because the model will happily produce an answer from whatever it did get. The causes trace straight back to earlier decisions: chunks split so the answer is fragmented across a boundary, a query whose wording is semantically distant from the passage, a rare term that dense search glossed over and that no keyword search was there to catch. Most retrieval misses are chunking and hybrid-retrieval problems in disguise. One further fix is to transform the query before retrieval, rewriting a terse or conversational question into a fuller standalone query or expanding it with likely synonyms, so the embedded query lands nearer the passage that answers it. Questions that need evidence found only after a first lookup push toward iterative retrieve-reason-retrieve loops, at the cost of added latency and more places to fail.

The second failure is subtler. The right chunk was retrieved, but it was ranked low and buried in the middle of a long context, and the model, attending less to the middle, effectively overlooked it. This is the "lost in the middle" effect, and it is why reranking and disciplined, shorter contexts matter so much: it is not enough to retrieve the answer, you have to place it where the model will actually read it. The third failure is staleness. The corpus moved, a policy changed, a document was deleted, but the index still holds the old vectors, so the system confidently serves outdated answers, which we address in the next section. The fourth is the one people fear most and that grounding is meant to prevent: the model ignores the provided context and answers from its training memory anyway, hallucinating in spite of correct retrieval. It happens less with good instructions and good models, but it happens, which is why you verify groundedness rather than assume it. These four fail in different places, and no single fix touches all of them, which is exactly why you have to measure the two halves separately.

Evaluating a two-part system

You cannot improve what you do not measure, and RAG is deceptively hard to measure because it has two halves that fail differently, and a single end-to-end "was the answer good" score cannot tell you which half failed. An answer can be wrong because retrieval never found the evidence, or because the model fumbled evidence it was handed. Those need opposite fixes. So you evaluate retrieval and generation separately.

how-rag-works diagram
Retrieval metrics like recall@k and precision measure whether the right chunks were found; generation metrics like faithfulness and answer relevance measure what the model did with them.

On the retrieval side, the metrics are the familiar ones from information retrieval, and they need a set of questions with known relevant chunks to score against. Recall@k asks whether the relevant chunk was somewhere in the top k you retrieved, and it is the single most important number in the whole system, because a chunk that was never retrieved can never be used, no matter how good everything downstream is. If recall@k is low, nothing else you tune will save you, and you go back to chunking, embeddings, and hybrid retrieval. Precision asks what fraction of what you retrieved was actually relevant, which matters because irrelevant chunks are the distractors that hurt generation. Ranking-aware metrics tell you not just whether the right chunk was retrieved but whether it was ranked near the top, which is what reranking is meant to move.

On the generation side, the questions are different. Faithfulness, also called groundedness, asks whether every claim in the answer is actually supported by the retrieved context, or whether the model added things that were not there. This is your direct hallucination measurement, and it is the one to watch most closely, because a faithful answer that admits ignorance is safer than an unfaithful one that sounds complete. Answer relevance asks whether the answer actually addressed the question the user asked, rather than something adjacent. These generation metrics are increasingly scored by using a strong model as a judge against the context, which is imperfect and needs its own spot checks against human judgment, but is far better than flying blind. The discipline that matters is the separation itself. When quality drops, the first question is always which half broke, and only separate metrics can answer it.

Keeping the index fresh

The last production concern is time. Your corpus is not a fixed thing you index once. Documents get edited, added, and deleted continuously, and a RAG system whose index reflects last month's data will confidently serve last month's answers. Ingestion is not a one-time job you run at launch. It is a pipeline that has to run for as long as the system lives.

how-rag-works diagram
Documents change, so ingestion runs continuously: incremental upserts for new and edited content, and deletes for removed content, so the index never drifts from the corpus.

The mechanics are a set of operations you have to build for from the start. When a document is added or edited, you re-chunk and re-embed just that document and upsert its vectors into the index, updating in place rather than rebuilding the world. When a document is deleted, you must delete its vectors too, and this is the step teams forget, which is how a system keeps citing a document that no longer exists. Detecting what changed is its own small problem, usually solved with content hashes or modification timestamps so you re-process only what actually moved rather than re-embedding the entire corpus on every run, which would be ruinously expensive at scale. Repeated work is also cached, embeddings for unchanged content and answers or retrieval results for common queries, with care to invalidate when documents change.

Two larger events force bigger work. The first is changing your embedding model. Because vectors from different models are not comparable, upgrading to a better embedding model means re-embedding your entire corpus, not just new documents, and rebuilding the index. This is expensive and is why the choice of embedding model has real switching cost, so it is worth getting reasonably right early. The second is versioning. In a serious system you want to know which version of a document produced a given answer, both to debug and to handle the case where a user is asking about the state of things at a particular time. Treating ingestion as a living pipeline, with incremental updates, honest deletes, and a plan for re-embedding, is what separates a RAG demo that impressed everyone once from a RAG product that stays correct in the months after launch.

The patterns worth keeping

Strip away the specifics and RAG teaches a handful of lessons that transfer to systems that have nothing to do with language models.

how-rag-works diagram
Four transferable lessons: retrieval quality caps everything downstream; retrieve cheap then rerank precise; measure retrieval and generation separately; ground and cite to control hallucination.

The first is that retrieval quality caps everything downstream. The model, the prompt, the instructions, none of them can rise above the quality of the passages retrieved, and no amount of polish on the visible final stage compensates for a weak stage upstream. This is a general truth about pipelines: the earliest stage that loses information sets a ceiling that later stages can only fail to reach, never exceed. Find that stage and fix it there.

The second is the two-stage pattern, cheap and approximate to cut the field, then expensive and precise on the shortlist. Retrieve-then-rerank is one instance, but the shape appears anywhere you cannot afford to run your best method over everything: filter broadly with something fast, then judge narrowly with something accurate. The third is to measure the halves of a compound system separately, because a single end-to-end score hides which part failed and sends you tuning the wrong thing. The fourth is that grounding and citation are how you make a generative system trustworthy: constrain the output to verifiable evidence, show the evidence, and let the answer be checked. RAG is a retrieval problem wearing a language model's clothes, and once you see it that way, the model stops being the mystery and the retrieval becomes the craft.

Where to go from here

If this was useful, the thing worth keeping is the reframe. When a RAG system disappoints, the instinct is to reach for a bigger model or a cleverer prompt, and the answer is almost always upstream, in how you chunked, what you embedded, whether you retrieved the right passages at all, and whether you ever measured it. Fix retrieval and the model looks brilliant. Ignore retrieval and no model will save you.

I teach RAG and the rest of system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.

Read the free lessons: https://systemdesign.academy

Top comments (0)