Gursharan Makol

Posted on Apr 4 Edited on Oct 18 Originally published at aiinpracticehub.com

RAG in Practice — Part 3: How RAG Works — The Complete Pipeline

#ai #architecture #rag #webdev

This article is Part 3 of my RAG in Practice series, where I explain retrieval-augmented generation in practical, production-oriented terms.

In this part, we walk through the complete RAG pipeline step by step — from ingestion to retrieval to generation — and the tradeoffs that matter in real systems.

Two Jobs at Two Different Times

A RAG system does its work at two different times. Long before anyone asks a question, it prepares your documents so they can be searched. Then, each time someone asks, it finds the right pieces and writes an answer from them.

Part 2 showed the RAG pattern as six components in a line. This article opens up each one and follows real TechNova content through it.

Why not paste the documents into the prompt? For one short document, you can. RAG exists because a company has hundreds of documents that change every week, and the answer to any one question sits in a paragraph or two of them.

	Shift 1, ingestion	Shift 2, answering
When it runs	Before any question, and again whenever a document changes	Every time someone asks a question
What it does	Parse, chunk, enrich (optional), embed, store	Route (optional), embed the question, search, rerank (optional), keep the top results, assemble the prompt, generate
What it produces	An index of searchable chunks	An answer

The two shifts meet at the index, the searchable form of what Part 2 called the knowledge base. Shift 1 builds it, and Shift 2 searches it.

Why the split matters. A wrong answer can start in either shift. If a chunk was damaged during ingestion, nothing at answer time can repair it. If the chunks are fine, the problem is in how they were found or used. Knowing which shift to look in comes first in every debugging session.

TechNova's support assistant answers from five documents. They are the returns policy, the warranty terms, the troubleshooting guide, the firmware changelog, and the product specs page. TechNova sells two wireless headphones, the WH-1000 and the cheaper WH-500. We'll follow these documents through both shifts.

Shift 1: Preparing the Knowledge

Shift 1 turns documents into chunks that can be searched by meaning. A customer might ask "Can I send my headphones back?" while the policy says "return window". Keyword search can miss that match. Search by meaning is built to find it.

Parse: Get Clean Text and Keep the Structure

Parsing turns each document into clean text, and it has to keep the structure along with the words. Real documents are messier than they look.

PDFs with two columns get read straight across, so sentences from both columns are mixed together.
Headers, footers and page numbers repeat on every page and land in the middle of paragraphs.
Help-center pages carry navigation menus, cookie banners and footers along with the article.
Scanned pages are images. They hold no text at all until OCR reads them.

Tables are where structure matters most. Here is the comparison table from TechNova's product specs page.

Model	Driver size	Battery	Codecs
WH-1000	40mm	30 hours	SBC, AAC, LDAC
WH-500	30mm	8 hours	SBC, AAC

A naive parser that just strips the HTML tags runs every cell together.

ModelDriver sizeBatteryCodecsWH-100040mm30 hoursSBC, AAC, LDACWH-50030mm8 hoursSBC, AAC

Every word survived, but the meaning didn't. "WH-100040mm" isn't a product name anyone will search for, and nothing ties "8 hours" to the WH-500 or to battery life. Ask "What is the battery life of the WH-500?" and the retriever has nothing clean to match. If this text is retrieved anyway, the model has to guess which number belongs to which product.

A structure-aware parser keeps each row with its column names. One simple way is to write each row as its own line.

WH-1000. Driver size 40mm. Battery 30 hours. Codecs SBC, AAC, LDAC.
WH-500. Driver size 30mm. Battery 8 hours. Codecs SBC, AAC.

Now the WH-500 question has a line that answers it directly.

For large tables, some teams embed a short summary for search and store the full table beside it, so a match on the summary hands the model the whole table. Part 5 works through a product specs document with a comparison table in code.

What goes wrong. Structure lost during parsing is lost for good. Every later step inherits the damage, and no change to chunking or embedding can bring it back.

Chunk: Split Documents into Pieces You Can Retrieve

Chunking splits each document into pieces small enough to retrieve one at a time, and large enough to carry a complete thought. Search returns chunks, not documents, so a chunk is what the model will actually read.

Where the split falls matters. Here is the pairing section of TechNova's troubleshooting guide, split by a chunker that cuts at a fixed length.

Pairing the WH-1000 with a phone or laptop
1. Charge the headphones for at least 10 minutes.
2. With the headphones off, hold the power button for 5 seconds,
   until the light flashes blue.
3. Open Bluetooth settings on your device and select WH-1000.
---------------------- chunk boundary ----------------------
4. Not listed? Reset the headphones. Hold the power and
   noise-cancellation buttons together for 10 seconds, then
   repeat step 2.
5. When you hear "Connected", the light turns solid blue.

A customer asks, "My WH-1000 doesn't show up in Bluetooth. What do I do?" Neither chunk works well on its own. The first has the setup steps but not the reset. The second has the reset, but "repeat step 2" points to an instruction it no longer contains. The answer was split across the boundary.

The fix is to split at sections and keep numbered steps together, instead of cutting at a fixed length.

Chunk size is a tradeoff.

Too small, and a chunk loses its context. "Hold for 10 seconds" means nothing without which buttons, and why.
Too large, and one chunk covers several topics. Its embedding represents all that content together, which can make it less specific to any one question. Large chunks also take more room in the prompt at answer time.
Too long for the embedding model is a hard limit. Every embedding model has a maximum input length. Longer text is cut off or rejected, depending on the model, so the end of an oversized chunk may never be represented in its vector.

Overlap helps at the boundaries. Repeating the last sentence or two of one chunk at the start of the next means a short thought that crosses a boundary can appear whole in at least one chunk. It won't keep a longer procedure together. Part 4 compares chunking strategies in detail.

What goes wrong. An answer split across two chunks, so neither chunk is enough on its own.

Enrich (Optional): Tell Each Chunk Where It Came From

Contextual enrichment means adding a short piece of context to the chunk's text before you embed it. This is different from metadata. Enrichment becomes part of what gets embedded and searched, while metadata is stored as separate fields beside the chunk. A chunk cut out of a document loses everything around it, including the heading that said what it was about.

Here is a chunk from the TechNova Outlet terms, the page for refurbished and open-box items.

Purchases can be sent back for a refund within 7 days of delivery.
After that, the sale is final.

On its own, this reads like TechNova's general return rule. A question about returning a new WH-1000 can retrieve it next to the real rule, which gives new products 15 days. The model now sees two return windows with nothing to tell them apart.

Here is the same chunk, enriched.

From the TechNova Outlet terms, for refurbished and open-box items.
Purchases can be sent back for a refund within 7 days of delivery.
After that, the sale is final.

Now its vector, and the model reading it, both carry the fact that it's about Outlet items.

The simplest version needs no model. Put the document title and section heading at the top of every chunk. Some teams go further and use an LLM to write one sentence of context for each chunk. Neither is required on day one, but enrichment is one of the first things to try when similar chunks get confused.

	Enrichment	Metadata
Where it lives	In the chunk's text	In fields stored beside the chunk
Is it embedded?	Yes	No
What it's for	Helping search, and the model, tell similar chunks apart	Filtering by product, version or status, and tracing an answer to its source

A metadata filter can remove a chunk however well it matches the question. If the query filters for product = WH-1000, every relevant chunk without that label disappears before ranking even begins.

What goes wrong. Chunks that look alike to the retriever, so it returns the wrong one with full confidence.

Embed and Store: Turn Each Chunk into a Searchable Record

An embedding model turns each chunk into a vector, a list of numbers that places text with similar meaning close together. "Can I send my headphones back?" and "You can return a new TechNova product within 15 days" share almost no words, but their vectors land near each other. That closeness is what Shift 2 searches for.

Each chunk is stored as one record, usually in a vector database, an index built to find the vectors closest to a query quickly. Here is one record.

{
  "chunk_id": "returns-policy#standard",
  "text": "Returns Policy: Return window for new products\n\nYou can return...",
  "vector": [0.021, -0.113, 0.087, 0.042, ...],
  "metadata": {
    "source_url": "/help/returns-policy",
    "section": "Return window for new products",
    "doc_type": "policy",
    "version": "2026-03",
    "effective_date": "2026-03-01"
  }
}

chunk_id names the document and the section, so a trace can show exactly which chunk was used.
text is what the model will read. This one starts with its document title and section heading, the simple enrichment from the last section.
vector is what search compares. The embedding model determines how many numbers the vector contains.
metadata is stored beside the chunk and is not embedded unless you deliberately add that information to the text. It's used to filter results and to trace an answer back to its source. Part 8 covers the full set of fields a production chunk carries.

Choosing the model. Embedding models can struggle with product codes, model numbers and internal jargon. Terms like WH-1000 may be rare or missing in the data the model learned from, so semantically related chunks can rank unexpectedly. That's one reason keyword and hybrid search exist (Part 4). Before you pick a model, test two or three on your own questions and check which one puts the right chunk at the top. Part 7 shows how to build that question set. If you switch models later, every chunk has to be embedded again, because vectors from two different models can't be compared.

What goes wrong. A model that captures general meaning but misses your terms. Retrieval returns chunks that sound right but are about the wrong product, and nothing in the logs looks broken.

Shift 2: Answering the Question

Shift 2 runs live for every question. We'll follow one question through the path.

"How long do I have to return my WH-1000?"

The step names match the sections of a trace in Part 7, so you'll recognize them there.

Route (optional). When a system has more than one place to look, a rule or model decides where the question goes. This one is about policy, so it goes to document search. A simple RAG system with one search path can skip routing.
Embed the question. The question goes through the same embedding model that embedded the chunks. Same model, same vector space, so the question's vector can be compared with every chunk's.
Search. The vector database returns the chunks closest to the question, often 10 to 50 of them. These are the candidates by similarity.
Rerank (optional). A reranker reads the question together with each candidate and puts them in a better order. It's slower than vector search, so it runs only on the candidates.
Keep the top results. Only the top few, often 3 to 5, go forward. These kept chunks are what Part 7 means by "retrieved". For our question, they include wh1000#returns and returns-policy#standard, which both give 15 days from delivery.
Assemble the prompt. The kept chunks are put together with everything else the model needs, within a token budget.
Generate. The model reads the prompt and answers "You can return the WH-1000 within 15 days of delivery." For TechNova's current policy, the prompt is the source it should rely on.

Two of these steps deserve a closer look.

Search Matches Meaning

Search compares vectors, so a chunk can match a question it shares no words with.

The same strength is a weakness. Search finds what is close in meaning, and close isn't always correct. The Outlet's 7-day return rule is very close in meaning to the standard 15-day return rule. That's why enrichment, metadata filters and reranking exist.

Assembly Decides What the Model Sees

The model never sees your index. It sees one prompt, built from four parts.

Instructions   Answer using only the context below. If the context
               doesn't answer the question, say so.
History        Earlier turns of this conversation, if any.
Context        [wh1000#returns] You can return the headphones within
               15 days of delivery for a full refund...
               [returns-policy#standard] You can return a new TechNova
               product within 15 days of delivery...
Question       How long do I have to return my WH-1000?

Every prompt has a token budget. The model has a hard limit, and teams usually set a smaller one to control cost and speed. When the kept chunks and the conversation history don't all fit, something gets cut, and in many systems it's the lowest-ranked chunk. In a long chat, the history alone can crowd out the chunk that held the answer.

That cut is easy to miss. The chunk was retrieved, so the retrieval logs look fine, but the model never saw it. Log the final prompt, not only the retrieved chunks.

The Structured Data Path

Some questions are answered by data, not documents. "How many WH-1000 units were returned last quarter?" No chunk holds that number. It lives in a database, so the router sends the question there instead.

On this path, the model writes a SQL query from the question (text-to-SQL), the system runs it, and the result goes into the prompt in place of chunks. In production, the model should see only the tables it needs, every query should be checked before it runs, and access should be read-only.

Both paths meet at assembly. The model doesn't know or care which path produced its context.

Query Rewriting (Optional)

Query rewriting turns the customer's words into a better search. Customers write the way they talk. "My headphones won't connect" may rank the pairing steps lower than a question that names the product. A rewriting step uses an LLM to turn the question into one or more search queries, such as "WH-1000 Bluetooth pairing" and "reset WH-1000 when it doesn't appear in Bluetooth". The system searches with each and merges the results.

Like reranking, rewriting is something teams add when retrieval falls short, not a day-one requirement.

Where This Pipeline Breaks

At answer time, a failure usually shows up in one of three diagnostic layers, and to the customer they can look the same. Each one can produce a fluent, confident, wrong answer.

Layer	What failed	TechNova example
Retrieval	The right chunk wasn't kept	"My WH-1000 doesn't show up in Bluetooth." The reset step sits in the second pairing chunk, which ranks below the top results. The model answers from steps 1 to 3.
Assembly	The right chunk was kept, but cut from the prompt	"How long is the WH-1000 warranty?" Two long product chunks rank above the warranty chunk and use up the budget. The warranty chunk is cut, and the model says it can't find the warranty length.
Generation	The right chunk was in the prompt, and the model misread or ignored it	"How long does the WH-500 battery last?" The prompt includes the specs for both headphones. The model answers "30 hours", which is the WH-1000's battery life.

Check them in the order they happen. Look at the chunks that were kept, then the final prompt, then the answer, and stop at the first one that's wrong.

The layer where a failure shows up isn't always where the fix goes. The retrieval failure in the first row started in Shift 1, at a bad chunk boundary. Reranking can't fix that. Rechunking can.

Part 7 turns this into a diagnostic method and shows how to measure each layer.

This article covers the core pipeline. Production concerns such as access control, sensitive data and keeping the index current come later in the series.

Three Takeaways

A RAG system works at two different times. Shift 1 prepares documents before any question arrives. Shift 2 answers live. A wrong answer can start in either one.
Damage done during ingestion can't be fixed at answer time. A flattened table or a bad chunk boundary stays broken through every later step. Check parsing and chunking on your real documents before you tune anything else.
When an answer is wrong, check the chunks, then the prompt, then the answer. Stop at the first one that's wrong. That tells you which layer failed and where to start investigating.

Next, Part 4 looks at the decisions inside this pipeline that decide whether RAG works, from chunking strategy to reranking and hybrid search.

Part of the RAG in Practice series, originally published on AI in Practice Hub alongside the full series and companion code.

DEV Community