RAG in Practice — Part 5: Build a RAG System in Practice

This article is Part 5 of my RAG in Practice series, where I explain retrieval-augmented generation in practical, production-oriented terms.

In this part, we build a baseline RAG system on real TechNova documents and show what retrieval evidence looks like when documents get messy.

TechNova is a fictional company used as a running example throughout this series.

What You'll Build

A first RAG system should show you what it retrieved, not only what it answered. Part 3 walked through the pipeline, and Part 4 explained the chunking and retrieval decisions. This article runs those decisions as code on five TechNova support files and reports what happened, including the parts that went wrong.

The code is a baseline. A baseline is the simplest version of a system that works end to end. Every later improvement is measured against it.

Each file has a different shape, and four of them get one customer question each.

Question	File	Shape	Size
Q1. "How long do I have to return my WH-1000?"	return-policy.md	Short policy document	231 words
Q2. "My WH-1000 doesn't show up in Bluetooth. What do I do?"	troubleshooting-guide.md	Procedures under headings	1,094 words
Q3. "What changed in the latest firmware update?"	firmware-changelog.md	Versioned changelog	343 words, three versions
Q4. "How long does the WH-500 battery last?"	product-specs.html	HTML page with two tables	258 words

The fifth file, warranty-terms.md (358 words), is indexed too. No question is about it, so it only competes with the others.

Each question shows a different way a pipeline can fail, in chunking, in parsing or in retrieval. The article takes them in order, from the simplest document to the hardest to parse.

Run It Free First

The code runs on your laptop with no API key. The repository README has the setup steps for Windows, macOS and Linux. The first run downloads a small embedding model, about 30 MB.

Prefer not to install anything? The Chunking and Parsing Explorer shows the same results in your browser. You can switch the chunker, the chunk size, the overlap and the HTML handling, and see what each setting does to the documents and to retrieval.

---

The Part 5 Baseline

The pipeline has the same steps as Part 3, and this article stops after retrieval on purpose. The failures examined below are already visible before a model writes a word, so the code prints what retrieval returned first. Generating an answer is available, but none of the recorded results use it.

The embedding model is Model2Vec potion-base-8M. It's small and free. If you run it yourself, the scores may differ slightly in the last decimal places, but the chunks and which ones hold the answer will be the same. It's not the most accurate model you could use, and the results below come from this model only.

Retrieval is a few lines of numpy. The chunks are embedded once, when the index is built. Each question is embedded when it's asked.

def build_index(records, texts, text_of):
    """Embed every chunk once. Rows are float32 and L2-normalized."""
    slices = [text_of(texts, record) for record in records]
    matrix = _l2_normalize(embed_texts(slices))
    return {"records": records, "matrix": matrix}


def search(index, question, top_k=3):
    """Rank by full-precision dot product. Ties break by file, then chunk number."""
    records = index["records"]
    matrix = index["matrix"]
    query = _l2_normalize(embed_texts([question]))[0]
    scores = matrix @ query
    order = sorted(
        range(len(records)),
        key=lambda row: (-float(scores[row]), records[row]["file"], records[row]["n"]),
    )
    ...

How the score works. Each chunk and the question become an embedding, a list of numbers, called a vector, that represents the text's meaning. The Model2Vec model used here produces 256 numbers per vector. The code scales every vector to the same length, so the comparison depends on the direction the vectors point rather than their original size. This is what L2-normalized means in the code above. The resulting score is the cosine similarity. A higher score means closer in meaning, and the three highest-scoring chunks come back. The scores only mean something compared with each other for the same question. In this article, chunks that don't answer the question often score between 0.4 and 0.7. There's no vector database. With fewer than 50 chunks, a matrix in memory is enough.

Decide What a Right Answer Looks Like First

Before running anything, write down what a full answer must contain. The repo calls this the answer key (evidence_rules.json). For each question, it lists the exact text a complete answer needs. For Q1, that's "15-day return window" and "date of delivery". For Q2, it's all five pairing steps.

After retrieval, the code checks each of the three returned chunks against the answer key and gives it a label.

• all means the chunk holds the whole answer.
• some means it holds part of it.
• none means it holds none of it.

So an all in the top 3 means the right chunk came back.

The answer key is only used for checking. Retrieval never sees it, so it can't change which chunks come back. Part 7 calls a set of questions with known answers an evaluation set. This is a very small one.

---

Two Chunkers on the Same Files

Fixed-size chunking cuts by count. Recursive chunking cuts where the document already divides itself. Both work in characters, and both default to 500.

The fixed-size chunker is short enough to read in one go.

def fixed_chunks(text, size=500, overlap=100):
    if overlap >= size:
        raise ValueError("overlap must be smaller than size")
    out, start = [], 0
    while start < len(text):
        end = min(start + size, len(text))
        out.append(Chunk(start, end, text[start:end]))
        if end == len(text):
            break
        start = end - overlap
    return out

It cuts every 500 characters, wherever that lands, even mid-word. Each new chunk starts 100 characters before the previous cut, so neighbors share about a sentence.

The recursive chunker is the one described in Part 4's "Three Ways to Split". For markdown files, it starts a new chunk at every heading, the lines that begin with #, so a chunk never crosses into the next section. If a section is longer than the size limit, it splits at the blank lines between paragraphs, then at single line breaks, and only cuts mid-line as a last resort. Then it packs neighboring pieces back together up to the limit. It has no overlap. The code is in chunkers.py.

At 500 characters, the fixed-size chunker makes 41 chunks across the five files, and the recursive chunker makes 45. Retrieval searches all of them for every question, not only the file the question is about.

Try it first. Before reading the four sections below, open the explorer, pick a question, and guess which chunk retrieval will return. Then check your guess in that question's section.

---

The Return Policy (Q1): Complete, but Not Retrieved

Short policy documents, like the return policy used for Q1, are usually the easy case. Each section is a few sentences, and one section usually answers a question on its own. What can go wrong is retrieval, not chunking.

Q1 asks "How long do I have to return my WH-1000?" The answer is the opening paragraph of return-policy.md.

# TechNova Return Policy

TechNova offers a 15-day return window on new products bought from
techNova.com or the TechNova app. The return period begins on the date
of delivery shown in Orders, not the date of purchase.

Chunking decides what each chunk contains. Retrieval decides which chunks come back. Getting the first right doesn't guarantee the second. The recursive chunker kept this paragraph whole, in one chunk. Retrieval returned three other chunks.

Rank	Chunk returned	Holds the answer?
1	Troubleshooting guide, "Battery and Charging Issues"	No
2	Return policy, "How to Start a Return"	No
3	Troubleshooting guide, "Pairing the WH-1000"	No

The scores were close, from 0.436 down to 0.423, and none of these three answers the question. The paragraph with the answer never mentions the WH-1000. The first and third chunks do.

Only fixed-size chunks of 1000 characters brought the answer into the top 3. At that size, the opening paragraph shares a chunk with the next three sections of the policy, and with 100 characters of overlap that chunk ranked first. Try both in the explorer.

For your own documents. Short documents rarely need clever chunking, so don't stop at checking that a chunk holds the answer. Check where that chunk ranked for the questions customers actually ask.

---

The Troubleshooting Guide (Q2): One Procedure, One Chunk?

The troubleshooting guide used for Q2 is a set of procedures under headings, and a procedure is a sequence. It has to reach the model whole and in order. If a cut falls inside it, retrieval can return part of the steps, or all of them out of order.

Q2 asks "My WH-1000 doesn't show up in Bluetooth. What do I do?" The answer is the pairing section of troubleshooting-guide.md, the same five steps as in Parts 3 and 4. Here they are, with the fixed-size cuts marked.

## Pairing the WH-1000 with a phone or laptop

1. Charge the headphones for at least 10 minutes.
2. With the headphones off, hold the power button for
------------ chunk 2 starts here (100 characters of overlap) ------------
 5 seconds, until the light flashes blue.
3. Open Bluetooth settings on your device and select WH-10
--------------------------- chunk 1 ends here ---------------------------
00.
4. Not listed? Reset the headphones. Hold the power and
   noise-cancellation buttons together for 10 seconds, then repeat step 2.
5. When you hear "Connected", the light turns solid blue.

The fixed-size chunker cut inside step 3, in the middle of "WH-1000". In Part 4's sketch the cut fell between step 3 and step 4. In the real file it lands wherever character 500 falls. Chunk 1 holds the guide's introduction, the heading, steps 1 and 2, and step 3 up to the cut. Chunk 2 starts in the middle of step 2 and contains steps 3 to 5, with no heading. The recursive chunker kept the whole section, heading and all five steps, in one chunk of 439 characters.

Rank	Recursive, 500	Fixed, 500 with 100 overlap
1	Pairing section, all five steps. All of the answer.	Starts in step 2 and contains steps 3 to 5, no heading. Part of the answer.
2	"Connection Drops", the factory reset note. No.	The multipoint note from "Connection Drops". No.
3	"Battery and Charging Issues". No.	Guide introduction, heading, steps 1 and 2, and most of step 3. Part of the answer.

With recursive chunking, the full procedure is the top result. With fixed-size chunking, both halves came back, but not in a form a customer can follow. Steps 3 to 5 come first, an unrelated note comes next, and steps 1 and 2 come last. A generator would have to notice the split and put the steps back in order.

This isn't the failure Part 4 warned about, where the second half is never retrieved. That happened here too, but only after parsing changed a different file, and it was the first half that went missing. The specs table section shows it. Size matters for recursive chunking as well. At 250 characters the pairing section doesn't fit, so the recursive chunker splits it into two chunks too.

See it in the explorer with fixed-size chunks, and then with recursive chunks.

For your own documents. Split at headings, so each procedure stays in one chunk with its title. Then check that your size limit is larger than your longest procedure, because recursive chunking cuts too when a section doesn't fit.

---

The Firmware Changelog (Q3): The Right Chunk, Ranked Third

The firmware changelog used for Q3 is a list of version entries that look alike. Every version talks about fixes, updates and improvements, so the right entry has to compete with text that sounds just like it.

Q3 asks "What changed in the latest firmware update?" The answer is the version 3.2 entry of firmware-changelog.md.

## Version 3.2 (January 2026)

Bug fixes and app improvements.

- Fixed noise cancellation briefly turning off when you switch between
  two connected devices.
- Fixed a rare issue where the right ear cup stopped responding to taps.
- Improved battery reporting in the TechNova app, which now shows the
  remaining charge in 5 percent steps.
- Updated the USB-C charging controller to better support third-party
  cables that meet the USB-C specification but were previously rejected.

The recursive chunker did what Part 4 recommends for a changelog, with one version entry per chunk. This whole entry is one chunk. The changelog's title, "TechNova WH-1000 Firmware Changelog", became a separate chunk of its own.

Rank	Chunk returned	Holds the answer?
1	Troubleshooting guide, "Firmware Update Problems", first half	No
2	Troubleshooting guide, "Firmware Update Problems", second half	No
3	Changelog, "Version 3.2 (January 2026)"	Yes, all of it

The two chunks above it explain what to do when an update fails. Both contain the words "firmware" and "update". The 3.2 entry doesn't contain the word "firmware" at all, because that word is only in the title, and the title is a different chunk. The 3.2 entry was still kept in the top 3, so a generator would see it, but at the bottom.

Nothing in this search knows which version is the latest, either. "Latest" is just another word in the question. One common approach is to store each entry's version and date as metadata, and filter or sort on them (Part 8 covers that metadata).

With fixed-size chunks, all three results come from the troubleshooting guide, and no changelog chunk reaches the top 3. See it in the explorer.

For your own documents. Keep one version per chunk, and don't expect similarity search to know which version is newest. If customers ask about the latest version, store the version and date with each chunk and use them.

---

The Product Specs Table (Q4): Parsing Puts the Labels Back

The product specs page used for Q4 keeps its facts in an HTML table. A value only answers a question if it stays next to its row and column labels. An HTML table spreads them out. The column names sit in a header row at the top, and each value sits in its own row further down.

Q4 asks "How long does the WH-500 battery last?" The answer is one cell of the table in product-specs.html. It needs three things together, the product name "WH-500", the row "Battery Life", and the value "8 hours". In the raw HTML they're far apart.

<th>Specification</th>
<th>WH-1000 Premium Headphones</th>
<th>WH-500 Sport Headphones</th>
...
<td>Battery Life</td>
<td>30 hours (ANC on)</td>
<td>8 hours</td>

With recursive chunks of 500 characters, the header row with "WH-500" lands in one chunk and the battery row in the next. No chunk of the raw specs page holds all three, at any of the nine chunker settings.

The parsed mode turns each table row into one labeled line before chunking. Here is the battery row, wrapped to fit the page.

Specification: Battery Life
| WH-1000 Premium Headphones: 30 hours (ANC on)
| WH-500 Sport Headphones: 8 hours

Now each value sits next to its labels. In all nine parsed settings, one specs chunk holds the full answer. The repo's parser uses Python's built-in HTML parser, so it needs nothing extra to install. In production you'd more likely use a library such as BeautifulSoup or lxml. Whichever you use, each value has to keep its labels before chunking.

The troubleshooting guide also says "The WH-500 has a rated battery life of approximately 8 hours", so Q4's answer exists in two files. Here is what retrieval returned, recursive at 500.

Rank	Raw HTML	Parsed
1	Troubleshooting guide, "Battery and Charging Issues", with the WH-500 sentence. All of the answer.	The same chunk. All of the answer.
2	Troubleshooting guide, "Pairing the WH-1000". No.	Specs page, the rows from Impedance to Bluetooth Version, including Battery Life. All of the answer.
3	Troubleshooting guide, charging steps. No.	Specs page, the Wired Mode, Foldable and Price rows. Names the WH-500 but has no battery value.

The top result is the same both times. Parsing didn't change the winner, but it turned the specs page from scattered fragments into a chunk that holds the answer, ranked second.

Parsing one file changed the results for another question. Every parsed row repeats "WH-1000 Premium Headphones". With parsed HTML, two specs chunks took the top two places for Q1, the return-window question from the first section. They pushed the return policy's "How to Start a Return" chunk out of the top 3, and neither specs chunk answers the question. Q2 moved as well. With fixed-size chunks of 500 and 100 overlap, a specs chunk took third place for the pairing question, and the chunk with steps 1 and 2 dropped out of the top 3. See it in the explorer. A fix aimed at one document can move results for questions about a different one. That's why you rerun the whole question set after any change. Compare raw and parsed in the explorer.

For your own documents. Turn each table row into a labeled line before chunking. Then rerun every question, not only the ones about tables.

---

What All 18 Explorer Settings Showed

The explorer has 18 settings, one for every combination of chunker, size, overlap and HTML handling. Fixed-size has three sizes and two overlaps, which is six settings. Recursive has three sizes, which is three. Each of those nine runs once with raw HTML and once with parsed HTML, which gives 18.

No single setting won everywhere. The best-performing settings here all used fixed-size chunks of 1000 characters, and that worked partly because these files are short. The table below shows four of the 18. Each cell gives the rank of the first chunk that holds the full answer, and which file it came from. Q4's answer can come from the specs page or from the troubleshooting guide.

Setting	Q1 return window	Q2 pairing	Q3 latest firmware	Q4 WH-500 battery
Fixed, 500 with 100 overlap	Not in top 3	Not in top 3	Not in top 3	1st, troubleshooting guide
Recursive, 500	Not in top 3	1st, troubleshooting guide	3rd, changelog	1st, troubleshooting guide
Recursive, 500, parsed HTML	Not in top 3	1st, troubleshooting guide	3rd, changelog	1st, troubleshooting guide. 2nd, specs page.
Fixed, 1000 with 100 overlap	1st, return policy	1st, troubleshooting guide	3rd, changelog	1st, troubleshooting guide

Only three of the 18 settings got a full answer into the top 3 for all four questions, and all three used fixed-size chunks of 1000 characters. That isn't a case for big fixed chunks. These files are short, so a 1000-character chunk holds several whole sections. The pairing procedure happens to sit at characters 250 to 689 of the guide, inside the first chunk. A fixed-size cut lands wherever the count says, so the same setting would split the procedure if it sat a few hundred characters further down. Recursive chunking follows the headings instead, so the procedure stays whole wherever it sits, as long as it fits the size limit.

Every result above came from the same checks, made in order.

1. Is the full answer in one chunk?
   If not, start with chunking or parsing.
   Q2 with fixed-size chunks and Q4's raw specs page fail here.

2. Did that chunk come back in the top 3?
   If not, it's retrieval.
   Q1 fails here.
   Q3 passes, but only at rank 3.

3. Did the chunk reach the prompt, and did the answer use it?
   Those are assembly and generation, the last two layers of Part 7's diagnostic spine.
   The recorded runs stop before those layers.

Stop at the first check that fails. Q1's answer was already in one complete chunk, so rechunking wasn't the fix. Q4's raw table was split, so no retriever could return the whole answer from the specs page.

What to try next follows Part 4's retrieval section. Hybrid search adds keyword matching, so a chunk can also compete on the exact words it shares with the question. A reranker can reorder the top results, which could help Q3, where the right chunk was found but ranked third. Part 3's contextual enrichment adds the document title to each chunk before embedding, so the 3.2 entry would carry the word "firmware". None of these is tested here. Each one is a single change to make, then rerun all four questions and compare.

---

Which Approach for Which Shape

Start from what must stay together in each kind of document. This table sums up the four sections.

Shape	What must stay together	How to chunk or parse it	What to check
Short policy	Each rule with its conditions	Keep each rule and its conditions together. Section-aware splitting is usually enough.	Where the answer's chunk ranks, not only that it exists
Procedure	The heading and every step	Recursive, with a size limit larger than the longest procedure	That one chunk holds every step, in order
Versioned changelog	Each version's entry	One version per chunk, with the version and date stored as metadata	That "latest" questions return the newest entry
HTML table	Each value with its row and column labels	Parse rows into labeled lines before chunking	Every other question too, after the parsing change

---

Generate an Answer (Needs an API Key or Ollama)

Generation is the last step, and it can only use what retrieval returned. To turn it on, you need an API key from one provider, or Ollama running on your machine. The recorded runs in this article have it switched off, so every result above is retrieval alone.

Copy .env.example to .env, set GENERATION_PROVIDER, and add the matching key.

GENERATION_PROVIDER	What you need
openai	OPENAI_API_KEY
gemini	GEMINI_API_KEY
xai	XAI_API_KEY
ollama	Ollama running locally, no key

All four go through one small adapter that uses the OpenAI Python SDK with a different base URL for each provider. The default model for each is in .env.example, and you can change it there. The prompt tells the model to answer only from the retrieved chunks and to say so when they aren't enough.

Retrieval output always prints first. If the key is missing or the provider can't be reached, the answer line says why and the run carries on, so you still see the evidence.

---

Three Takeaways

1. Print the evidence before the answer. The retrieved chunks and their scores tell you where a failure starts.
2. Chunking makes an answer findable, not found. Recursive chunking put every full answer in one chunk somewhere in the files, and retrieval still missed one and ranked another third.
3. Change one thing, then rerun every question. A fix for one document can move results for another.

Keep Going

Try it in the browser. The Chunking and Parsing Explorer shows all 18 settings on the same files and questions, with the chunk boundaries drawn on each document.

Next, Part 6 asks when RAG is the right tool at all, compared with fine-tuning and long context.

References & artifacts

• RAG in Practice examples repository, Part 5 code in part5/
• Model2Vec, and the potion-base-8M model used here

---

Part of the RAG in Practice series, originally published on AI in Practice Hub alongside the full series and companion code.