Why your RAG returns garbage (and it's not the model)

Your RAG bot just gave a confident, detailed answer. And it's completely wrong.

Here's the part that'll annoy you: the model did nothing wrong. It answered perfectly — using the text you handed it. The bug isn't in the AI. It's in the five steps before the AI.

Prefer to watch? Full 6-minute walkthrough with the "lost in the middle" animation:

The pipeline, in one line of code

RAG is simple on paper. You search your own documents, grab the best matches, paste them into the prompt, and the model answers from those. So when the answer's garbage, everyone blames the model.

Wrong suspect. There are five places this pipeline breaks — and four of them happen before the model even runs:

const chunks  = split(docs)                   // 1 · chunking
const vectors = embed(chunks)                 // 2 · embedding
const hits    = search(embed(query), vectors) // 3 · retrieval
const best    = rerank(query, hits)           // 4 · ranking
const answer  = llm(prompt(best, query))      // 5 · generation

Five lines. Every bug I'm about to show you lives on one of them.

1. Chunking — a fixed cut splits your answer in half

Your documents are long, so you cut them into pieces before you store them. Most people cut every few hundred characters and move on. That's where it starts to rot.

A fixed-length cut doesn't care about meaning. It'll slice a sentence in half. It'll split a question from its answer. Now the one chunk that held your answer is two halves — and each half looks irrelevant on its own. So retrieval never even finds it.

Fix: cut on structure, not length. Split on paragraphs and headers — where the meaning actually breaks. And let chunks overlap a little, so nothing gets stranded at the edge.

2. Embedding — your question doesn't look like its answer

Every chunk goes through a model that turns text into a vector: a long list of numbers that captures its meaning. Similar meaning, similar numbers. That's the whole trick that makes search work.

But here's the trap. Your question gets embedded by the same model — and a question rarely looks like its answer.

You ask "how do I reset my password?" The doc says "account recovery procedure." Same thing to a human. Different words — and a weak embedder puts them far apart.

Fix: pick an embedding model built for retrieval, and test it on your data. A model that's great on legal text can be useless on code.

3. Retrieval — vector search fumbles exact strings

You take the question's vector and grab the closest chunks. This is the "vector search" everyone talks about. On its own, it's got a blind spot.

Vector search matches meaning, not exact words. Which is great — until someone searches for an error code. A product name. SKU-4417. Those barely have a meaning to embed. They're just exact strings, and nothing is "close in meaning" to a serial number.

Fix: hybrid search. Run keyword search and vector search, then merge the results. Old-school keyword matching catches the exact strings embeddings miss.

4. Ranking — the best chunk was #8, and you dropped it

Now you've got a pile of candidates — say the top twenty. But you can't paste twenty chunks into the prompt, so you keep the top five and drop the rest.

Here's the question nobody asks: what if the best chunk was ranked number eight? Then you just threw it away. Your answer was in the pile, and you cut it.

Fix: a re-ranker — a second, smarter model that reads the question and each chunk together and re-scores them. Cheap vector search casts a wide net; the re-ranker picks the real winners out of it. Now your top five are actually the top five.

And order matters too — "lost in the middle"

Even the order you paste chunks in changes the answer. Models pay the most attention to the start and the end of the context. Put your most important chunk in the middle, and the model can skim right past it.

This isn't a hunch — it's measured. The same model gets more accurate just by moving the right chunk to the edges of the context (Liu et al., Lost in the Middle, 2023). So put your best chunk first, or last. Never buried in the middle.

5. The prompt — where people ruin it two ways

Right chunks, right order. This is your last chance to ruin it, and people do it two ways:

1. They dump in everything. Ten thousand tokens of maybe-relevant text, hoping the model sorts it out. More context isn't more accuracy — it's more noise to get lost in.
2. They never tell the model what to do when the answer isn't there. So it does the one thing you don't want: it makes something up.

Fix is basically one line in your prompt:

"Answer only from the context below. If it's not there, say you don't know."

That single instruction turns a confident liar into a system you can trust. A RAG bot that admits "I don't know" beats ten that guess.

Put the blame where it belongs

• Bad chunks
• Wrong embedder
• Vector-only search
• No re-ranker
• A sloppy prompt

Five failure points — and notice the model was the last one. Usually the innocent one.

So next time your RAG bot lies to you, don't touch the model first. Walk the five steps backward — prompt, ranking, retrieval, embedding, chunks. The bug is almost always upstream.

What's the worst answer your RAG bot has ever given you? Drop it in the comments — I actually want to read those.

---

I make Vlad's Stack https://www.youtube.com/channel/UCUO8Uo5LsEy1b9eRkrH6JNg — how the AI tools you use every day actually work, for people who write code. Full video walkthrough of this one is above.