Why Your RAG Pipeline is Lying to You

Tushar Sharma — Wed, 29 Jul 2026 23:53:07 +0000

RAG (retrieval-augmented generation) is how most AI tools search your company's documents to answer questions. Instead of guessing, the model pulls relevant text from your files and uses it to respond

Sounds reliable. Usually isn't.

Last month, our team shipped a RAG-powered search feature to a Fortune 500 client. Three days later, this Slack message showed up :

"The AI pulled a revenue number from our Q3 report. But when I checked the PDF, that number was from a footnote about projected revenue, not actual revenue. How did this happen?"

We checked the retrieval logs. The system worked exactly as designed. Found the most semantically similar chunk. Returned it with a high confidence score. Even cited the source document.

And it was wrong.

The chunk containing "Revenue: $142M" got retrieved, but the chunk was cut mid-paragraph. The preceding sentence ("Based on current growth projections, we estimate...") lived in a different chunk entirely. The system had no way of knowing this was a projection.

We had an architecture problem. The model did its job. We fed it a fragment without context, and it did exactly what we asked.

This is what production RAG actually looks like. The pipeline retrieves information that's technically correct but contextually wrong. It happens more than anyone wants to admit.

The retrieval failures that keep showing up

After that incident, we audited 50 of our most problematic queries. Same patterns, over and over.

The headless table

User query: "What's our discount policy for enterprise customers?"

Retrieved chunk: "| Enterprise | 25% | Annual contract |"

The column headers were in a previous chunk. The user got a table row but couldn't tell which column was the discount and which was the contract requirement.

Picture a sales rep about to jump on a call. They ask the AI for pricing tiers. It returns a row of numbers. They quote the wrong one to the customer. That's a citation without context.

The orphaned clause

User query: "Can we terminate the vendor contract early?"

Retrieved chunk: "Either party may terminate this Agreement with 30 days written notice."

"Agreement" was defined earlier in the document as the Master Services Agreement, which has a 12-month minimum term. The definitions section was in a completely different chunk. Technically accurate. Operationally misleading.

Legal teams rely on AI to surface contract terms. When the AI returns a clause stripped of its governing definitions, people make decisions on incomplete information. And they don't realize it until something goes wrong.

Where the intelligence actually dies

The root cause is in the architecture. Look at the standard RAG pipeline:

Document → Fixed-size chunks → Embeddings → Vector DB → Retrieval → LLM

Step 2 is where it falls apart. You take a richly structured document (headings, tables, sections, page numbers, definitions, cross-references) and flatten it into uniform text blobs.

You destroy the document's organizational intelligence before the AI ever sees it. It's like ripping chapters out of a book, shuffling the pages, and asking someone to answer questions about the plot.

Section headings tell you what topic you're reading about. Table headers tell you what each column means. Page numbers tell you where you are. Definitions tell you what terms mean. Cross-references point you to related information.

Flat chunking throws all of that away.

Watch a real document get destroyed

Here's a page from an internal operations document:

SECTION 4: SERVICE LEVEL AGREEMENTS

4.1 Response Time Requirements

The following response times apply to all Priority 1 incidents:

Note: Resolution targets assume standard business hours (9 AM – 6 PM EST). For after-hours coverage, see Section 7.2: On-Call Procedures.

Now here's what a standard text splitter does to it:

Chunk 1: "SECTION 4: SERVICE LEVEL AGREEMENTS 4.1 Response Time Requirements The following response times apply to all Priority 1 incidents:"

Chunk 4: "hours (9 AM – 6 PM EST). For after-hours coverage, see Section 7.2: On-Call Procedures."

The section heading got separated from its content. The table was split across three chunks. The header row is in a different chunk than half the data rows. The "Note" about business hours is torn in two. The cross-reference to Section 7.2 is orphaned from everything it relates to.

If a user asks "What's the resolution time for Critical incidents?", the system might retrieve Chunk 2 or 3. But can it tell the user that resolution times assume business hours? That after-hours procedures are different? No.

The chunking paradox

"Just use bigger chunks!" is the first thing everyone tries. Bigger chunks keep more context, right?

Not quite.

Bigger chunks mean blurrier embeddings. A 4,000-token chunk might contain 15 different topics. Embed it as a single vector and you get a blurry average. Queries that should match precisely now match vaguely.

Bigger chunks also waste tokens. You retrieve 4,000 tokens when you needed 200. Your context window fills up with noise. API costs go up. Responses slow down.

Small chunks give you precision but lose context. Big chunks keep context but lose precision. Neither lets the system decide based on what the query actually needs.

Sometimes you need one table row plus its headers. Sometimes the full section. Sometimes the entire document. Flat chunking forces you to pick one size for everything.

The detective analogy

You're a detective investigating a complex fraud case. The evidence room hands you a bag of puzzle pieces from 50 different puzzles, all mixed together. Box covers thrown away.

You can match some pieces that look similar. You might even assemble fragments that seem to form part of a picture. But you'll never know which puzzle you're solving, or if you're missing the piece that cracks the case.

That's your AI agent reasoning over flat chunks.

Now picture the alternative: all 50 puzzles in labeled boxes, a photo of each completed puzzle, an index showing which pieces have edge patterns and which have sky. You'd find the right puzzle in seconds, zoom to the relevant section, and pull exactly the pieces you need.

That's hierarchical document architecture.

The question worth asking

When your RAG pipeline retrieves a chunk, can it tell you which section of the document it came from? What the nearest heading was? What page it was on? If it's a table row, can it tell you the column headers? If it's a list item, what the parent list was about?

If the answer is "no" to most of these, you're flying blind. Your citations are technically correct and operationally useless.

Building the hierarchy

At PipesHub , we spent months figuring out why RAG breaks in production. The answer was better document architecture.

We built a 4-level hierarchy:

Record: the complete document
Block Group: a logical section (a headed section, a full table, a code listing)
Block: an atomic unit (a paragraph, a table row, a code block)
Chunk: an embedding-optimized segment

Every level carries metadata. Every level is accessible to the AI agent. When chunks aren't enough, the agent can ask for more context, based on what it actually needs.

In the next article, we'll get into how this hierarchy works, and the tools that let agents decide how much context they need for any given query. Stay tuned.

DEV Community: Tushar Sharma