Most "our RAG isn't working" problems aren't model problems. The model is fine. The retrieval is feeding it garbage — stale documents, duplicated chunks, broken tables, context that's technically relevant and practically useless. In other words: a data pipeline problem, wearing an AI costume.
Once you see RAG as a pipeline, the fixes stop being vibes ("try a bigger model," "tweak the prompt") and start being the boring, reliable discipline data engineers have practiced for twenty years.
TL;DR — Retrieval-augmented generation is ingest → transform → index → serve, with an LLM on the end. The failure modes are classic data-engineering failures: bad chunking (schema), stale indexes (freshness), duplicates (dedup), no lineage (debuggability), and no evals (data quality). Fix the pipeline and the "AI" gets dramatically better without touching the model.
RAG is ETL with an LLM stapled to the end
Strip away the branding and a RAG system is four stages you already know:
- Ingest — pull documents from wherever they live (Confluence, S3, a Postgres table, a Slack export).
- Transform — clean, chunk, and enrich into retrievable units.
- Index — embed and load into a vector store (or a hybrid keyword + vector index).
- Serve — at query time, retrieve the top-k units and hand them to the model.
That's a pipeline. Every hard-won lesson about pipelines applies: idempotency, schema contracts, backfills, monitoring, and lineage. The LLM is just the last transform.
Chunking is schema design
The single biggest lever in RAG quality is how you split documents — and teams treat it like an afterthought (split every 500 tokens). But a chunk is a record, and how you define that record is schema design.
Split a pricing table down the middle and neither half is retrievable. Chunk a legal contract by fixed size and you sever clauses from their definitions. Good chunking respects the document's real structure: sections, headings, table boundaries, function definitions. Attach metadata to every chunk — source, section path, last-modified, permissions — exactly like you'd add columns to a well-designed table. Retrieval quality is mostly a function of chunk design, not embedding model.
Freshness is a first-class SLA, not a cron afterthought
A vector index is a cache of your knowledge, and every cache goes stale. The doc changed this morning; your embeddings are from last month; the model confidently answers with last month's policy. To a user that's not "stale data" — it's the AI lying.
Borrow the data-engineering answer: change data capture. Track source updates, re-embed only what changed (incremental, not full re-index), and put a freshness SLA on it — "no chunk older than its source by more than an hour." Fully re-embedding a corpus nightly is the RAG equivalent of TRUNCATE + reload every table every night: expensive, slow, and a sign you skipped incremental design.
Deduplication, or death by near-identical chunks
Real corpora are full of the same content five times: the doc, the wiki copy, the email that pasted the doc, the slide that summarized the email. Without dedup, your top-k retrieval returns the same paragraph four times and one genuinely useful chunk — and you've spent 80% of the context window saying the same thing.
Dedup at the pipeline level (content hashing, near-duplicate detection on embeddings) is the same problem as dedup in any ingest job. Solve it once, upstream, instead of praying the model ignores the repetition.
No lineage, no debugging
When a RAG answer is wrong, the first question is "what did it retrieve?" — and most teams can't answer it, because they logged the response but not the retrieved chunks, their scores, or their sources. That's shipping a pipeline with no lineage and being surprised you can't debug it.
Log the retrieval set for every query: which chunks, which scores, which source documents, which version. Now a wrong answer is a traceable data incident ("it retrieved the deprecated v1 doc because we never expired it") instead of a mystery you blame on the model.
Evals are data quality tests
You wouldn't ship a pipeline with zero tests on the output. RAG evals are exactly that — data quality checks on retrieval and generation. Build a small golden set of question → expected-source pairs and assert that retrieval surfaces the right documents. Track retrieval precision/recall over time so a bad ingest run shows up as a regression in a dashboard, not a spike in angry users a week later.
This is the part everyone skips because it's unglamorous, and it's the part that separates a demo from a system.
The reframe that fixes most of it
Next time retrieval underperforms, don't reach for a bigger model. Ask the pipeline questions:
- Are my chunks coherent records, or arbitrary token slices?
- Is my index fresh, with an SLA and incremental updates?
- Have I deduped the corpus?
- Can I see the lineage of any answer?
- Do I have evals that catch a bad ingest before users do?
The teams winning with RAG aren't the ones with the fanciest model. They're the ones who realized the "AI system" was a data system all along — and staffed it, monitored it, and tested it like one.
Your model is probably fine. Go fix your pipeline.
Top comments (0)