Joshua Nwachinemere

Posted on Jul 25 • Originally published at Medium

The Hidden Complexity of RAG

#ai #rag #programming #machinelearning

TL;DR

A production RAG system is not one model call wrapped around vector search. It has two pipelines—knowledge preparation and runtime retrieval—plus cross-cutting evaluation, authorization, observability, freshness, and cost controls.

If the final answer is wrong, inspect the stages separately. The root cause may be a stale source, damaged parsing, a bad chunk boundary, missed retrieval, an incorrect filter, weak reranking, noisy context, or generator overreach.

A basic retrieval-augmented generation demo can fit on a whiteboard:

Load → chunk → embed → retrieve → generate

It is a useful abstraction. It is also where many bad production decisions begin.

Imagine an employee asks:

Can contractors expense international travel in 2026?

The application retrieves a policy paragraph and the model returns a confident answer with a citation. The response looks good. It may still be wrong because:

the indexed policy expired last year;
the PDF parser separated a table from its heading;
the chunk omitted the sentence that applies specifically to contractors;
dense search missed the exact policy code;
a permissions filter exposed an internal exception;
the retrieved passages contradicted one another;
the answer added a claim that no source supported.

All of those failures can produce the same visible symptom: a fluent wrong answer.

That is the hidden complexity of RAG. The model is only the final stage in a much larger system. Production RAG is a knowledge supply chain, a search engine, a context-construction layer, an evaluation program, and a security boundary.

The difficult part is not making the pipeline run. It is knowing whether the right evidence survived every step.

RAG is better understood as two pipelines—knowledge preparation and runtime retrieval—surrounded by evaluation, security, and operational controls.

RAG starts before retrieval

Before a user asks a question, an offline pipeline has already made decisions that constrain every possible answer.

It decided which sources to trust. It parsed files. It removed or preserved structure. It split content into chunks, attached metadata, generated embeddings, and wrote records into an index. It also needs to update or delete those records when the source changes.

This is not clerical preprocessing. It is part of the model’s effective knowledge system.

Parsing is an information-loss problem

A parser can extract every visible word from a document and still destroy its meaning.

Consider a policy PDF with:

section headings that define scope;
footnotes that list exceptions;
a table whose column headers appear on another page;
scanned signatures or handwritten dates;
repeated headers and navigation text;
diagrams with labels that do not appear in the text layer.

A plain-text extractor may flatten that structure into a plausible-looking stream of words. The pipeline succeeds technically, but the relationship between those words is gone.

The fix is not always a better prompt. It may require layout-aware extraction, OCR, table handling, media-specific parsing, or a review queue for documents that fail quality checks.

The practical rule is simple:

If the ingestion pipeline loses the evidence, the retriever cannot recover it later.

Chunking is a retrieval policy

Chunk size is often treated like a configuration value to copy from a tutorial. In practice, it determines what the system can retrieve as one unit.

Chunks that are too small can separate a claim from its qualifier, title, date, or exception. Chunks that are too large can mix several topics, weaken retrieval precision, and waste the context budget.

There is no universally correct number of tokens. The right strategy depends on document structure, query patterns, retrieval method, and what counts as sufficient evidence.

Useful approaches include:

preserving headings and section paths;
keeping tables with their titles and column labels;
storing parent-child relationships;
adding controlled overlap where ideas cross boundaries;
attaching source, version, date, author, and access metadata;
using structure-aware or semantic splitting when fixed windows lose meaning.

Microsoft’s current RAG design guidance treats chunking as an experimental phase that should be evaluated against representative documents and queries, not chosen by folklore.

Chunk boundaries determine which facts, qualifiers, metadata, and table labels can be retrieved together.

Freshness requires a deletion story

Adding documents is easy. Synchronizing reality is harder.

A production knowledge pipeline needs to answer:

How quickly should a changed source appear in search?
How are renamed and duplicated documents detected?
What happens when a page is deleted?
Can an old embedding survive after its source has been revoked?
Which source wins when two versions conflict?
Can an answer show the version and effective date it used?

Without stable document identifiers, versioning, change detection, and deletion propagation, a RAG system can become a polished interface to stale information.

Temporal retrieval is harder than simply preferring the newest document. A question may refer to when an event happened, when a source was published, or when a rule was valid. The 2024 TempRALM study is useful evidence that semantic relevance and temporal relevance need separate treatment—but its reported gains come from specific temporal benchmarks, not a universal production recipe.

Retrieval is a ranking pipeline, not a database lookup

After ingestion, the online system has a different job: translate a user’s language into a ranked set of useful evidence.

That involves more than nearest-neighbor search.

Dense retrieval is useful, not magical

Embedding search is good at matching related meanings even when a query and a document use different words. It can still miss exact identifiers, product names, acronyms, dates, and rare terms.

If the user asks about policy TRV-2026-07, lexical search may recognize the exact string more reliably than a semantic vector alone. If the user says “the overseas contractor expense rule,” dense retrieval may be more helpful.

This is why hybrid retrieval remains important. Current Azure AI Search documentation describes running full-text and vector queries in parallel and merging their results with Reciprocal Rank Fusion. The larger lesson is vendor-independent: lexical and semantic signals fail differently, so combining them can be stronger than assuming one replaces the other.

Filters are part of relevance—and security

Metadata filters narrow the candidate set by attributes such as region, product, language, document type, effective date, or tenant.

They can improve quality, but they can also remove the correct answer when metadata is missing or wrong. The timing of a filter—before or after vector search—can also affect recall and latency.

Current Azure vector-filter guidance makes that tradeoff explicit: filter mode, selectivity, index structure, and candidate count can change both recall and performance. This is implementation-specific guidance, but the general lesson travels well—filters belong in retrieval evaluation, not only in application logic.

Permission filters are more serious. They must ensure that content the user cannot access does not enter the retrieved candidate set. Telling the model “do not reveal confidential information” is not access control.

If an unauthorized chunk reaches the prompt, the security failure has already happened.

Reranking is a separate model decision

A first-stage retriever is optimized to find plausible candidates quickly. A reranker spends more computation comparing the query with a smaller candidate set and reorders those results.

That can improve relevance, but it introduces new choices:

How many candidates enter the reranker?
Which fields does it see?
How much latency and cost does it add?
Does it work for the domain and language?
Does improved ranking actually improve final answers?

Current semantic-ranking documentation makes those constraints concrete: reranking operates over an initial ranked set and has input limits. “Add a reranker” is therefore not a free quality upgrade. It is another stage to benchmark.

Query rewriting can help—or quietly change the question

Real questions are messy. They contain typos, pronouns, missing context, and several requests in one sentence.

A query layer may expand synonyms, rewrite the question, decompose it into subqueries, or use conversation history. This can improve coverage for multi-part and multi-hop questions. It can also drift from the user’s intent, retrieve too broadly, or multiply latency and cost.

Current agentic retrieval guidance describes decomposition, parallel subqueries, reranking, and source tracking. It also states that agentic retrieval adds latency, and some capabilities remain preview features.

Agentic RAG is not simply “advanced RAG.” It is a trade: more adaptive retrieval in exchange for more planning, nondeterminism, cost, failure modes, and observability work.

Good retrieval narrows a broad, authorized candidate set into a small evidence set while measuring the tradeoffs at every transition.

Context construction decides what the model is allowed to see

Retrieval produces candidates. Those candidates should not automatically become the prompt.

A context-construction layer may need to:

remove duplicates and near-duplicates;
keep source diversity;
attach neighboring or parent sections;
preserve titles, dates, and provenance;
detect conflicting versions;
fit evidence within a token budget;
order passages intentionally;
create stable citation anchors;
exclude low-confidence or unauthorized material.

This matters because more context is not always better.

The “Lost in the Middle” study found that language models can use relevant information less reliably depending on where it appears in a long input. Separately, an EMNLP 2024 comparison of RAG and long-context models found a real quality-cost tradeoff rather than a universal winner.

A larger context window does not remove the need to select, organize, and evaluate evidence. It merely raises the amount of information the model can potentially process—and the amount of noise you can send it.

Generation cannot repair missing evidence

Once the context reaches the language model, prompting still matters. The model should know what task it is performing, what sources it may use, what output structure is required, and when it must abstain.

But prompting cannot make absent evidence appear.

Useful generation controls include:

require claims to be supported by supplied evidence;
return stable citation identifiers;
distinguish quoted facts from inference;
state when sources conflict;
abstain when evidence is insufficient;
validate required fields and allowed values;
verify that cited passages actually support the answer;
route consequential outputs to human review.

None of these is a perfect “hallucination detector.” Groundedness checks and LLM judges can also make mistakes. For high-impact tasks, deterministic rules, source verification, and human review still matter.

The same fluent wrong answer can begin in the source, parser, chunker, index, retriever, context builder, or generator.

Evaluation must tell you where the failure happened

A single end-to-end score is not enough.

Suppose the correct answer is absent from the response. Several explanations are possible:

The source never contained it.
The parser dropped it.
The chunker separated it from necessary context.
The retriever failed to find it.
A filter removed it.
The reranker pushed it down.
Context assembly excluded it.
The model ignored or misread it.
The evaluator marked a correct answer as wrong.

If evaluation only inspects the final text, the team may tune the prompt for a retrieval failure or replace the embedding model for a parsing failure.

Recent work reflects this need for diagnosis. The 2024 RAGChecker paper separates retrieval and generation diagnostics. MIRAGE, published in Findings of NAACL 2025, evaluates dimensions including noise vulnerability, context acceptability, context insensitivity, and context misinterpretation.

A useful evaluation program works at several levels.

Ingestion

Were all expected files processed?
Were headings, tables, dates, and relationships preserved?
Is metadata accurate?
Did updates and deletions propagate?

Retrieval

Does the candidate set contain the evidence?
How often does the correct passage appear in the top k?
Are exact terms and semantic paraphrases both handled?
Do filters preserve relevant authorized results?
Does reranking improve measured relevance?

Metrics such as Recall@k, Precision@k, mean reciprocal rank, and nDCG can help, depending on the task and labels. None is sufficient by itself.

Context and generation

Does the assembled context cover the required evidence?
Is it redundant, noisy, or contradictory?
Are answer claims supported by the context?
Are citations correct?
Does the system abstain when the corpus cannot answer?

System behavior

Task success with real users
Latency percentiles rather than averages alone
Cost per successful answer
Freshness and deletion service levels
Permission failures and security events
Failure rates by document type, language, tenant, and query class

The test set matters as much as the metric. It should include answerable, unanswerable, ambiguous, exact-identifier, multi-hop, conflicting-source, freshness-sensitive, permission-sensitive, and adversarial questions.

End-to-end scores show whether the system failed; stage-level contracts, versions, and traces help explain where and why.

LLM-based evaluators are useful for scaling review, but they are models—not ground truth. Calibrate them against human labels, monitor disagreement, and keep deterministic checks where possible.

Security is not a filter you add at the end

RAG connects a model to data that may be private, user-generated, or externally controlled. That makes retrieval a security boundary.

Two risks are especially easy to underestimate.

Indirect prompt injection

A retrieved document can contain instructions aimed at the model rather than information for the user. Those instructions may be hidden in webpages, files, tickets, emails, or poisoned knowledge-base content.

OWASP’s current prompt-injection guidance explicitly covers indirect injection from external sources. Retrieved text should therefore be treated as untrusted data, not trusted system instructions.

Embedding and index weaknesses

An attacker—or a simple ingestion bug—can introduce misleading records, manipulate similarity, leak sensitive relationships, or break tenant isolation. OWASP’s vector and embedding guidance discusses poisoning, unauthorized access, and data leakage in RAG-related systems.

The risk is not merely theoretical. PoisonedRAG, published at USENIX Security 2025, demonstrates targeted knowledge-base poisoning under specific experimental threat models. Its attack rates should not be generalized to every corpus or retriever, but the study establishes that retrieved knowledge can be an attack surface—not just a quality problem.

Practical controls include:

source allowlists and provenance;
retrieval-time authorization;
tenant isolation;
sanitization and content classification;
audit logs for sources and citations;
permission-aware caches;
index update and deletion controls;
adversarial evaluation;
output validation and human approval for consequential actions.

Then comes the operational tax

A production request may involve authentication, policy checks, query rewriting, several searches, embedding calls, reranking, context assembly, generation, citation validation, and logging.

Each stage adds latency and a new way to fail.

The system needs more than an overall timer. Trace each stage so a slow answer can be attributed to the search service, a model call, a retry, an oversized candidate set, or an external source.

The same applies to cost. Track the contribution from:

parsing and OCR;
embedding new or changed content;
storage and indexing;
query embeddings;
multiple retrieval calls;
reranking;
generation tokens;
evaluation and monitoring.

Caching can help, but a cache key that ignores tenant, permissions, source version, or model configuration can return stale or unauthorized information.

Version the components that affect behavior: parser, chunker, embedding model, index schema, retrieval settings, reranker, prompt, generator, and evaluation dataset. Otherwise, a quality regression becomes an archaeological investigation.

A diagnostic matrix for wrong answers

Symptom	First place to inspect	Useful evidence
Correct passage never appears	Parsing, chunking, retrieval	Parse samples, chunk boundaries, Recall@k
Exact code/name is missed	Lexical and hybrid retrieval	BM25 baseline, hybrid result set
Relevant passage ranks too low	Candidate depth and reranker	Before/after rankings, nDCG or MRR
Wrong tenant or restricted text appears	Authorization and filters	ACL evaluation, retrieval trace, cache key
Answer ignores strong evidence	Context order and generation	Final context, citation support, model trace
Old answer persists after update	Sync and deletion pipeline	Source version, index version, deletion log
Quality drops after a release	Component versioning	Dataset run by parser/index/prompt/model version

A practical build order

The safest way to build RAG is not to add every advanced technique. It is to add complexity only when evidence identifies a failure it can solve.

Define the task. Specify authoritative sources, users, permissions, freshness requirements, and what a successful answer means.
Build the evaluation set early. Use representative documents and questions, including questions the corpus cannot answer.
Make ingestion observable. Inspect parsing, chunk boundaries, metadata, versions, and deletions before tuning retrieval.
Establish retrieval baselines. Test lexical and dense retrieval, then measure whether hybrid retrieval improves the actual query set.
Add reranking only after measurement. Record quality, latency, and cost changes.
Engineer context deliberately. Handle duplicates, source conflicts, ordering, citations, and abstention.
Evaluate components and the whole system. A good retriever can still feed a model that overreaches; a grounded generator cannot use evidence it never receives.
Test security and failure behavior. Include permission boundaries, poisoned content, timeouts, partial failures, and stale data.
Introduce agentic retrieval selectively. Use it for demonstrated query classes that need decomposition, iteration, or dynamic source selection.
Re-run evaluation after every material change. A new parser, embedding model, index, prompt, or generator changes the system.

The real mental model

RAG is often presented as a way to give an LLM access to your data. That description hides the engineering responsibility.

A better definition is:

RAG is a controlled process for turning changing, permissioned sources into evidence that a generative model can use—and for measuring whether the resulting answer remains supported.

The hard part is not the connection between the model and the database. It is maintaining a trustworthy path from source to evidence to answer, while being able to explain exactly where that path failed.

DEV Community