The notebook worked. Retrieval found the right passages, the model wrote a good answer, and the demo landed. Then the same design met real users and four things happened: first tokens took ten seconds or more, a document the user was never allowed to see turned up in the prompt, a wrong answer could not be traced to retrieval or to generation, and the monthly bill stopped being predictable.
Most of those are not primarily model-selection problems. They are architecture problems, and they are the reason this article treats retrieval-augmented generation as a system with eight layers rather than a pattern with two steps. What follows is a blueprint: the layers, the contract each one owes the next, where authorization has to live, how a first-token budget is worked backwards from a target, and where it is worth buying a layer instead of building it. Where JarvisBitz has verified production evidence, I say so; everywhere else the guidance is engineering reasoning you should test against your own corpus.
Where RAG demos break in production
Most production failures fall into five groups, and each one maps to a layer that the demo skipped.
Stale index. Documents change; the index does not. Users get last quarter's policy with a confident citation. The demo had no ingestion pipeline, only a one-time load.
Leaked documents. The vector store returns the nearest chunks regardless of who is asking. A filter applied after retrieval helps until the top results are all restricted and the user sees an empty answer, or until someone forgets the filter on one code path. The demo had one user.
Unattributable errors. An answer is wrong. Was the right chunk never indexed, never retrieved, retrieved but ranked out, or retrieved and then contradicted by the model? Without per-layer logs there is no way to know, so the fix is guesswork. The demo had no observability.
Latency that compounds. Authorization lookup, embedding, approximate nearest neighbour search, reranking and model time to first token each add their own share of latency, and the demo measured none of them.
Unpredictable spend. Long prompts, no caching, and a retrieval step that returns fifty chunks because fifty was the default. The demo ran twenty queries.
The reference architecture: eight layers and their contracts
user request
|
[1] Ingest raw documents -> parsed, cleaned, versioned text + metadata
|
[2] Index chunks -> embeddings + lexical index + ACL metadata per chunk
|
[3] Query request -> intent, rewritten query (or no retrieval at all)
understanding
|
[4] Authorized query + principal -> candidate chunks the user may see
retrieval
|
[5] Ranking candidates -> ordered, deduplicated shortlist
|
[6] Context shortlist -> token-budgeted prompt with citation payloads
assembly
|
[7] Grounded prompt -> answer + citations, or an honest "not in the sources"
generation
|
[8] Evaluation every layer -> logs, metrics, offline evals, alerts
and observability
A layer contract is the shape of what a layer promises to hand to the next one. Writing them down is the single most useful thing a team can do before scaling, because it turns "the RAG is wrong" into "layer 4 returned no candidates for this principal".
| Layer | Owes the next layer | Must record |
|---|---|---|
| 1 Ingest | Parsed text with structure preserved, a document version, source metadata | Parse status per page, pages dropped, parser used |
| 2 Index | Chunks with stable IDs, embeddings, lexical terms, ACL and tenant metadata | Chunk count per document, embedding model version |
| 3 Query understanding | Intent label, rewritten query or "skip retrieval", confidence | Intent, confidence, rewrite applied |
| 4 Authorized retrieval | Candidates the principal is allowed to see, with scores | Filter applied, candidate count, latency |
| 5 Ranking | Ordered shortlist, duplicates removed | Ranker, top scores, items dropped |
| 6 Context assembly | Prompt within a token budget, one citation payload per passage | Tokens per source, passages included |
| 7 Grounded generation | Answer with citations, or a refusal | Model, cached tokens, time to first token, citations emitted |
| 8 Evaluation and observability | Attribution for any answer, drift alerts | Everything above, joined by request ID |
Query understanding and intent routing
The cheapest retrieval is the one you do not run. A production assistant receives greetings, follow-ups that only need the conversation so far, requests for actions, and questions that the corpus cannot answer at all. Sending every one of those through embedding, search and reranking wastes latency and, worse, gives the model irrelevant passages to be confidently wrong with.
Intents that skip retrieval
Define a small set of intents up front and classify each request before anything else runs: informational (retrieve), transactional (route to a tool or workflow), conversational (answer from the dialogue), and out of scope (decline and log). Classification is a short model call or a lightweight classifier over a few example phrasings per intent; it costs far less than the retrieval it prevents. The intent also decides the rest of the path: which corpus, which filters, and whether a person needs to be involved.
When to rewrite
Rewriting the query helps when users write the way people talk and the corpus is written the way documents are written. "Why does my export keep failing since the update" becomes "export failure after version upgrade" and retrieval improves. Rewriting hurts when the query already contains exact identifiers (error codes, part numbers) that a rewrite paraphrases away. A workable rule: rewrite conversational queries, pass identifier-heavy queries through unchanged, and log both forms so you can see which one retrieval actually used. The managed retrieval tools now expose this directly; OpenAI's vector store search accepts rewrite_query=true and returns the rewritten form in the result, which is convenient for exactly this kind of logging.
Authorized retrieval
This is the layer that separates a demo from a system you can put in front of customers, and the rule is short: authorization is a retrieval filter, never only a prompt instruction. A sentence in the system prompt saying "only use documents the user may access" is a request to a model that has already been shown the documents. By then the leak has happened.
Filter before rank
The filter belongs inside the search query, applied by the store before similarity scoring, so that restricted chunks are never candidates. Filtering after retrieval has two failure modes: the result set can be exhausted by restricted items (the user gets nothing, or the system quietly widens the search), and every new code path is a place to forget the filter. The common retrieval stacks discussed in this article all support authorization-aware filtering, though the exact mechanism differs by product: pgvector through ordinary SQL predicates and Postgres row-level security, Qdrant and Weaviate through payload and tenant filters, Elasticsearch through document-level security, and the managed tools through attribute or metadata filters (OpenAI vector stores, Gemini File Search). Metadata filtering in Google's RAG Engine was documented as Preview on 18 September 2026; treat Preview features as things to test, not to assume.
ACL metadata on every chunk
For the filter to work, each chunk must carry what the filter needs. A minimal record:
{
"chunk_id": "doc_8812:p14:c03",
"doc_id": "doc_8812",
"doc_version": "2026-09-04T10:12:00Z",
"tenant_id": "t_acme",
"acl": {
"allow_groups": ["g_service_advisors", "g_admins"],
"allow_users": [],
"classification": "internal"
},
"source": { "uri": "s3://kb/manuals/2026/brake-service.pdf", "page": 14, "section": "3.2 Torque specifications" },
"text": "...",
"embedding_model": "text-embedding-x@2026-06",
"lexical_terms": ["torque", "caliper", "bracket"]
}
And the retrieval call that uses it, in framework-free pseudocode:
def retrieve(query: str, principal: Principal, k: int = 40) -> list[Chunk]:
# 1. cache: same principal scope + same normalised query => reuse
cache_key = hash(principal.tenant_id, sorted(principal.groups), normalise(query))
if (hit := retrieval_cache.get(cache_key)) is not None:
return hit
# 2. authorization filter is part of the query, not a post-step
acl_filter = {
"tenant_id": principal.tenant_id,
"any_of": {"acl.allow_groups": principal.groups, "acl.allow_users": [principal.user_id]},
"doc_version": {"current": True},
}
q_vec = embed(query)
dense = store.search(vector=q_vec, filter=acl_filter, limit=k)
sparse = store.lexical_search(terms=tokenize(query), filter=acl_filter, limit=k)
# 3. fuse, then hand to the ranking layer; nothing unauthorised can reach it
candidates = reciprocal_rank_fusion(dense, sparse)
retrieval_cache.set(cache_key, candidates, ttl=seconds(300))
return candidates
Two details in that snippet matter more than they look. The cache key includes the principal's authorization scope, so a cached result can never be served to someone with narrower access. And the filter includes the document version, so a superseded chunk cannot outrank its replacement.
Group expansion (turning a user into the set of groups they belong to) happens before retrieval, is cached briefly, and is invalidated when memberships change. Test this layer adversarially: seed a canary document that only one group may see and assert, in CI, that every other principal gets zero results for a query that matches it exactly.
Ranking and context assembly
Retrieval returns candidates; ranking decides what the model sees. Hybrid retrieval (dense plus lexical, fused with reciprocal rank fusion) fixes the case where a query contains an exact term the embedding blurs. A cross-encoder reranker over the fused top forty or so lifts precision further. Reranking adds another latency stage, and the cost varies materially with provider, candidate count and model. Measure it on the actual request path rather than assuming a fixed budget. Whether reranking pays off is an evaluation question, not a belief: measure recall at k with and without it on your own query set.
Token budget per source
Context assembly gets a fixed budget and spends it deliberately. A practical scheme allocates a maximum share per source document so that one long manual cannot crowd out three short ones that each contain part of the answer, and orders passages deliberately. Where the highest-ranked evidence should sit is model-dependent, so test passage ordering on the chosen model with your own evaluation set rather than assuming a universal rule.
Deduplication
Overlapping chunks from the same page, near-identical paragraphs across document versions, and the same table captured by two parsers all waste budget and mislead the model into treating repetition as corroboration. Deduplicate on normalised text and on (doc_id, page) before assembly.
Citation payloads
Every passage in the prompt carries a citation payload: chunk ID, document title, page or section, version. The model is instructed to cite by ID, and the application resolves IDs to links after generation. This keeps citations verifiable (the ID either exists in the prompt or it does not) and gives layer 8 the join key it needs to attribute a bad answer to a specific passage.
The first-token latency budget
Users judge a chat interface by when the first token appears. Work the budget backwards from a target and give each layer a share. The figures below are illustrative allocations for a two-second first-token target, not measurements; your numbers come from your own traces.
| Step | Illustrative allocation | What removes it |
|---|---|---|
| Authorization and group expansion | 50 ms | Short-lived principal cache |
| Intent classification | 150 ms | Skip for follow-ups with a cached intent |
| Query embedding | 100 ms | Embedding cache on normalised query |
| ANN plus lexical search | 150 ms | Retrieval cache keyed by principal scope and query |
| Reranking | 150 ms | Rerank fewer candidates; skip when top scores are separated |
| Context assembly | 50 ms | Precomputed citation payloads |
| Model time to first token | 1,000 ms | Prompt caching of the stable prefix; smaller model for simple intents |
| Network and serialisation | 100 ms | Streaming from the first token |
Three different caches are doing three different jobs here, and conflating them is a common mistake.
Prefix caching is provided by the model API and removes the cost of re-reading the stable part of the prompt (system instructions, tool definitions, long standing context). It is enabled by default on OpenAI's platform with a minimum cacheable prefix of 1,024 tokens on GPT-5.6 and later, and cached reads are discounted by up to 90 percent; Anthropic offers automatic caching or explicit breakpoints with 5-minute or 1-hour lifetimes; standard cache hits are priced at 0.1x the base input price, with a 0.025x rate on Claude Fable 5.1 and Claude Mythos 5.1. Prefix caching only helps if the prompt is built so that the stable part comes first and the retrieved passages come last.
Retrieval caching stores the candidate set for a normalised query within an authorization scope. It removes embedding and search time on repeated questions, which in support-style traffic are a large share.
Semantic caching stores whole answers for questions judged equivalent. It is the biggest saver and the most dangerous: a wrong equivalence serves a wrong answer, and a cached answer can outlive the document it was built from. If you use it, key it by principal scope and document version, and expire it on ingestion events.
Build or buy, one layer at a time
The decision is rarely all or nothing. In 2026 the practical options per layer are a self-managed search stack, a managed retrieval tool attached to a model API, or, for some corpora, no retrieval at all.
| Layer | Self-managed | Managed | Notes |
|---|---|---|---|
| Ingest and parse | Docling, Unstructured, your own parsers | Document AI layout parser and LLM parser in Google's RAG Engine; automatic chunking in OpenAI vector stores and Gemini File Search | Managed parsing is fast to adopt and hard to inspect page by page |
| Index and search | pgvector for vector search combined with PostgreSQL full-text search and result fusion (RRF or reranking) in your own code; Qdrant, Weaviate and Elasticsearch with built-in hybrid search; all support pre-filtering | OpenAI vector stores (hybrid search weights, attribute filters, up to 50 results), Gemini File Search (chunking config, metadata filters, file citations), RAG Engine (RagManagedDb, Vector Search, Pinecone, Weaviate) | Managed stores tie you to one model provider's request path |
| Authorization | Your filter, your tests | Attribute or metadata filters you populate | Either way the ACL data is yours to maintain |
| Ranking | Cross-encoder rerankers, provider rerank APIs | Built-in rankers with limited tuning | Measure before paying for it |
| Generation | Any model | Any model | Cache the prefix regardless |
| Evaluation | Open-source eval frameworks plus your own labelled set | Provider tracing dashboards | The labelled query set is the asset; tools are interchangeable |
Long context as a layer
The frontier models now offer roughly one million tokens of input: Claude Opus 5, Sonnet 5 and Fable 5.1 list a 1M token window (Haiku 4.5 stays at 200K); OpenAI's gpt-6-astra has a 1,050,000 token window with 922,000 maximum input tokens, and prompts above 272K input tokens are billed at twice the input rate and 1.5 times the output rate for the whole request; Gemini 3.8 Flash accepts 1,048,576 input tokens, with promotional pricing that runs until 31 December 2026. Many 300-page text-heavy manuals can fit within these context windows, depending on extraction, images and tokenization. Whether they should is a different question.
Long context replaces layers 2 through 5 with "send everything", which is attractive for small, static, single-tenant corpora where every user may see every document. It is the wrong tool when the corpus is larger than the window (a 40,000 document library is not going in), when different users may see different documents (there is no filter to apply to a prompt), when the same content is read on every request (you pay the full input each time unless the cache holds), or when you need to attribute an answer to a passage. In practice the strongest 2026 designs use both: retrieval to select an authorized, versioned subset, and a large window to give the model generous, well-ordered context from that subset.
Failure modes and the detectors that catch them
| Failure | What the user sees | Detector |
|---|---|---|
| Retrieval miss | "I could not find that", but the document exists | Offline recall at k on a labelled query set; alert on recall drop after index rebuilds |
| Ranking miss | The right chunk was retrieved but not shown | Log positions; measure "gold chunk rank" distribution; compare with and without reranker |
| Faithfulness failure | Answer contradicts or exceeds the passages | Faithfulness eval on sampled traffic; citation ID validation (every cited ID must exist in the prompt) |
| Permission leak | A restricted passage appears in a citation | Canary documents; assert zero cross-scope results in CI; audit log per request |
| Index drift | Answers cite superseded versions | Version field on every chunk; alert when retrieved doc_version is not current |
| Cache poisoning | A stale or wrong answer keeps being served | Cache keys include scope and version; invalidate on ingestion events; sample cached answers into evals |
| Cost spike | The bill jumps with no traffic change | Track tokens per request by layer; alert on prompt length percentile and on cache hit rate drops |
The common thread: every detector depends on a log line from a specific layer, keyed by the request ID. If layer 8 is bolted on after launch, most of these detectors cannot be built.
Production checklist
- Every chunk carries tenant, ACL, version and source metadata; retrieval filters on all four inside the store query.
- Intent classification runs before retrieval; at least one intent skips retrieval entirely.
- Hybrid retrieval is fused, deduplicated and reranked; recall at k is measured on a labelled set before and after each change.
- Context assembly has a token budget per source and emits one citation payload per passage.
- The prompt is ordered stable-prefix-first so prefix caching applies; retrieval and semantic caches are keyed by principal scope and document version.
- The first-token budget is written down per layer and compared with traces weekly.
- Canary documents and cross-scope assertions run in CI.
- Every request produces per-layer logs joined by a request ID, and a bad answer can be attributed to a layer within minutes.
- Index rebuilds and re-embedding are events that invalidate caches and trigger the recall eval.
The lesson from a production system
JarvisBitz rebuilt the AI assistant inside a US automotive SaaS product whose knowledge base held more than 120 GB of content across roughly 40,000 PDF documents. Before the rebuild, answers could take approximately 10 to 30 seconds to begin, depending on the query, and intent accuracy was poor. The redesign was the architecture above: intent detection and routing before retrieval, authorization-aware document retrieval, caching, targeted retrieval and optimised context construction, built in Python on Google Vertex AI Platform (now Gemini Enterprise Agent Platform) with Gemini. After it, first-token latency in typical interactions was under three seconds, the system achieved approximately 98 percent accuracy across intent recognition and information retrieval in the client's evaluated use cases, and operating cost fell by roughly 30 percent. The full account is in the case study on our site.
What that project taught, and what this blueprint is really about, is that the difficult part of production RAG is rarely the model call. It is the surrounding system: retrieval, permissions, caching, context management, latency controls, verification and monitoring. These are the kinds of production AI engineering problems JarvisBitz Engineering focuses on, and they are the ones that decide whether a RAG system survives contact with real users.
The rest of this series covers each layer in turn. If your open problems are exact-term queries or tenant isolation, start with hybrid search and multi-tenant retrieval.
Drafted with AI assistance and reviewed, edited and approved by the author.
Top comments (0)