Designing a RAG System That Knows What You’re Allowed to See

A design case study: choosing between fine-tuning, long context and RAG, and why access control belongs in retrieval, not in the prompt.

Part of the AI Architecture Casebook: real-world AI problems, redesigned one decision at a time. Start here

An employee in Bengaluru asks the company HR assistant how much parental leave she gets. The answer comes back confident, well written and wrong: it is the UK policy.

A week later, a junior engineer asks about relocation support and receives details of the executive relocation package, a document he was never meant to see.

Neither failure came from a bad model. Both came from the architecture around it. This post walks through how I approached the redesign as a design exercise: the options, the decisions, the trade-offs, and one problem that better search alone cannot solve.

I work on enterprise APIs, where one rule is non-negotiable: authorization is decided at the gateway, from a verified token, never by trusting what the caller says. When I started looking at how RAG systems handle access control, I kept seeing the opposite: permissions written into prompts and left to the model. This case study applies the gateway mindset to retrieval.

The company, people and documents in this case study are fictional. The architecture patterns are real.

The problem

Globex is a fictional company with 20,000 employees in 12 countries and about 4,000 HR policy documents. Leave, travel, benefits and expenses all differ by country, and some policies are restricted to senior grades. HR updates around 40 documents every week.

The first chatbot was built in two weeks. It found the documents that best matched a question, placed them in a long prompt, and asked the model to answer. Within a month, HR had logged three kinds of failure:

1. Wrong country. The UK and India parental leave policies match “parental leave” almost equally well. The bot picked the wrong one.
2. Leaked policy. A junior employee received text from a policy restricted to senior grades.
3. Stale and uncited answers. Answers had no source, and some quoted a travel policy that had been replaced two weeks earlier.

The bot was switched off. HR still answers about 6,000 policy questions a month by hand, and most of the answers are already written down somewhere.

The second failure is the serious one. A wrong answer is an inconvenience. A restricted document shown to the wrong person is a compliance incident.

What the fix had to satisfy

Before choosing any technology, I wrote down what a good answer must look like. Most design mistakes come from skipping this step.

The first line is the one that cannot bend. Every other decision is tuned around it.

Four options, and why I rejected three

All four options use a language model. The real question is where the model gets its policy knowledge: trained into the model, or looked up from the documents for each question.

Why not fine-tuning? A model is billions of numbers, called weights. Fine-tuning blends the policies into those numbers: no document remains inside the model. Think of a student who memorized the whole handbook, versus one who looks up the right page each time. The memorizer can’t tell you which page a fact came from, so answers can’t be cited. Every one of the 40 weekly policy changes means retraining. And because one model answers everyone, the executive policy can’t be hidden from a junior engineer; it’s mixed into the same numbers. Fine-tuning is good for how a model writes, not for what it knows when facts change and have access rules.

Why not long context? Every model has a context window: how much text it can read for one question. Modern windows hold hundreds of pages, so it’s tempting to skip search and paste a large pile of policies into every prompt. If fine-tuning is the student who memorized the handbook, this is handing the student the whole handbook for every single question. It fails three ways.

• Cost: you pay for every word read, and 100,000 tokens per question costs far more than the 3,000 that matter.
• Accuracy: models miss facts buried in long prompts, and with two near-identical parental leave policies in the pile, they pick the wrong one.
• Security: if the executive policy is in the pile, the model has already read it. And once you choose which documents to send based on who is asking, you have built a retrieval system anyway.

Long context shines when the material is small, shared by everyone and rarely changes, like one contract or one product manual.

Why not pure vector search? This is the subtle one. Vector search finds text with similar meaning, and the UK and India policies mean almost exactly the same thing. What separates them are exact words: country names, policy codes, grade labels. That is what keyword search is good at.

The architecture

The Answer API is the only component that sees both who is asking and what is being searched. It reads country and grade from the verified sign-in token, builds the filter itself, and sends only the five best allowed chunks to the model. On AWS this maps to API Gateway and Lambda, Bedrock Knowledge Bases over OpenSearch Serverless, and a Bedrock model with a grounding check.

Three decisions that carry the design

I recorded each decision as an Architecture Decision Record (ADR): the context, what we chose, what we rejected, and the signal that would make us change our mind.

1. Hybrid search with a reranker

Run keyword search (BM25) and vector search together, merge the results, rerank the top candidates and keep the best five. Keyword search catches “India” and “HR-PL-IN-001”; vector search catches “time off after having a baby”.

The AWS detail that almost undid this: Bedrock Knowledge Bases support hybrid search on several vector stores, including OpenSearch (Serverless and managed clusters), Aurora PostgreSQL and MongoDB Atlas. On other stores, such as the low-cost S3 Vectors, a hybrid request quietly falls back to semantic-only search, with no error. Choosing the cheapest store would have silently brought back the original bug. That is why the store choice is part of the decision, not an implementation detail.

Trade-off: OpenSearch Serverless has a minimum cost even when idle, and reranking adds latency. I accept both: a cheaper store would bring back the wrong-country bug, and the reranker still fits within the 3-second budget.

2. Access control in retrieval, not in the prompt

Every document carries the countries and grades allowed to see it. The API reads the user’s country and grade from their verified sign-in token, builds a filter, and passes it into every search. The model only ever receives chunks the user is entitled to.

The tempting alternative is to tell the model: “Do not reveal documents the user is not allowed to see.” That is a security control in name only. The model has already received the restricted text, and a cleverly worded question can talk it out of the instruction.

Two other alternatives I rejected:

• Filtering after retrieval, in code: restricted chunks still take up slots in the results, pushing out good ones, and one bug leaks data.
• A model-generated filter (some platforms can build a filter from the question text): useful for relevance, but it is controlled by the user’s words, so it must never carry security.

The rule: the security filter is built from verified identity only. Never from user input, never from model output.

3. Chunk by policy section

HR policies have numbered sections, and the rules often depend on each other. “Up to 26 weeks of leave” in section 3.1 means little without “after 12 months of service” in section 3.2. A fixed 500-token split can cut a rule away from its condition, so the model answers with half the policy.

So I split on section headings, and long sections on sub-headings, keeping the parent heading as a prefix.

The part that surprised me: better search is not enough

While building the test documents, I put the UK and India parental leave policies side by side. They are the same template with different numbers. Here is section 3.1 from each:

Notice what is missing: the country. Once a policy is split into sections, the chunk that holds the answer contains no country name and no policy code. Those live in the title and the first section. So neither vector search nor keyword search can tell these two chunks apart.

That leads to two separate fixes for two separate cases:

• The question names the country (“How much parental leave do employees in India get?”). Hybrid search can win here, but only if every chunk is prefixed with its document title, country and policy code. Section chunking without that prefix throws away exactly the words keyword search needs.
• The question doesn’t name the country (“How much parental leave do I get?”). Here the text contains nothing to search on. No retrieval method can pick the right version from the words alone. Only the identity filter can, because it knows who is asking.

This is the real argument for access control in retrieval. It is not only a security feature. For a large share of everyday questions, the user’s identity is the only signal that separates the right answer from the wrong one.

What can still go wrong, and how I’d measure it

No design removes every risk. These are the ones I would watch:

Measure retrieval separately from answers. Most RAG failures are retrieval failures: if the right section isn’t in the top five, no prompt will fix it. So the evaluation set has 60 questions with a known correct document and section: 30 ordinary questions, 15 where the answer differs by country, 10 with no matching policy, and 5 probes asking about something the user may not see.

One rule for the leak count: if a result can’t be verified (say, it has no document id), count it as a leak. A broken system should make the security number worse, never better.

At 10x scale (200,000 employees), I would expect these to change first:

• Country and grade lists become too coarse; a dedicated authorization service takes over.
• Repeated questions (open-enrollment week) need caching.
• Indexes split by region for data residency.
• Personal questions (“how many leave days do I have left?”) need access to the HR system, which turns this into an agent.

Takeaways

1. Write the constraints before choosing the technology. The one that cannot bend (access control here) decides most of the design.
2. Fine-tuning stores style, not changing facts. For policies that change weekly and need citations, retrieval wins.
3. Security belongs in retrieval, not in the prompt. If the model has seen the text, a prompt instruction will not reliably keep it secret.
4. Small platform details can undo big decisions. A cheaper vector store would have silently turned hybrid search back into semantic search.
5. Identity is also a relevance signal. When a question doesn’t name the country, only knowing who asks can pick the right answer.

This is the first case study in a series where I take a realistic AI problem, compare the options, and record the decisions as ADRs. The design doc and ADRs are on GitHub: ai-architecture-casebook.

One simplification to note: real enterprises usually have far richer access rules, such as departments, projects and temporary access. Country and grade are enough to show the pattern. At larger scale, a dedicated authorization service makes the decision, but the principle stays the same: the filter comes from verified identity, never from the prompt.

How would you have handled the wrong-country problem? I’d like to hear other approaches in the comments.