Permission-Aware RAG: What It Really Costs to Get Right

Permission-aware RAG means your retrieval step filters by who's asking, not just what they asked — and that filter has to live inside the vector query itself, not as a second pass the LLM does after the fact. Most teams find this out the hard way: their RAG demo works great on a shared knowledge base, then someone asks it to enforce per-user document access and the whole retrieval layer needs to change.

Why "just add a permission check" doesn't work

The instinct is to keep your existing retrieval pipeline and bolt a check onto the end: retrieve the top-k chunks, then ask an LLM (or a rules engine) "should this user see this?" before passing context to the answer step. This fails in two ways:

• It's non-deterministic. An LLM asked to police access control will occasionally get it wrong, and "occasionally wrong" is not an acceptable access-control policy, especially with health records, legal documents, or HR files in the mix.
• The leak already happened. By the time you're filtering post-retrieval, the restricted content has already been pulled into a context window and sent somewhere. Logging, caching, or a stray debug trace can leak it before your filter ever runs.

The fix is to push permissions into the query, the same way you'd push a WHERE user_id = ? clause into a SQL query instead of fetching everything and filtering in application code. We've seen the same principle play out in our own tooling: in an internal outreach engine we built, a single call that filters and extracts in one pass consistently beat a multi-step chain that fetched broadly and reasoned about relevance afterward — cheaper, faster, and fewer places for something to go wrong. Permission-aware RAG is the security-critical version of that same lesson: filter at the source, don't filter after the fact.

What actually changes in your architecture

If you're running Postgres with pgvector or a managed vector store, permission-aware retrieval typically requires:

1. Permission metadata on every chunk. Not just a document_id — you need to know which roles, teams, or user IDs can see each chunk at index time. This usually means re-chunking and re-embedding your existing corpus, not a metadata patch.
2. Row-level security or an equivalent filter in the query path. Postgres has native support for this via row-level security policies, and Supabase builds directly on top of it for multi-tenant RLS. The similarity search and the access filter run as one query, so a user's vector search literally cannot return rows they're not permitted to see.
3. A permissions model that changes over time. Documents get shared, roles get revoked, teams get restructured. If your embeddings and your permissions live in different systems that sync on a schedule, there's a window where a fired employee's session can still retrieve documents they lost access to an hour ago. Real-time permission checks are non-negotiable once this goes near anything regulated.
4. Re-indexing strategy, not just retrieval logic. This is the part that blows up estimates. Adding ACLs to a RAG system that already has 50,000 indexed documents means you're not writing a new query — you're re-processing the entire corpus with permission tags attached, then validating that the tags are correct before you trust the system with sensitive content.

Where the cost actually goes

Teams budget for the model and the retrieval logic. On that same 50,000-document corpus, the real cost shows up in:

• Data audit. Someone has to determine who's actually supposed to see what, and at 50,000 documents that's not a spreadsheet filled out in an afternoon — it's confirming access department by department, often against tribal knowledge that was never written down, before a single permission tag gets attached.
• Re-embedding at scale. Re-chunking and re-embedding 50,000 documents isn't a job you queue up overnight and check on tomorrow — if the permission model changes what counts as one retrievable unit (splitting a shared folder into per-team sections, for instance), the chunk boundaries themselves change, so this is a re-processing pass over the whole corpus, not a metadata update layered on top of it.
• Testing for negative cases. Testing that the system retrieves correctly is easy. Testing that it never retrieves something a user shouldn't see, across every role combination, is the part that takes real QA time — this is the same category of work covered in an AI agent security checklist, and it deserves the same rigor you'd apply to authentication code, not "we tried a few prompts and it seemed fine."
• Audit logging. In healthcare or government contexts, you often need to prove, after the fact, exactly what a user retrieved and why — which means logging the filtered query, not just the final answer. This overlaps directly with what we've written about court-ready architecture for healthcare AI: the log has to be defensible, not just present.

When it's worth building vs. buying

If your access model is simple — one tenant, one shared knowledge base, everyone sees everything — don't build permission-aware retrieval at all. It's unnecessary complexity for a problem you don't have. Start with the basics in RAG explained for founders and move on.

Build it yourself when:

• Your permission model is core to the product (multi-tenant SaaS, healthcare records, internal knowledge bases with confidential HR or legal content).
• You need real-time revocation — access changes have to take effect immediately, not on the next sync.
• You're in a regulated space where "we filtered it after retrieval" won't survive an audit.

Consider a managed layer (many enterprise search and RAG platforms now ship ACL-aware retrieval out of the box) when your permission model maps cleanly onto standard role/group structures and you don't need to customize the filtering logic itself. Whichever you choose, put the same scrutiny into the vendor's retrieval guarantees that you'd put into any AI vendor contract — see evals in AI vendor contracts for what to actually ask for.

The one-sentence version

Don't ask the LLM to enforce access control after retrieval — enforce it in the query that does the retrieving, and budget for re-indexing your corpus, not just writing new retrieval code.

If you're scoping a RAG system that needs real permission boundaries — healthcare records, multi-tenant SaaS, internal docs with confidential sections — let's talk.

---

Originally published on the Pykero blog.