There is a pattern I keep seeing in AI systems, and it fails the same way every time. You retrieve some data, or you expose a tool, and then you add a security check afterward. Filter out the documents the user should not see. Deny the tool call if the caller lacks the scope. It looks safe. It reads like a sound design in a code review. And it leaks anyway.
The argument I want to make is narrow and I believe it holds: the security check belongs inside the query or the tool boundary itself, not in a step that runs after the data has already been fetched or the surface has already been exposed. Once the unauthorized thing exists in your process, you have already lost. Everything after that is hoping nobody logs it.
I built three projects that pushed me to this conclusion from three different angles.
The retrieve-then-filter trap
Start with RAG, because it is where I first felt this. The textbook pipeline is: embed the question, retrieve top-k chunks, filter out what the user is not allowed to see, generate the answer. That filter step is the problem.
By the time you filter, the unauthorized chunks are already in your process. They can land in a log line, a trace span, an error report, or a prompt you assembled one step too early. And there is a second, quieter failure: a top-k of 5 that filters down to 1 silently degrades the answer, with no signal that it happened.
So in vaultrag, I moved the access-control predicate into the same SQL query as the vector search and the keyword search. A chunk the user cannot see is never selected, never scored, never ranked, never logged. It cannot leak, because it was never fetched. Access is evaluated per query, not baked into the index, so revoking someone takes effect on the very next question rather than after a reindex. And the groups a user belongs to are read from the database, never from the request, because if a caller could assert its own group membership the whole ACL would be decorative.
The number that made me believe it
Slogans about security are cheap. I did not want "access control is enforced at retrieval" to be a sentence in a README. I wanted a number.
So vaultrag has an eval harness that runs a gold set of (user, question, what-they-should-and-should-not-see) tuples against a real Postgres corpus, and reports two metrics that only mean something together: leak rate and recall. The pairing is the entire point, because each is trivial to fake alone. Retrieve nothing and you score a perfect 0 percent leak rate. Retrieve everything and you score perfect recall.
On the working build, the 11-case gold set reports a leak rate of 0.0 percent at 100 percent recall. Then I delete the ACL predicate from the retrieval query and run the exact same eval:
leak rate: 0.0% -> 81.8%
mean recall: 100.0% -> 100.0%
Read the second line. Recall did not move. The broken build answers every question correctly and completely, while handing one user the CEO's private notes and another team's salary bands. A quality-only eval scores that build as perfect. That is exactly why the leak number is never reported on its own, and why CI fails on a strict flag rather than on a threshold. Every document in the test corpus contains the phrase "quarterly bonus payout policy", so a retriever without access control would happily serve private notes to anyone asking about bonuses. The only thing standing between the two is that predicate living inside the query. Delete it and 9 of 12 ACL tests fail immediately with leak assertions. That is the difference between a test suite and decoration.
The same principle, one layer down: MCP tools
RAG is one place unauthorized data enters a process. MCP tools are another, and the shape of the mistake is identical. An MCP server hands a language model real capabilities: run commands, read files, hit internal URLs, mutate a database. A single over-scoped tool, or a .env exposed as a resource, turns a helpful agent into a data-exfiltration path. The trap is thinking you will catch the dangerous call later, at the moment it happens. But if the tool is exposed at all, the model can reach it.
So I wrote mcp-audit, which connects to an MCP server (or lints its manifest offline), enumerates every tool, resource, and prompt, and runs 18 security rules over that surface. It flags arbitrary command execution tools, destructive actions with no confirmation or scoping, secrets exposed as resources, SSRF-prone URL arguments, unauthenticated HTTP transports, and unconstrained input schemas that let the model pass anything anywhere. It is deterministic, runs offline, and drops into CI with SARIF output so a broad, unscoped tool surface breaks the build before an agent ever gets near it. The check moves earlier: you catch the over-broad surface at review time, not at call time.
And the boundary done right: scoped tools from the start
Bridgekit is the constructive version of the same idea. It is a scoped MCP server that exposes a company's tools (Shopify, Triple Whale, Postgres) with per-client permission boundaries baked into the boundary itself. Each client key carries the exact list of tools it may use and whether it may write. And this is the part I care about: tools/list only advertises the tools the calling client is scoped for. The check is not "let them list everything, then block the call." The unscoped tool is never even shown. A write tool called with a read-only key is denied and written to an append-only audit log, so the day someone asks "did this ever happen", the answer is a query, not an archaeology dig.
That is the whole thesis in one line of behavior. The permission boundary is not a gate you pass through after picking up the data. It is the shape of what you are allowed to see at all.
The honest caveat
Putting the check inside the query is not free, and I would be lying if I said it was universally cleaner. It couples your authorization model to your retrieval layer. In vaultrag the ACL predicate lives inside the retrieval query, which means the query is now harder to reason about, the database is doing security work, and you cannot swap retrieval engines without re-implementing the predicate. That is a real cost. My claim is not that it is cheaper. It is that the after-the-fact filter, which is genuinely cheaper and cleaner to write, is the one that leaks, and I would rather pay the coupling tax than ship the version that scores perfect on a quality eval while handing out salary bands.
If you want to see any of this in code, including the eval that turns "we enforce access control" into a number that fails CI, all three projects are at github.com/royalpinto007.
Top comments (0)