Enforcing RAG access control inside the retrieval query, not after it

Most RAG pipelines have the same shape: embed the question, retrieve the top-k chunks, filter out what the user is not allowed to see, then generate an answer. I built vaultrag because that ordering has always bothered me. By the time you filter, the unauthorized chunks are already inside your process.

The problem with retrieve-then-filter

Think about where a retrieved chunk goes before you filter it. It can land in a log line, a trace span, an error report, or a prompt you assembled one step too early. If any of those happen before the filter runs, you have leaked, and you probably will not notice.

There is a quieter failure too. If your top-k is 5 and the filter removes 4 of them, you now answer from a single chunk with no signal that this happened. Access control silently degraded your answer quality, and nothing told you.

I wanted a design where an unauthorized chunk is never selected. Not fetched then dropped. Not ranked then trimmed. Never selected in the first place.

The core idea

vaultrag puts the ACL predicate in the same SQL query as the vector search and the keyword search. A chunk the user cannot see is never selected, never scored, never ranked, never logged. It cannot leak, because it was never fetched.

The retrieval query starts from a visible CTE, and both search arms read only from it. Here is the part that matters, straight from app/retrieval.py:

WITH visible AS (
    SELECT c.id, c.doc_id, c.heading, c.text, c.embedding, c.tsv,
           d.title, d.updated_at, d.is_official, d.owner, d.url
    FROM chunks c
    JOIN documents d ON d.id = c.doc_id
    WHERE d.deleted_at IS NULL
      AND EXISTS (
          SELECT 1 FROM doc_acl a
          WHERE a.doc_id = d.id
            AND a.principal = ANY(%(principals)s)
      )
)

The vector arm and the keyword arm both SELECT ... FROM visible. There is no code path in the function that can reach a chunk outside that set, because both arms begin there.

Two design choices around this predicate carry the security:

• Deny by default. A document with no ACL rows is visible to nobody. Fail-open here would mean one ingestion bug silently publishes a document to the whole company.
• Groups come from the database, never from the request. The principals passed into the query are resolved by resolve_principal, which looks up the user's groups in a users table. The /ask endpoint deliberately does not accept a groups field. If a caller could assert its own group membership, the ACL would be decorative.

Retrieval is hybrid. Vector search is good at "what is our policy on remote work" and useless at ERR_4021 or "Policy 7.3". Keyword search is the opposite. Real questions contain both, so both arms run and the two ranked lists are fused with Reciprocal Rank Fusion at k=60, which combines them without needing their raw scores to be comparable.

Measuring it, not asserting it

"Permissions are enforced at retrieval" is a slogan until there is a number attached. So vaultrag eval runs a gold set of (user, question, what-they-should-and-should-not-see) against a real corpus and reports two metrics that only mean something together:

• leak rate: did anything the user may not see surface. One leak is a failure, and the target is exactly zero.
• recall: of the documents they may see that answer the question, how many came back.

The pairing is the entire point, because each is trivial to fake alone. Retrieve nothing and you score a perfect 0% leak rate. Retrieve everything and you score perfect recall.

The demonstration I like most is deleting the ACL predicate and re-running the diff:

$ vaultrag diff before.json after.json
LEAK
  leak rate: 0.0% -> 81.8%
  mean recall: 100.0% -> 100.0%

Read the second line. Recall did not move. The broken build answers every question correctly and completely, while handing the user the CEO's private notes and HR's salary bands. A quality-only eval scores that build perfect. That is exactly why leak rate is never reported on its own, and why CI fails on --strict rather than on some threshold.

The proof lives in the corpus design. Every document in the test set contains the phrase "quarterly bonus payout policy". A retriever without access control would happily hand the CEO's private note to anyone who asks about bonuses. tests/test_acl.py asserts the boundary 12 different ways, and if you delete the ACL predicate, 9 of those 12 fail immediately. These run against a real Postgres, not a mock, because mocking the database would mean mocking the thing under test.

An honest limitation

The refusal question, whether the model declines to answer when it has no real evidence, is deliberately reported as not measured under the test setup. The test suite uses a deterministic offline embedder and a scripted stub LLM so that verifying access control does not require a 90MB model download or an API key. But refusal is a property of the actual model, and scoring a stub on it would launder a fake into a metric. So CI proves only the two things it can actually prove: leak rate and recall.

There is no reranker model either. A cheap score blend with an official-first, most-recent tie-break stands in for one. And there is no UI yet. These are real gaps, not deferred marketing.

Worth adding: the eval harness caught two real bugs in this repo before it was committed. A refusal threshold was set below the maximum score RRF at k=60 can produce for a single-arm hit, so those hits were always refused. And conftest.py once reset the schema of whatever DATABASE_URL pointed at, which let the eval report 0% leaks and 100% pass rate against an empty corpus. The most convincing wrong answer a tool can give is the one that says everything is fine.

Closing

The stack is FastAPI, Postgres with pgvector, sentence-transformers running locally with no API key, and Groq for generation, so it costs nothing to run. There are 56 tests passing against real Postgres plus pgvector, and an 11-case gold set at 0% leak rate, both gating CI.

If retrieve-then-filter has ever made you nervous, the code is here: https://github.com/royalpinto007/vaultrag