Filter at Query Time: Getting RAG Permissions Right

Here is a bug that ships to production constantly, passes code review, and looks fine in every log you have.

1. user asks a question
2. embed query
3. vector search  ← runs as service account, full corpus access
4. take top-k chunks
5. build prompt, call model
6. filter citations the user can't see   ← authorisation happens HERE
7. return answer

Step 6 is too late. By then the model has consumed the chunks from step 4, and the answer in step 7 is derived from them. You hide the citation; the content still goes out in prose.

The user gets an accurate summary of a document they cannot open. Nothing in your logs looks unusual, because the retrieval service was authorised, the model call was authorised, and the session was authorised. This is a real data leak that generates zero signal.

The correct order

Authorisation belongs inside the search, not after it:

1. user asks a question
2. resolve requesting user's identity → principal set (user + groups)
3. embed query
4. vector search FILTERED by acl ∩ principal set   ← authorisation HERE
5. top-k chunks — all already readable by this user
6. build prompt, call model
7. return answer + citations (no suppression needed)

The model can only ever see what the user could have opened directly. Citation suppression becomes unnecessary, which is a good smell — you are no longer hiding evidence of a problem.

Implementing it

Every chunk carries its source document's ACL as queryable metadata at ingestion time:

{
  "chunk_id": "doc_8842:c17",
  "text": "...",
  "embedding": [0.0123, -0.0456, "..."],
  "source_doc": "doc_8842",
  "acl": ["group:finance", "group:exec", "user:1182"],
  "classification": "confidential",
  "ingested_at": "2026-09-19T08:00:00Z"
}

At query time you resolve the caller into a principal set and pass it as a pre-filter:

principals = identity.resolve(request.user)
# e.g. {"user:1182", "group:engineering", "group:all-staff"}

results = index.search(
    vector=embed(query),
    filter={"acl": {"$in": list(principals)}},   # applied BEFORE ranking
    top_k=8,
)

The critical detail is $in evaluated before ranking, not as a post-filter on results. Several vector databases will happily accept a filter and apply it after retrieving top-k, which silently gives you fewer results rather than correctly-scoped ones. Read your engine's docs on whether filtering is pre- or post-ranking. If it only does post-filtering at your scale, that is a database selection problem.

Four traps

Stale ACLs. The permission was copied into the index at ingestion. Someone leaves the finance group on Tuesday; your index still says they can read. You need either a re-sync job on permission-change events, or resolution of the ACL against the live identity system at query time. Event-driven sync is the usual compromise — full re-index on every group change does not scale.

Deleted documents. Deleting the source document does not delete its chunks. Wire deletion into the ingestion pipeline as a first-class operation, and test it. This is also what makes your system able to honour erasure requests, which matters more than it sounds.

Principal explosion. A user in 400 nested groups produces a filter clause that some engines handle badly. Flatten group membership at resolution time and cache it with a short TTL rather than expanding nested groups inside the query.

Chunk-level vs document-level. If you chunk across document boundaries — merging a public intro with a restricted appendix, say — a single chunk can carry two different ACLs. Do not intersect them into something permissive. Chunk within document boundaries and inherit the document's ACL unmodified.

Classification belongs in ingestion

While you are touching the pipeline, put a classification gate in it. Once a document is chunked, embedded, and indexed without a sensitivity label, applying one later means re-embedding the corpus — a cost that grows every week you wait.

The pattern:

ingest → classify → {
    below threshold → general index
    above threshold → restricted index (own endpoint, own logging, own retention)
}

Two indexes with simple, auditable rules beat one index that depends entirely on filter correctness. The restricted path can also use a different model deployment or region, which is what makes residency commitments tractable.

Model-based classifiers are genuinely good at this now — far better than the regex-and-keyword approach, which could never distinguish a real card number in a complaint from one in a test fixture. This is one of the places AI improves a security problem instead of creating one.

While you are here: agent tools

If you are also running agents, the same principle applies to tool grants. An agent's effective permissions are the union of every tool you gave it. An agent with a database read tool and an email send tool has an exfiltration capability, regardless of what its system prompt says.

• Prefer lookup_order_status(order_id) over run_sql(query) — narrow, parameterised tools are enforceable in a way that general ones are not.
• Run the agent under the requesting user's identity so existing authorisation applies.
• Human approval in front of anything that writes externally or sends messages.
• Egress allowlists, so a successful prompt injection has nowhere to send data.
• Log the full tool-call sequence, not just the final answer.

Prompt injection is not solvable with better instructions. Treat it as privilege escalation and contain the blast radius.

Frequently Asked Questions

Why not just filter the citations?

Because the model already read the content and the answer is generated from it. Hiding the citation hides provenance, not information. The user still receives the restricted content, in prose.

Does pre-filtering hurt recall?

It changes what top-k means — you get the best k results the user can see, which is correct behaviour. If recall feels poor afterwards, the real issue is usually that the user genuinely lacks access to the relevant material.

How do I keep ACLs fresh in the index?

Event-driven sync on permission changes is the practical middle ground. Resolving fully against the live identity system at query time is the most correct and the most expensive; full re-indexing on every change does not scale.

Should sensitive content live in a separate index?

In regulated environments, yes. It lets you apply different endpoints, logging, retention, and even a different model region, and it is far easier to defend in an audit than relying solely on filter correctness.

Can embeddings leak the source text?

Yes, meaningfully — inversion research reconstructs substantial portions of short passages. Treat the index as a copy of the corpus at the same classification level.

What about fine-tuning on internal data?

Avoid it for anything that might need deletion. You cannot remove a record from model weights; honouring erasure means retraining. Retrieval deletes cleanly — remove the doc, the chunks, and the vectors.

---

Full write-up with the audit logging schema, residency mapping, and a ninety-day rollout sequence: Enterprise Data Protection in the Age of AI Agents.

We build this layer for teams shipping AI into regulated environments — LLM integration and AI development services.