Last week ParadeDB asked HN whether we still have to denormalize to search across a schema. The comments under it argued about something older than search. One person put the cost plainly: "if you simply denormalize everything, each time you need to eg update a user name, you suddenly have to update its whole posts and comments everywhere." Another said "once you denormalize it immediately becomes a form of cache, and cache carries a ton of footguns."
Both are right, and in that thread the stakes are a stale display name. Now look at the field most retrieval stacks denormalize first: who is allowed to read this chunk. A stale username is a typo. A stale permission is a document shown to somebody who no longer has the right to see it.
Most internal RAG systems can tell you, correctly and in detail, how they decide whether someone may read a document. Far fewer can tell you what happens in the minutes after that answer changes.
The decision that creates the problem
You're indexing a few million chunks. At query time you must return only what the caller may read. The obvious way to make that fast is to keep the answer next to the data: write an allowed_principals array into each chunk's payload at ingest and let the vector store filter on it. One index, one round trip, no join.
That array is a cache of an authorisation decision. Like every cache, it's correct until the thing underneath it changes.
From the moment you write it, "who may read this document" lives in two places. One is whatever actually governs access: your IdP, your group memberships, a grants table, the source system's own ACLs. The other is your index. Something has to keep the two in agreement, and that something is where revocation lives. It rarely gets the care the permission model got.
Three shapes, one failure
The copy turns up in different places depending on the stack. They fail the same way.
In the vector payload. ACLs written per chunk at ingest. Revoking means rewriting every chunk of every affected document, and it's per chunk, not per document. A 40-page policy is not one write. It's a few hundred.
In the session. Permissions resolved when the token is issued and carried in its claims, or cached against the session on first use. This one feels fine because the usual check passes: log out, log back in, the change is there. But a token issued five minutes before the revoke keeps the old answer until it expires. If your access tokens last an hour, your revocation takes an hour, whatever the admin console said when you clicked.
In a sidecar table synced on a schedule. The honest version. A job reconciles group membership into the retrieval layer every fifteen minutes, or nightly, or on a webhook with a retry queue. Nobody pretends it's instant. What goes unsaid is that the schedule is the exposure window. You picked it, and you own what happens inside it.
None of these is a bug. Each is a reasonable decision, and each turns "revoke access" from an operation into a propagation. A propagation has a duration, and sooner or later a security review will ask you for it.
Groups are where it gets expensive
Single-document grants are the easy case, and the one people test.
The cost is in groups. One person leaves one group, and the set of chunks whose ACL must change is every chunk of every document that group could reach. That's a bulk rewrite of a large slice of the index, triggered by an HR event, competing with ingest for the same write path.
So the sync job gets a batch size. The batch size gets turned down after it causes a latency spike, and the window gets longer. Nobody approves the final number. It arrives through a series of sensible operational choices.
What the vector store APIs let you do
An argument like that is easy to wave away, so look at the primitives you'd actually call. Both references below were read on 19 November 2026.
Qdrant's set_payload takes a payload plus either point ids or a filter, and it assigns the payload "to each point that satisfy this filter condition". Its delete_payload takes a list of keys and removes those keys.
Pinecone's update guide has the same shape and says it outright: "A by-filter update applies one value to every matched document." Its by-filter set_fields and remove_fields "accept metadata fields only", and remove_fields drops whole fields.
Read that with a revocation in mind.
Every bulk write is a broadcast. One value, one filter, applied identically to everything that matches. That's exactly right for a grant: add group:claims to every chunk of the claims corpus in one call.
Revoking has a different shape. If the ACL is a list (allowed_principals: ["alice", "bob", "carol"]), taking Bob out gives every matching chunk a different new value, because each one has a different set of other people in it. A filter can find the chunks. It can't say what each one should now contain, and neither API has an operation that removes one element from an array field. So removal drops to per-record writes, and Pinecone's limit for those is "up to 1,000 documents, or 96 when the schema declares embed or sparse_embed (even a metadata-only patch)". A large corpus, chunked, is a lot of requests for one leaver.
Then the last line on the page: "Pinecone is eventually consistent, so there can be a slight delay before updates are visible to queries." That's the window, named by the vendor, on the happy path where your job ran at once and succeeded.
To be fair about it: these are good APIs. Broadcast-by-filter is the right primitive for what a vector database is for, and an array-element-removal operation would be an odd thing to find in a nearest-neighbour store. The point is about those of us building on top. Put the authorisation decision in the payload and you inherit the payload's update semantics, which are generous toward granting and grudging toward revoking. Revoking is the direction anyone will ever ask you about.
Check this on whatever you run. It's a documentation page, not a benchmark.
The test that finds it
Most revocation tests I've seen are built in a way that hides the answer.
The usual version: revoke the grant, log out, log back in, run the query, confirm the document is gone. That passes on a system with an hour of propagation delay, because logging back in refreshes the very cache you meant to measure.
The version that tells you something:
- Sign in and get a token. Run a query that returns a document you can see.
- Revoke the grant, or remove yourself from the group, as an administrator, through the real path.
- With the same token, no refresh, no new session, run the same query again.
Whatever comes back is your actual revocation behaviour. If the document is still there, you have a propagation delay, and repeating step 3 on a timer tells you how long. That number is the honest answer to the question security will ask.
Run it on your own stack whether or not you ever change anything. A window you can state is a risk you can manage. One you can't is how a routine review turns into a bad quarter.
The other design
The alternative has a real cost, so the cost goes first: resolve permissions per request, inside the search.
The index stores no ACLs. It stores what the chunk is and where it belongs. When a query arrives, the caller's grants and group memberships are read as they are at that moment and applied as a constraint inside the index traversal. Not as a filter over results afterwards, which leaks in other ways, some of which the MCP piece goes through.
Revocation then stops being a propagation. There's no second copy, so nothing can be stale, so there's no window to size. It isn't fast because someone optimised it. It's immediate because nothing has to catch up.
What it costs: your retrieval layer has to do the join, so the vector store can't be a dumb nearest-neighbour service with a metadata filter bolted on. That's a genuine constraint and the reason many teams went the other way. You also read permission state on every query, so it has to live somewhere cheap to read. Neither is free.
We built the cache, then took it out
We make one of these, so weigh this section accordingly. It's also the part I'd most want to read if I were you, because it's the failure mode above happening to us.
Nacre's original specification had exactly the payload cache. Each chunk was to carry hashes of the principals allowed to read it, kept fresh by a background sweep, with a lag metric and an alert. All of that was built. Our authorization spec now says what happened next: the filter builder "never emitted" a clause that read the tags, so "the subsystem kept a payload field fresh that no query read, and the one alert an operator was paged by measured the freshness of something unused."
It was removed rather than fixed. The reasons are in that section, and two of them are general enough to apply to your stack:
- The cache was there to avoid a join, but the resolver already computes the caller's whole permitted set on every request. The tags were "a second constraint from the same table, one request staler."
- It was stale in the wrong direction. Intersecting a live permission set with a cached tag set "delays grants by up to the sweep interval while doing nothing for revocations", because the live set already reflected those.
Today the payload carries the organisation, layer and document ids, a deleted flag and your own metadata, and nothing about who may read it. The one cache left in front of the permission check is keyed on a permission epoch that database triggers bump on every write to groups, group members and grants. A changed membership isn't served stale, because the next request builds a different cache key.
What we measured today
That's the design. Here is the three-step test, run on 19 November 2026 against a clean install of the current release (0.41.4) from the repository, following the quickstart's demo block. The demo seeds a small example corpus in three layers and three people whose grants differ. The contractor is the narrowest of them, reaching one layer. One bearer token is held across every step: no re-login, no new token, no reindex, no restart.
| # | step | items the contractor gets back, same token |
|---|---|---|
| 1 | baseline | 3 |
| 2 | admin creates a group and grants it read on a second layer; the contractor isn't in it |
3 |
| 3 | admin adds the contractor to the group | 8 |
| 4 | admin removes the contractor from the group | 3 |
| 5 | admin deletes the group | 3 |
Step 2 is the control: a grant to a group nobody is in changes nothing. Step 3 widened the answer on the very next search; the membership write and that search together took 75 ms. Step 4 narrowed it back, 71 ms for the same pair. Step 5 matches step 1, so the round trip left nothing behind. We also put two groups in a cycle (A in B, B in A) and granted through it: the search resolved through the loop to the same 8 items and came back, rather than hanging.
Two caveats this audience will apply anyway. The corpus is a handful of documents on one machine, so the timings say nothing about scale. They show the permission check isn't the expensive part, and that's all they show. And the shape of the result is the design, not tuning: any system that resolves permissions per request should look similar.
If you'd rather not install anything, demo.nacre.work prints two read-only logins for a hosted stand. Sign in as each at playground.nacre.work, ask the same question, and the two get different answers over the same corpus. It's a stand, not a service, and every organisation on it is erased after 24 hours. What you'd run is the self-hosted Apache 2.0 build, which covers ingest, hybrid search, the MCP server, the REST API and basic RBAC, everything this test touches. Some of what a larger deployment wants is commercial: multi-tenancy, SSO with SCIM, and issuing document-level grants and deny rules. Evaluating them is in the core.
The part that isn't about us
Run the three-step test on whatever you already have, with the same token across the change. It takes ten minutes and swaps a belief for a number.
If the number is zero, good: you built it the hard way and it worked. If it isn't, you now know what it is, which is where you want to be before somebody else asks.
Originally published at https://nacre.work/blog/revocation-is-a-property-of-your-index.
Top comments (0)