It depends on where the permission check happens. If permissions are copied into the index when a document is ingested, a withdrawn document can keep feeding answers until the next index run. Ask any vendor whether permission is checked at the moment the question is asked. Mickai checks then, and seals consequential actions in a tamper-evident record.
Why does revoked access sometimes still show up in AI answers?
Because the permission was checked once, when the document was ingested, and then stored as part of the index. Removing a person from a folder in the source system does not, by itself, change anything the assistant has already written down about who may read what.
A retrieval assistant is two systems wearing one coat. There is the source of truth: the document store, the case system, the shared drive, each with its own access control list. Then there is the retrieval layer: passages of text, vector representations of those passages, and a metadata record saying which groups each passage belongs to. That metadata is a copy, and copies go stale. When an administrator withdraws access, the source is corrected at once. The copy is corrected whenever the pipeline next runs.
So the honest answer to "how quickly" is not a number of seconds. It is a question about where the check lives. Either the system resolves entitlement at the moment of the question, or it trusts a snapshot, in which case your exposure lasts as long as the snapshot does. I would not accept a stopwatch figure in place of that answer, from us or from anyone else.
What is the difference between index-time and query-time permissions?
Index-time permission is a label written during ingestion and filtered on later. Query-time permission is a check made when the question is asked, against the person's current entitlement, before any passage reaches the model.
Index-time filtering is common because it is cheap and fast. It also fails in the ways permissions usually change in a real organisation. Somebody leaves a group rather than losing a document. A file moves into a restricted area and inherits a new access list. A matter is reclassified. A contractor's role ends while their account stays open for handover. In each case the document's own identifier never changes, so nothing in the pipeline notices that the label attached to it is now wrong.
Query-time checking costs you something. Entitlements have to be resolved on every request, and retrieval has to be able to discard candidate passages after the search rather than before it, which means searching more broadly to return the same quality of answer. That is the trade I chose. In Mickai, knowledge sits in private knowledge bases read by our specialised models, which we call brains, and a passage only reaches an answer if the person asking is permitted to read it at that moment. The NCSC's guidance on identity and access management makes the underlying expectation plain: access is granted on the basis of need and removed when the need ends. An index that answers from last week's need is not implementing that.
What happens to stored embeddings when a document is withdrawn?
Usually less than buyers assume. Most vector stores implement deletion as a soft delete: the record is marked as gone and filtered out of results, while the vector itself stays in the index file until a compaction rewrites it.
That matters twice over. Filtering is a behaviour, and behaviours can be undone by a bug, a rollback, a restored snapshot, or an index rebuilt from an older file. And the vector is not an opaque number. A June 2026 preprint, "Ghost Vectors: Soft-Deleted Embeddings Remain Reconstructible in HNSW Vector Databases", showed that soft-deleted embeddings can be recovered from raw index files and partially inverted back into text, and proposed encrypting vectors and destroying the key on deletion so that recovery fails even when the bytes survive.
Read practically, that means treating a deletion as pending until you can show either that the bytes are gone or that the key is gone. Ask whether deletion is hard or soft. Ask what triggers compaction. Ask the same question of backups, of answer caches, of generated summaries, and of question logs that may quote passages. The ICO's guidance on security and data minimisation in AI expects that assessment to cover the whole lifecycle, not only the front door. Derived copies are where most of the surprise lives.
One caveat I would rather state than be asked. A local OCR runtime has read scanned PDFs for us in controlled tests, and extraction and ingestion integration inside SIOS is still being completed. Where scanned material is involved, that is work in progress rather than a finished claim.
What tests should we run before we trust the answer?
Run a canary test with two accounts and one document. Put a phrase in the document that appears nowhere else in your estate, so that any reappearance is unambiguous.
The sequence I would use: confirm the permitted account can retrieve and quote the phrase, then withdraw access, then ask again as that account, immediately, without waiting for any scheduled job. Ask directly for the phrase. Ask a question whose answer only exists in that document. Ask for a summary of the area the document covers, because summarisation is where leakage hides. Check the citation list as well as the prose. Then repeat the whole thing three more ways: by removing the person from a group instead of from the document, by moving the document into a restricted location, and after restoring the system from backup. Finally, test the opposite direction. Granting access should take effect on the next question too, and a system that is slow to grant is usually a system that caches.
Do this on your own corpus, with your own directory, before anyone signs anything.
What should we ask a vendor about permission changes?
Ask where the permission check happens, and require a mechanism rather than an adjective in reply. Five questions separate the two quickly.
Is entitlement resolved at query time against the live directory, or filtered from index metadata? When a person is removed from a group, what has to run before the assistant stops using the affected documents? Is deletion from the vector store hard or soft, and what happens to the index file, the backups and the caches? Which artefacts does the assistant create that outlive the permission check, for example saved conversations, exported answers and scheduled reports? And can you show me, in a record I can verify myself, which sources an answer drew on and who approved any action taken from it?
If the last question is met with a screenshot of a dashboard, you have a marketing answer to an audit question.
How do leavers and movers change the risk?
Leavers are the easier half, because the account usually goes with them. Movers are harder: the person is still employed, still authenticated, and only their entitlements have changed, which is exactly the case an index-time label handles worst.
A leaver process built before retrieval assistants existed tends to stop at accounts, mailboxes and devices. It needs three more lines now. Delegated and shared access, because a departing head of function often holds standing access through a shared mailbox or a service account. The assistant's own artefacts, since a saved conversation can hold quoted passages from documents the reader no longer sees. And a verification step, because "we removed the group membership" is a ticket, not evidence. Movers need the same treatment on a change of role, which is the event most organisations still handle informally.
What does a tamper-evident record add when access is disputed?
It turns an argument into an examination. When someone asks what the system used to answer a question in March, you want a record the other side can check without having to trust you.
The Open Audit Record is append-only and hash-chained, and entries are signed with ML-DSA-65, the post-quantum signature scheme NIST published as FIPS 204 in 2024. An auditor exports a record and verifies it offline with a public key, using tools that are not ours. Consequential actions wait for a named person to approve them before they run, and that approval is part of the sealed record.
Tamper-evident is not tamper-proof, and the difference is the whole point. Nothing stops a sufficiently privileged person editing a file on disk. What the chain does is make the edit fail verification, so alteration becomes visible instead of silent. The assurance has limits I would rather set out than have found: it covers history up to the most recent checkpoint a verifier has already seen, and it rests on custody of the signing keys. Say both to your auditor at the start and you will not be arguing about them later.
None of this is an argument against the companies building the compute and cloud layer. Cloud remains the right answer for plenty of work. What I dispute is the assumption that a regulated organisation must rent its intelligence, ship its documents offsite, and take a supplier's word for what was read and by whom. SIOS runs on hardware you own, offline where you need it, with no data egress. Our closed beta is open, and one regulated company is onboarding as a design partner.
For the mechanics, see internal knowledge AI, on-premise retrieval and knowledge sovereignty and subject access requests about AI processing, which is the question you will be asked next.
Frequently asked questions
Can an AI assistant quote a document I no longer have access to?
Yes, if it filters on permissions copied into its index rather than checking your entitlement when you ask. The source system has withdrawn you, but the retrieval layer still holds the passage and a stale label saying you may read it. Ask where the check happens, then prove it with a canary document rather than accepting the answer.
How do we test that revoked access actually works?
Plant a phrase that exists in one document and nowhere else. Confirm a permitted account can quote it, withdraw that account's access, then ask again straight away without waiting for any scheduled job. Ask for the phrase, for a summary of that area, and for a fact only that document holds. Repeat after a backup restore.
Does removing a document from the index remove it from answers?
Not reliably. Most vector stores soft delete: the record is filtered from results while the vector stays in the index file until compaction rewrites it, so a rollback, a rebuild or a restored snapshot can bring it back. A June 2026 arXiv preprint, "Ghost Vectors", shows soft-deleted embeddings can also be recovered from raw index files. Check caches, backups and derived summaries too.
What should a leaver process cover for AI search?
Four things beyond the account itself. Entitlements and group memberships, including nested groups. Delegated and shared access such as shared mailboxes and service accounts. The assistant's own artefacts: saved conversations, exported answers and scheduled reports that may quote documents the person can no longer open. And a verification step, because a closed ticket is not evidence.
Is a nightly re-index good enough for permission changes?
Only if you are content for the interval between runs to be your exposure window, because that is what it is. A withdrawn document can keep feeding answers until the next run completes, and a failed or partial run extends it silently. Checking entitlement when the question is asked removes the question rather than shortening it.
Related briefings
Documents and retrieval
- What Is RAG? Retrieval-Augmented Generation Explained
- How to Prepare Documents for a Private AI Knowledge Base
- AI Permissions Audit: Stop Oversharing Before Rollout
- From Documents to Decisions You Can Defend
Governance, audit and oversight
- Tamper-Evident vs Immutable Log: The Real Difference
- Human on the Loop vs In the Loop: AI Oversight Explained
Part of a series of 60 briefings on deploying and governing AI in UK regulated organisations, archived with a DOI at 10.5281/zenodo.22975756.
Evaluating AI for a regulated organisation? Mickai runs on hardware you own, offline. Consequential actions wait for a named person to approve them, and what the AI did is sealed into a signed record an auditor can check without us. Applications for the invitation-only closed beta are open. Apply for the closed beta.
Written by Micky Irons, founder and chief executive of Mickai LTD.
Top comments (0)