Prepare documents before you index them, or the system will quote a retired policy with total confidence. Remove duplicates, retire superseded versions, give every collection a named owner, label sensitivity, and scope each set to the department that owns it. Mickai builds these knowledge bases on hardware you own, so preparation stays internal and nothing is uploaded.
Why does document preparation decide whether AI answers are any good?
Because a retrieval system has no opinion about which of your documents is correct. It finds the passage that best matches the question and puts it in front of the person who asked. If three versions of the expenses policy sit in one folder, the one it quotes is whichever reads most like the question, not whichever is in force.
That is the failure mode people underestimate: the worry is usually invention. The commoner problem is accurate retrieval of the wrong document: it arrives with a filename, a date and a page reference, so it looks like diligence. A manager approves a spend at a threshold that changed eighteen months ago, and the reason given is that the system said so.
Preparation decides whether the answers can be relied on. I would rather index four thousand documents that someone owns than forty thousand that nobody does.
How do we find duplicates and superseded versions?
Start with the file system, not with the AI. Hash every candidate file and group the identical hashes. That clears the exact copies, which in most estates is a surprising share of the volume, because one attachment has been saved into six places by six people.
Near-duplicates are the harder half. Same title, different dates, the family of names ending in final, final v2 and FINAL with someone's initials in brackets. Cluster by title and content similarity, then hand each cluster to the department that owns it. The owner picks the one that answers. The rest are marked superseded.
Marking is not deleting. Retention obligations almost certainly mean you keep the old version, you just stop answering questions out of it: retire the document from the index, not from the record. Superseded versions move to an archive a records team can search and the assistant cannot.
Write the rule down, because the rule outlasts the clean-up. One version wins: for any given policy, exactly one document in the index answers for it.
What do we do about documents with no owner?
They do not go in. If nobody will put their name against a document, it is not fit to be quoted back at your staff as company policy.
Circulate the unclaimed list by department with a date on it. Much of what nobody claims is genuinely dead: superseded, abandoned in draft, or belonging to a function that no longer exists. The rest finds an owner quickly, once people grasp the document might be cited in a decision.
Ownership is not authorship. Whoever wrote the procedure may have left. The owner is the role accountable now, and roles survive staff turnover in a way that names do not.
How should we label sensitivity before indexing?
Use a small number of tiers and make the label operational. Open, internal, restricted, and a tier for anything holding personal data is enough. The label has to govern retrieval, not just file permissions, because an assistant that can read a restricted document will quote from it to whoever asks.
A document sitting on the shared drive is not a reason to index it. The ICO's guidance on AI and data protection deals with security and data minimisation directly, and the principle is the ordinary one: personal data must be adequate, relevant and limited to what is necessary in relation to the purposes for which it is processed. An index built by pointing at everything fails that test before it answers a single question.
Where a collection covers staff or customer personal data, a data protection impact assessment is the right instrument. The ICO is explicit that a DPIA is required where processing is likely to result in a high risk to individuals, and that it must be carried out prior to the processing. Do it during preparation, while you can still change what goes in.
Should we index everything, or scope by department?
Scope by department, then share specific collections deliberately. One index across the whole organisation gives worse answers and a wider blast radius at the same time.
Worse answers, because retrieval quality falls as a collection grows less coherent. A finance question asked against a corpus that also holds marketing copy, engineering notes and eleven years of minutes competes with text that is superficially similar and substantively irrelevant.
Wider blast radius, because one index means one permission boundary, and the first time someone retrieves what they should not have seen, the whole thing gets suspended. We build separate knowledge bases per function for that reason, with the platform's fifty specialised models available to them, so a legal collection and an operations collection are different objects with different owners and different access rules.
Which file types cause the most trouble, and how do we handle scans?
Two categories cause nearly all of the pain: scanned documents with no text layer, and spreadsheets where the meaning lives in the layout. Word processor files and digitally generated PDFs are largely straightforward.
A scanned page is an image. It holds no text until something reads it, and the quality of that reading depends on the quality of the scan. Faxed pages, second-generation photocopies, dense tables and handwritten annotations are where accuracy drops. Sample the worst part of the archive and read the output yourself before you commit.
On our side, a local optical character recognition runtime has read scanned PDFs in controlled tests. The extraction and ingestion path into the operating system is still being completed, so plan a scanned archive as a later phase rather than day one. Size the hardware early, because reading an archive at volume is a GPU workload and everything we build is hardware-gated by design.
There is a governance point buried in those boxes. Old personnel and case files often hold personal data that has sat effectively unfindable for years. Making it instantly searchable is a change in processing, not a change in storage, and it deserves a decision rather than a default.
Spreadsheets need a different answer. A figure means what it means because of the column heading above it, and that context is frequently lost in extraction. Export the parts that matter into a labelled narrative form, or leave the workbook out and let the system point to it rather than quote from it.
How do we keep the knowledge base current after go-live?
Tie the refresh to the document, not to the calendar. A single global interval will be too slow for what changes often and pointless for what never changes.
Every collection has an owner, and the owner sets the trigger. A policy reviewed on a fixed cycle is re-indexed on that cycle. A procedure that changes when a regulator changes its expectations gets a trigger tied to that, not to a date. Signed minutes are added when they are signed. The event that changes the document is the event that updates the index.
Two mechanics make this survivable. The index must be able to say which version it holds, so a question about currency has a real answer. And answers must show the source document and its date, because the fastest detector of a stale index is a person who knows the policy and sees the wrong date on it.
What changes when the whole exercise stays on your own estate?
Preparation stops being a data transfer. You are sorting your own files on your own machines, so nothing is uploaded and there is no separate argument about whether the policy library, contracts and board papers may leave the building. For a regulated organisation that removes the slowest part of the procurement.
Two things follow from running on hardware you own. Consequential actions wait for a named person to approve them: the system proposes, a person decides. And every consequential action is sealed in the Open Audit Record under ML-DSA-65, the post-quantum signature scheme NIST published as FIPS 204 in 2024. That record is tamper-evident rather than tamper-proof. Nothing stops a person altering a file, but altering it breaks the signature, so verification fails. Your auditor checks an exported record offline with a public key, using tools that are not ours.
This is not an argument against the companies building the compute or the cloud platforms. Cloud remains the right answer for a great deal of non-regulated work. The argument is narrower. A regulated organisation should not have to rent its intelligence, ship its documents offsite and take a vendor's word for what happened to them. That is what the sovereign approach is for. The NCSC's guidelines for secure AI system development place documenting your data, models and prompts inside secure development, which makes a provenance record a security activity rather than an administrative one. The register you build during preparation is what you hand an auditor later.
Our closed beta is open, with one regulated company onboarding as a design partner.
Frequently asked questions
Do we need to clean up our documents before using AI?
Yes, at least to the point where every indexed document has an owner and one current version. You do not need a perfect archive. You need duplicates removed, superseded versions retired to an archive the assistant cannot read, and sensitivity labels that govern retrieval. Do that per department rather than attempting the whole estate at once.
Will the AI quote an out-of-date policy?
It will if you index one. Retrieval has no way of knowing which version is in force, so it quotes whichever passage best matches the question, complete with a filename and a date that make it look authoritative. The fix is version control before indexing: one document answers for each policy, and everything superseded moves out of the index.
Can AI read scanned PDFs from our archive?
A scanned page is an image, so it has to be read before it can be searched. A local optical character recognition runtime has read scanned PDFs in our controlled tests, and the extraction and ingestion path into the operating system is still being completed. Treat the scanned archive as a later phase, sample it first, and size the hardware for the workload.
How often should an AI knowledge base be refreshed?
As often as the documents change, which is a different cadence for each collection. Do not pick one global interval. Give every collection an owner and a trigger: re-index a policy when its review completes, a procedure when the regulatory expectation behind it moves, minutes when they are signed. Then make answers show the source document and its date so staleness is visible.
Do we need to train a model on our documents?
No, and it is usually the wrong instinct. Retrieval over a prepared, owned collection answers policy questions better than training, because you can see exactly which document produced an answer and retire that document when it changes. Training bakes the content in, so a superseded policy becomes very hard to remove. Prepare the documents instead.
Related briefings
Documents and retrieval
- What Is RAG? Retrieval-Augmented Generation Explained
- AI Permissions Audit: Stop Oversharing Before Rollout
- Revoked Access in AI Search: Does the AI Stop Using It?
- From Documents to Decisions You Can Defend
Governance, audit and oversight
- Tamper-Evident vs Immutable Log: The Real Difference
- Human on the Loop vs In the Loop: AI Oversight Explained
Part of a series of 60 briefings on deploying and governing AI in UK regulated organisations, archived with a DOI at 10.5281/zenodo.22975756.
Evaluating AI for a regulated organisation? Mickai runs on hardware you own, offline. Consequential actions wait for a named person to approve them, and what the AI did is sealed into a signed record an auditor can check without us. Applications for the invitation-only closed beta are open. Apply for the closed beta.
Written by Micky Irons, founder and chief executive of Mickai LTD.
Top comments (0)