AI that works in a pilot fails in the real environment because the pilot removed the four things that decide the outcome: messy real data, enforced permissions, restricted networks and ordinary staff. Put those back and most systems struggle. Evaluate on representative files, under real identities, on hardware you own.
What is actually different between a pilot and the real environment?
Four things change: the data, the permissions, the network and the people. A pilot usually removes all four. Someone assembles a tidy folder of documents, runs it through a hosted system on an open connection, and presents the output to a room that already wants it to work.
None of that survives contact with the organisation. The real corpus is decades deep and inconsistently scanned. The real permission model says that several people in that room are not entitled to see half of it. The real network sits behind a proxy that strips the headers the vendor's client library expects. The real users are busy, sceptical, and paid to do something other than help a pilot succeed.
I build the Mickai Sovereign Intelligence Operating System (SIOS), which runs on hardware the customer owns. I mention that because it shapes my view. When the software runs inside your building, you meet all four problems in the first week instead of the ninth month. That is uncomfortable, and it is the only honest sequence.
Why does real data break a system that handled the demo set?
Because demo sets are curated and real corpora are not. The gap is rarely about model quality. It is the distance between the files someone chose to show you and the files that actually exist on your estate.
Real documents are photocopies of faxes. They carry handwritten margin notes that reverse the clause above them. They exist in three versions, two of them superseded, none of them clearly labelled. Tables run across page breaks. Headers repeat mid-document. A scanning artefact turns a decimal point into a comma and a figure moves by a factor of a hundred.
A system evaluated on twenty clean PDFs learns nothing about any of that. Worse, it produces a number that feels like evidence. A high score on the sample becomes "accurate" in the business case, and the first time anyone runs it across the shared drive the figure collapses, because the shared drive is where the real distribution lives.
The remedy is dull. Evaluate on files drawn at random from the live store, including the ones nobody wants to show you. If that cannot be done because the data is not permitted to leave the building, that is not an obstacle to the evaluation. It is the most important finding of the evaluation.
What happens when permissions enter the picture?
Retrieval ignores entitlements unless somebody builds entitlements into retrieval. In a pilot, the person running it usually holds administrator rights, so everything is visible and nothing is tested.
Then it goes live, and a junior analyst asks a perfectly reasonable question. The system answers it correctly, using a board paper they were never allowed to open. Nothing was hacked. The index simply had no idea who was asking.
That is not an abstract governance point. The obligations sit in the UK GDPR and the Data Protection Act 2018, and the ICO guidance on AI and data protection is clear that access control and data minimisation apply to an AI system exactly as they apply to anything else, with a data protection impact assessment likely to be needed where personal data is processed at scale.
So test under real identities. Give three people at three entitlement levels the same question and compare what comes back. If the three answers are identical, the system is enforcing nothing, and your pilot has proved the opposite of what you believe it proved.
Why do network constraints decide more outcomes than accuracy does?
Because the best system in the world is useless if your data is not allowed to reach it. Many regulated environments do not permit outbound traffic from the segment where the interesting material sits, and no amount of model quality changes that rule.
Pilots hide the problem by running somewhere convenient: a laptop on guest wifi, a tenancy stood up for the trial, a sandbox with a firewall exception written specifically for it. The exception expires. The production segment has an egress proxy, TLS inspection and a standing rule that nothing leaves. Sites lose connectivity as a matter of routine: a plant, a vessel, a depot or a forward location works offline by default, not as an edge case.
The NCSC guidelines for secure AI system development treat the whole pipeline, including everything the system talks to, as part of the attack surface. If you are an FCA-regulated firm, the operational resilience rules ask you to know which services your important business services depend on and what happens when one becomes unavailable. A capability that stops the moment a third party's data centre is unreachable is a dependency you do not control.
The evaluation question is therefore not only whether the answer is right. It is whether producing the answer required anything to leave. Disconnect the network during the test. If the system stops, you have learned precisely what you would be buying.
What do real staff do that a pilot never sees?
They paste. They paste the wrong thing into the wrong box at ten to five on a Friday. They ask questions shaped like their job rather than shaped like your prompt guidance. They accept a plausible answer because they are behind, and a plausible answer is exactly what a confident system produces when it is wrong.
A pilot staffed by five enthusiasts measures enthusiasm. A real environment contains people who do not want the tool, people who will try to break it, and people who will trust it far more than you intended. All three are useful. The sceptic finds the edge cases nobody scripted. The over-truster shows you where a named human has to stand in the path, because that is precisely where an unchecked action would have caused damage.
That is why consequential actions in SIOS wait for a named person to approve them. Not a role, not a service account, a person, recorded at the moment of approval. It is slower. It is also the difference between an incident you can explain to a regulator and one you cannot.
How do you run an evaluation that actually tells you something?
Run it where the work happens: your hardware, your files, your identities, your staff. Everything else is a demonstration wearing the word evaluation.
A workable shape looks like this. Take several hundred documents drawn at random from the live store rather than selected by the vendor. Have your own subject matter experts write down the correct answers before they see any output, otherwise you will grade on plausibility rather than accuracy. Run the same questions under three entitlement levels. Run once connected and once disconnected. Log every consequential action and confirm a named person approved it. Then export the record and verify it somewhere else entirely.
That final step matters more than it sounds. In SIOS, every consequential action is sealed in the Open Audit Record under ML-DSA-65, the post-quantum signature scheme NIST published as FIPS 204 in 2024. The record is tamper-evident: altering an exported record makes verification fail, and that failure is the signal you need. An auditor can check the export offline with the public key, using tools that are not ours. If a vendor's audit trail can only be verified inside the vendor's own product, you do not have an audit trail. You have a claim.
What should you ask before you commit to anything?
Ask where the evaluation will run, and insist on your own hardware. Ask what happens when the connection drops. Ask to watch the same question answered for three people with different entitlements. Ask who approves a consequential action, by name, and where that approval is recorded. Ask how an auditor verifies the record with the vendor absent from the room.
For our part: Mickai LTD is a UK company, number 17166618, held privately by me. MICKAI is a registered UK trade mark, UK00004373277. We hold 104 filed UK patent applications carrying 2,340 claims, filed and not granted, with me as named inventor. SIOS ships with 63 studios, 14 production-ready at launch and 49 in development, alongside 50 specialised models. The closed beta is open and one regulated company is onboarding as a design partner.
I would rather you tested all of that inside your own building than took my word for any of it. We are not against the companies that build the compute or cloud layer, and cloud remains sensible for work that is not regulated. What I will argue with is the assumption that a regulated organisation has to ship its data offsite and accept a supplier's account of what happened to it. A pilot that flatters the software tells you nothing you can act on. A test that runs in the real environment, with the real constraints in place, tells you whether the thing works.
Frequently asked questions
Why do AI pilots succeed when production deployments fail?
A pilot removes the four things that decide the outcome: messy real data, enforced permissions, network restrictions and ordinary users. Remove all four and almost any system looks capable. Put them back and most do not cope. Test with real files, real entitlements, real network rules and real staff, on hardware you own, before committing to anything.
How many documents should an AI evaluation actually use?
Several hundred at minimum, drawn at random from the live store rather than selected by a supplier. Twenty curated PDFs tell you nothing about the distribution you actually hold. Include the poor scans, the superseded versions and the handwritten annotations, because those are the files that produce confident wrong answers once the system is live.
Can an AI system respect our existing access permissions?
It can, but only if entitlements are enforced at retrieval rather than bolted on afterwards. Test it directly: ask three people at three clearance levels the same question and compare what comes back. If all three receive the same answer, nothing is being enforced, and the index will eventually surface a document to someone not cleared for it.
Does AI have to run in the cloud?
No. The Mickai Sovereign Intelligence Operating System runs on hardware the customer owns, is capable of running offline, and sends no data out. Cloud remains sensible for work that is not regulated. The point is that a regulated organisation should not be obliged to ship sensitive material offsite and accept a supplier's account of what happened to it.
What evidence should a buyer require from an AI supplier?
An exportable record of what the system did, verifiable without the supplier present. In SIOS every consequential action is sealed in the Open Audit Record under ML-DSA-65, published by NIST as FIPS 204 in 2024. It is tamper-evident: altering an export makes verification fail. An auditor checks it offline with a public key.
Written by Micky Irons, founder and chief executive of Mickai LTD, which builds a sovereign AI operating system for regulated organisations. More at mickai.co.uk.
Top comments (0)