Because a demo proves a model can produce an output, and a deployment must prove the whole system: real data under real permissions, integration with systems of record, evidence of what happened, and a named owner after the vendor leaves. Those four things are rarely in scope when the demo is built.
I have sat in both chairs. I have given the demo, and I have been the person who had to make the thing work on the Monday after everyone agreed it was impressive. The two jobs have very little in common, and the distance between them is where enterprise AI budgets quietly disappear.
What does a demo actually prove?
A demo proves one narrow claim: given clean input and a well-chosen question, a system can produce an output that a person recognises as useful. That is worth knowing. It is also a small fraction of what a deployment has to be true about.
Demos are built backwards. You already know the answer you want on the screen, so you choose the documents that produce it, you phrase the question the way the system handles best, and you run it on a machine with no network policy, no access control and no retention rules. None of that is dishonest. It is simply a different exercise. A demo is an argument about capability. A deployment is an argument about an organisation.
The failure is not that the demo misled anyone. It is that everyone in the room assumed the remaining work was engineering measured in weeks, when most of the remaining work is not engineering at all.
Why does real data break what the demo hid?
Because demo data is curated and production data is archaeology. Point the same system at the actual file share and you meet scanned documents with no text layer, four spellings of the same counterparty, a decade of superseded templates and the spreadsheet everyone knows is the real system of record.
That is the easy half. The harder half is that real data carries obligations. If personal data is in scope you need a lawful basis, a defined purpose, a retention position and a record of how you reached each of those, and you need to be able to show the reasoning rather than assert it. The ICO's guidance on AI and data protection is clear that accountability sits with the organisation deploying the system, not with whoever supplied the model (ICO). The underlying duties come from the Data Protection Act 2018 (legislation.gov.uk).
Nobody asks those questions of a demo, because a demo touches nothing that matters. They all arrive in week one of a deployment, usually from a person who was never invited to the demo.
Who is allowed to see what, and can you prove it afterwards?
A demo runs as one user with unrestricted access. A deployment runs as hundreds of users whose entitlements already exist in other systems, and the assistant has to inherit them exactly.
In my experience this is the most common reason a working pilot gets stopped. If the system can read a folder that the person asking cannot, you have not built an assistant: you have built a search box over your access controls. Entitlements live in the directory, in the matter management system, in the case system, in the finance ledger, each with its own idea of what a user is. Reconciling them is slow, political and unglamorous, and it cannot be deferred. The NCSC's guidelines for secure AI system development run from secure design through to secure operation and maintenance, which is a useful reminder that access is a lifecycle question and not a task for the week before go-live (NCSC).
Why does integration eat the timeline?
Because the value sits behind systems that were never designed to be read by software, and touching them is a controlled change. The demo read a folder. The deployment has to read the document management system, write back to the ledger, respect a freeze window and pass a change advisory board.
Each connection brings a service account somebody must own, a rate limit, a test environment three versions behind, and a supplier whose contract says nothing about AI. In a regulated firm, every one of those is a change request with a named approver and a queue in front of it. I would rather tell a buyer that at the start than have them discover it in month four with the budget already committed.
What evidence does an auditor need that a demo never produces?
An auditor needs a record: what was asked, which data was used, what the system did, who approved it, and proof that the record has not been altered since. A demo produces a screen and somebody's memory of it.
This is the part we built first rather than last. In the Mickai Sovereign Intelligence Operating System, every consequential action is sealed in an Open Audit Record under ML-DSA-65, the post-quantum signature scheme published by NIST as FIPS 204 in 2024 (NIST). The record is tamper-evident: alter it and verification fails. An auditor can export a record and verify it offline with a public key, using tools that are not ours. That last clause is the whole point. Evidence you can only check inside the supplier's own product is not evidence, it is a claim.
The other half is approval. Consequential actions wait for a named person to approve them. A system that acts and then explains is a very different object from a system that proposes and waits, and only one of the two survives a serious conversation about control.
Who owns the system once the vendor goes home?
In most stalled deployments, nobody does. That answer has to exist before the pilot starts, and it has to name a person in the business, not a team in IT.
Ask where the model runs. Ask who can restart it at three in the morning, who approves a change that affects a regulated output, and what happens if the supplier stops trading. Firms stay accountable for their own resilience even where a critical third party is involved: the FCA is explicit that the critical third party oversight regime does not change the accountability of firms (FCA). An exit position you could actually execute follows from that. Renting intelligence from an endpoint you cannot inspect makes every one of those questions harder to answer honestly.
That is why SIOS runs on hardware the customer owns, offline capable, with no data egress. Not because the cloud is bad: it stays valuable for work that is not regulated. It is because ownership of the running system is the thing that turns a demo into a deployment, and you cannot own what you can only reach through somebody else's endpoint.
What should you ask for instead of a demo?
Ask for two weeks on your data, in your environment, under your rules. Concretely: run it against a real corpus including the ugly parts nobody cleaned; run it as three genuine users with their existing entitlements and confirm each sees only what they should; take one consequential action and make it wait for a named approver; export the audit record and verify it with a tool the supplier did not write; name the internal owner and the exit position before anyone signs anything.
If a supplier cannot survive that, the demo was the product.
We are building for the organisations that have to answer those questions anyway: finance, law, defence, public sector, manufacturing, energy and professional services. SIOS has 63 studios, 14 production-ready at launch and 49 in development, with 50 specialised models behind them. Mickai LTD is a UK company (Companies House 17166618), held privately by me. MICKAI is a registered UK trade mark, UK00004373277, and we hold 104 filed UK patent applications carrying 2,340 claims, filed and not granted. The closed beta is open, with one regulated company onboarding as a design partner.
None of that makes a deployment easy. It makes the hard parts visible early, which is the only version of easy available here.
Frequently asked questions
Why do so many AI pilots stall before production?
Because a pilot tests capability and production tests the organisation. Real data is messy and carries legal obligations, user entitlements live in other systems, integrations are controlled changes, and someone internal has to own the result. A pilot that skipped those four is not nearly finished: it has not started the part that takes the time.
How long should a proper AI evaluation take?
Long enough to run on real data under real permissions, which usually means weeks rather than an afternoon. Two weeks on your own corpus, as three genuine users with their existing entitlements, tells you more than six polished demos. If a supplier needs your data cleaned before they can start, you have already learned something useful.
What is the Open Audit Record and why does it matter?
It is the evidence layer in SIOS. Every consequential action is sealed under ML-DSA-65, the post-quantum signature scheme published by NIST as FIPS 204 in 2024. The record is tamper-evident: alter it and verification fails. An auditor can export it and check it offline with a public key, using tools we did not write.
Does running AI on our own hardware really change the risk?
It changes who holds the risk. When the system runs on hardware you own, offline capable with no data egress, your data does not leave your estate and you control restarts, updates and exit. The cloud remains sensible for work that is not regulated. The difference matters most when a regulator asks where the data went.
What should be agreed before a pilot starts?
A named internal owner, the exact data in scope and its retention position, the entitlement model the system must inherit, which actions require a named human approver, how audit records are exported and verified independently, and what happens on exit if you walk away. Settling this before the pilot is far cheaper than settling it afterwards.
Written by Micky Irons, founder and chief executive of Mickai LTD, which builds a sovereign AI operating system for regulated organisations. More at mickai.co.uk.
Top comments (0)