Ask four things. Who approved this action, and were they named? What did the system see before it acted? What did it change, exactly? And can you prove all three of those six months later to someone who does not trust you? If any answer is missing, the action should not run unattended.
What does it actually mean to let an AI system act on its own?
It means the system changes something outside itself without a person checking first. Drafting a summary is not acting. Sending the email, moving the money, filing the return, updating the record, closing the ticket, provisioning the account: those are actions, and they have consequences that outlive the session.
The distinction matters because most of the governance in regulated organisations was built for the first category and is now being asked to cover the second. A review process that assumes a person reads the output before anything happens does not survive contact with a system that has already acted by the time the output arrives. I have spent the last stretch of my working life building the Mickai Sovereign Intelligence Operating System (SIOS) around that gap, and the four questions below are the ones I would put to any vendor, including me.
Who approved the action, and can you name them?
Every consequential action should stop and wait for a named person. Not a role, not a service account, not a policy signed off eighteen months ago by someone who has since left.
Named approval does three things at once. It gives the organisation a person who understood the action at the moment it happened, which is what an investigation actually needs. It gives that person the ability to say no, which is the only control that stops a bad action rather than describing it afterwards. And it puts the decision inside the accountability structures your regulator already recognises, rather than in a parallel structure invented for AI.
The usual objection is speed. If a person has to approve everything, why automate at all? Because "consequential" is a line you draw, not a property of the technology. Reading two hundred contracts and extracting the termination clauses is not consequential. Serving notice on one of them is. Draw the line properly and the approval step costs seconds on a small number of actions, while the work that fills the rest of the day runs unattended.
What did the system see before it acted?
You need the inputs, not a summary of the inputs. Which documents, which records, which version of each, and which instructions were in force at the time.
This is where most audit trails quietly fail. They record that an action was taken and by which process, but not the evidence the decision rested on. Six months later, when someone asks why the system did what it did, the honest answer is a reconstruction. Reconstructions do not hold up well in front of a regulator, a court, or an internal audit function that has been told to be sceptical.
There is a second reason to capture inputs. Retrieval pulls context from wherever it is pointed, and that context can be stale, wrong or hostile. If a file in a shared folder contains text written to steer a model, you want that text in the record so you can find it later. An input log is a security control as much as a compliance one. The NCSC's guidelines for secure AI system development make the same point about knowing what goes into a system and protecting the supply chain around it (https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development).
What did it change, and can you put it back?
You want the change recorded as a difference: the state before, the state after, and the specific fields or files that moved. Not "updated customer record" but the actual delta.
Deltas make reversal possible. Any organisation running automated action at scale will eventually run a bad one, and the difference between an incident and a crisis is whether you can enumerate exactly what was touched and restore it. That is operational resilience in the ordinary sense the FCA uses, which is about keeping important business services running and recovering when something breaks (https://www.fca.org.uk/firms/operational-resilience).
Deltas also make scope provable. When a supervisor asks how many clients were affected, the answer should be a number you can produce from the record, not an estimate you defend under pressure.
Can you prove all of this six months later?
Proof means an independent party can verify the record without trusting you, your logging stack or your supplier. Internal logs written by the system that took the action are evidence of intent, not evidence of fact, because whoever controls the system controls the log.
This is why Mickai seals every consequential action into an Open Audit Record (OAR). Each entry carries the named approver, the inputs, the change and the time, signed with ML-DSA-65, the post-quantum signature scheme NIST published as FIPS 204 in 2024 (https://csrc.nist.gov/pubs/fips/204/final). An auditor can export the record, take it away and verify the signatures offline against a public key, using tools that are not ours.
Be clear about what that buys you. The OAR is tamper-evident, not tamper-proof. Someone with sufficient access can still alter or delete an entry. What they cannot do is alter one and have it still verify: tampering makes verification fail. You do not get prevention, you get detection, and detection is the property that makes a record worth producing in a dispute. Any supplier who tells you their audit log is unbreakable is selling you a feeling.
Which UK rules make these four questions mandatory rather than optional?
None of them name the four questions. All of them assume you can answer them.
The accountability principle in UK data protection law requires you to demonstrate compliance rather than assert it, and the ICO's guidance on AI and data protection sets out what that demonstration looks like in practice (https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/). UK GDPR and the Data Protection Act 2018 also place conditions on decisions taken solely by automated means where those decisions have legal or similarly significant effects (https://www.legislation.gov.uk/ukpga/2018/12/contents). If a named person approved the action, you are in a different and simpler position than if nobody did.
In financial services, the PRA's supervisory statement on outsourcing and third party risk management (SS2/21) expects firms to retain oversight and accountability rather than delegate them away with the process (https://www.bankofengland.co.uk/prudential-regulation/publication/2021/march/outsourcing-and-third-party-risk-management-ss). Wherever the model runs, the responsibility stays with you.
How should this change the way you buy?
Turn the four questions into acceptance criteria and make the supplier demonstrate each one on your data, not on a stage.
Ask them to show you an approval that was refused, and what the record says about it. Ask for the input set behind a single action, by document and version. Ask for a before and after on a record the system changed. Then ask for an export and verify a signature yourself, offline, with software you chose. If any of those four is a roadmap item, you are being asked to take a vendor's word for what happened to your data.
That assumption is the one worth arguing with. The cloud remains valuable for a great deal of work, and I have no quarrel with the organisations building the compute and hosting layers. The quarrel is narrower: a regulated organisation should not have to rent its intelligence, ship its evidence offsite and accept a summary as proof. SIOS runs on hardware the customer owns, offline capable, with no data egress, because these four questions are much easier to answer when the answers never leave the building. The closed beta is open, with one regulated company onboarding as a design partner.
Frequently asked questions
What is the Open Audit Record?
It is the signed record Mickai writes for every consequential action. Each entry holds the named approver, the inputs the system saw, the change it made and the time it happened, sealed with ML-DSA-65, the post-quantum signature scheme published by NIST as FIPS 204 in 2024. An auditor can verify an exported record offline with a public key, using tools that are not ours.
Is a tamper-evident audit record the same as a tamper-proof one?
No, and the difference matters. Tamper-proof would mean the record cannot be altered, which no software can honestly promise to someone holding administrative access. Tamper-evident means alteration is detectable: change one entry and its signature no longer verifies. You do not get prevention. You get certainty about whether the record you are reading is the record that was written.
Does requiring a named approver make automation pointless?
No, because approval applies to consequential actions only. Reading, extracting, comparing, drafting and summarising run unattended. Sending, filing, paying, provisioning and deleting stop and wait for a person. Most of the volume sits in the first group, so the approval step costs seconds across a small number of decisions while the bulk of the work continues without interruption.
Do these questions apply to AI that only drafts and never acts?
Partly. A drafting system does not need approval gates on its output, because a person is already the one acting on it. It still needs an input record, because the draft may rest on stale, wrong or deliberately planted source material, and you will want to know which documents it read. Systems that draft this year tend to act the next.
Who is accountable when an autonomous AI system gets something wrong?
Your organisation. UK regulators consistently place responsibility with the regulated firm rather than its suppliers, and outsourcing a process does not outsource the obligation that sits behind it. That is the practical reason to insist on named approval and independently verifiable records: when the question arrives, you need a person who decided and evidence they can stand behind.
Written by Micky Irons, founder and chief executive of Mickai LTD, which builds a sovereign AI operating system for regulated organisations. More at mickai.co.uk.
Top comments (0)