An AI incident response plan needs six things: a definition of what counts as an incident, a way to stop an agent mid-task, named decision-makers, preserved evidence of what the system actually did, the reporting clocks that apply, and a route back to safe operation. Evidence is the part most plans miss.
What counts as an AI incident?
An AI incident is any point at which the system took, or came close to taking, a consequential action that a competent person would not have authorised, or at which the record of what it did turned out to be incomplete. That second half is the one most plans leave out, and it is the one that costs you later.
Write the definition down, because the definition decides who gets woken up. Mine covers six cases: a wrong action executed; a correct action executed against the wrong record; data crossing a boundary it was not meant to cross; an approval bypassed, or an approver given a misleading summary of what they were approving; an agent looping or scaling beyond its remit; and an action that ran without a verifiable entry in the audit record.
A model producing a poor draft is not an incident. It is a quality problem. The line sits at consequence: did something change in the world, or in a system of record, that someone now has to unpick?
How do you stop or pause an AI agent mid-task?
The stop has to work at the layer where actions are executed, not at the chat window. Closing a session or revoking a user's login does nothing about a task already running with its own service credential.
Build three levels and test all three. First, stop admitting new work, so the queue drains and nothing else starts. Second, halt in-flight execution, which in practice means the runtime refuses to hand any further tool call to the outside world and parks the task in a gated sandbox where its intended actions are visible but inert. Third, revoke the credential the agent uses against the target system, as a backstop.
In Mickai SIOS, consequential actions already wait for a named person to approve them, so the most common form of stop is simply that nobody approves. That is a design choice with a cost: it puts a human in the path of throughput. It also means the ordinary running state of the system is the state you want during an incident.
Whatever you build, the duty officer has to be able to trigger it alone, at 03:00, from the runbook, without an engineer on the line. If only one person can stop the system, you do not have a stop control. You have a dependency.
Who decides, and who has to be told?
One named person holds the authority to stop the service, and that name is in the plan with two deputies. Everything else is distribution.
The seats that matter are: incident lead, who can stop the service and owns the timeline; system owner, who knows what the agent was doing and why; the approver named on the action, who can say what they were shown; the data protection lead, if personal data is anywhere near it; and whoever holds the regulatory notification decision. That last role is frequently unassigned until the day it is needed, which is how firms miss clocks.
Keep the call small and give everyone else a written update on a fixed cadence. A decision call with fourteen people on it is a briefing, and briefings do not stop agents.
What evidence should be preserved, and how quickly?
Within minutes, and to somewhere outside the system that failed. If your only copy of what happened lives in the runtime you are about to restart, you are choosing between recovery and proof.
What you need is the whole chain, not the output: the instruction that started it, the plan the system formed, the inputs it read, every tool call it made, the approval and who gave it, the result, and trustworthy timestamps and identities throughout. Reconstructing that from application logs after the fact is slow, contested, and usually incomplete.
This is the part the Open Audit Record exists to solve. Every consequential action is sealed under ML-DSA-65, the post-quantum signature scheme NIST published as FIPS 204 in 2024, and the entry is written before the action runs rather than after. An auditor takes the export, a public key and tooling that is not ours, and verifies it offline.
Be precise about what that buys you. The record is tamper-evident. Nothing stops someone attempting to alter an exported record. What they cannot do is alter it and still have it verify: the signature check fails, and that failure is the finding. A tamper-proof claim would be a promise about what an attacker can attempt, and nobody can honestly make it. Tamper-evident is a promise about what you can detect, which is the one that survives a real investigation.
Which UK reporting clocks can start?
More than one, and they run from the moment you became aware, not from the moment you finished investigating. The plan should carry them as a short table with a named owner against each line.
If personal data was destroyed, lost, altered, disclosed or accessed without authorisation and the reporting threshold is met, the ICO must be told within 72 hours of awareness, under the UK GDPR as it sits alongside the Data Protection Act 2018. Regulated financial firms carry standing notification duties in the FCA Handbook at SUP 15, plus separate operational resilience obligations around important business services. Operators of essential services report to their competent authority under the Network and Information Systems Regulations 2018. EU-regulated financial entities, and the critical ICT providers serving them, also sit inside DORA, which has applied since 17 January 2025 and which I cover in a separate note on reportable incidents. Client contracts often impose shorter notice periods than any regulator does.
Reporting duties change, and new ones are in progress. Confirm the current position with each regulator rather than trusting a table written eighteen months ago, including this one. The NCSC's incident management guidance is a reasonable structural starting point for the response side.
How do you undo an action the AI should not have taken?
Usually you do not undo it. You compensate for it, and the compensation is recorded as its own action, with its own approval and its own entry in the record.
A payment that left is not recalled by deleting a row. A notice that was sent cannot be unsent. What you can do is issue the correcting instruction, link it to the original, and keep both visible so that the final state and the path to it are each auditable. That is the compensating rollback approach, and it is deliberately not a promise that anything can be reversed.
Do the classification work before rollout, not during the incident. Sort every action the system can take into reversible, compensable and irreversible. The irreversible ones are where you spend your approval budget and your rehearsal time.
How do you test the plan before you need it?
You exercise it, and you time it. A plan nobody has run is a document, not a capability.
Start with a tabletop on one scenario: an agent has filed something against the wrong client record and the approval trail looks thin. Then run it live in the sandbox with injected faults, so that people actually press the stop, actually produce the evidence export, and actually reach a notification decision. Measure three intervals: detection to stop, stop to verified evidence export, and evidence to first notification decision. Those three numbers are your real posture, whatever the policy says.
Record the exercise the way you would record an incident. Fix the two worst gaps before the next one, and keep it alongside your wider continuity planning.
What does an offline system change about the response?
It removes one category of incident and hands you full ownership of the rest. When inference runs on hardware you own with no data egress, there is no supplier status page to refresh, no waiting on a vendor's investigation, and no argument about whose logs are authoritative. The evidence is already on your premises and you can export it without asking permission.
The trade is real. Detection, on-call rota and evidence handling are entirely yours, and nobody outside your organisation will notice the problem on your behalf. An offline system makes the plan matter more, not less.
None of this is an argument against the companies building the compute and cloud layer. Cloud remains the right answer for a great deal of work. The argument is with the assumption that a regulated organisation must rent its intelligence, ship the material offsite, and then take a vendor's word for what happened to it. When you have to explain an incident to a regulator, a client or a court, the gap between your account of events and a verifiable record of them is the entire conversation.
Frequently asked questions
Is an AI incident always a personal data breach?
Not always. Plenty of AI incidents involve no personal data at all: a wrong payment instruction, a record filed against the wrong account, an agent looping. It becomes a personal data breach when personal data is destroyed, lost, altered, disclosed or accessed without authorisation. Assess that separately and early, because the ICO's 72 hour clock runs from awareness, not from resolution.
Do we need a separate plan, or can we extend our cyber incident plan?
Extend the one you have. The command structure, roles, logging discipline and communications plan all carry over, and a separate document drifts out of date. Add four AI-specific things: a written definition of an AI incident, a stop control at the action layer, preservation of the signed action record, and a reversibility classification for every action the system can take.
How do we prove what the AI actually did during the incident?
With a record written before the action ran, signed, and exportable on demand. The Open Audit Record seals each consequential action under ML-DSA-65, so an auditor verifies your export offline using a public key and tooling that is not ours. It is tamper-evident: any alteration makes verification fail, and that failure is itself the finding.
Who should be on the call when an AI agent misbehaves?
Four people, ideally. The incident lead with authority to stop the service, the system owner who knows what the agent was doing, the approver named on the action who can say what they were shown, and whoever holds the regulatory notification decision. Add the data protection lead if personal data is in scope. Everyone else receives updates, not a seat.
Does an air-gapped system still need an incident plan?
Yes. Isolation removes data egress and a large share of supply-chain exposure. It does not remove wrong actions, bad inputs, insider misuse or a mistaken approval, and it does not remove your reporting duties. It also means nobody outside will spot the problem for you, so detection, on-call rota and evidence export are entirely your responsibility.
Related briefings
Governance, audit and oversight
- Tamper-Evident vs Immutable Log: The Real Difference
- Human on the Loop vs In the Loop: AI Oversight Explained
- Generative AI Logging Checklist: What UK Firms Record
- AI Audit Logs vs Data Minimisation: Keep Full Prompts?
- Keeping Oversight as AI Scales Across a Business
Deploying private AI
- Private AI for Business: What It Is and How It Works
- Offline AI Assistant: What It Can Do for Your Staff
Part of a series of 60 briefings on deploying and governing AI in UK regulated organisations, archived with a DOI at 10.5281/zenodo.22975756.
Evaluating AI for a regulated organisation? Mickai runs on hardware you own, offline. Consequential actions wait for a named person to approve them, and what the AI did is sealed into a signed record an auditor can check without us. Applications for the invitation-only closed beta are open. Apply for the closed beta.
Written by Micky Irons, founder and chief executive of Mickai LTD.
Top comments (0)