The most important test for an autonomous AI system may not be whether it can complete a task. It may be whether people can stop it after it completes the wrong one.
This matters because autonomy changes the shape of failure. A chatbot can produce a bad answer. An agent can turn a bad answer into a sequence of actions: call a tool, update a record, send a message, schedule a follow-up, and trigger another system before anyone notices.
In August 2026, the US Food and Drug Administration opened a public discussion about generative AI-enabled medical devices. Its paper asks how autonomous multi-step action, tool use, and reduced opportunities for human review should affect acceptance criteria and oversight. It also proposes competency testing inspired, at a high level, by how clinicians are evaluated.
That is a useful direction, but the licensing analogy has a missing half.
A professional license does not only certify competence. It also creates a mechanism for investigation, suspension, remediation, and revocation. An autonomous system needs an operational equivalent.
Call it the revocation test.
1. Can you reconstruct one action from source to consequence?
Pick a real or synthetic action the agent took. Ask the team to produce:
- the input that initiated it
- the model, prompt, policy, and tool versions involved
- every tool call and returned value
- each human-oversight checkpoint
- the records or people affected downstream
An activity log that says only "task completed" is not an audit trail. The useful unit is the entire action chain.
This applies well beyond healthcare. A hiring agent may screen a candidate, update an applicant record, and send a rejection. A finance agent may classify a payment, move it into a queue, and notify a vendor. A logistics agent may reroute an order and create a customer promise. In each case, reconstructing the final output is not enough. You need the path that made the output consequential.
2. Is there a stop threshold written before deployment?
"We monitor it" is not a stop rule.
A stop rule names an observable condition and the action that follows. Examples include:
- two unauthorized tool calls in a rolling hour
- one action outside the approved scope
- a failed human-approval checkpoint before an irreversible step
- an unexplained jump in escalation or refusal behavior
- a mismatch between the action log and the system of record
The threshold should exist before the incident. If a team invents it afterward, the threshold can quietly move to protect the deployment.
3. Does a named person have authority to hit pause?
A system can have excellent monitoring and still be unsafe if nobody owns the decision to stop it.
Name the role, the backup role, and the maximum time allowed to act. Then test whether that person can actually pause new work, cancel queued work, and prevent downstream systems from continuing the chain.
The owner must have both responsibility and control. Giving someone accountability without a working breaker is theater.
4. Can the team contain actions already in flight?
Stopping new actions is only the beginning. Autonomous systems create partial state.
A useful containment plan answers:
- Which queues still contain work?
- Which messages have already been sent?
- Which external systems accepted a change?
- Which actions can be reversed automatically?
- Which affected people need a human follow-up?
The more tools an agent can call, the more important this becomes. A kill switch that leaves the last ten minutes of actions propagating is not a complete kill switch.
5. What evidence is required before restoration?
Teams often spend more time defining deployment than defining restoration.
Before an agent is turned back on, require a replay of the failure case, evidence that the stop threshold now behaves correctly, review of adjacent cases, and a written decision from the restoration owner. If the system changed, preserve the old and new versions in the record.
Restoration should be a new decision, not the default result of elapsed time.
Competence gets an agent into production. Revocability determines whether it belongs there.
Benchmarks help establish what a system can do under test conditions. Post-deployment monitoring helps show what it is doing in the world. The revocation test connects those phases to a real control surface.
For any autonomous system, ask five questions before launch:
- Can we reconstruct the action chain?
- What exact threshold stops it?
- Who can trigger that stop?
- How do we contain and repair actions already in flight?
- What evidence is required before restoration?
If any answer is vague, the system may be autonomous in capability but ownerless in failure.
The FDA discussion paper is open for public feedback through October 19, 2026: https://www.fda.gov/media/194242/download
Krasyn publishes practical tests for high-stakes AI, workflow handoffs, and recovery. The aim is simple: turn broad claims into checks a real team can run.
Top comments (0)