Last week, a Cursor agent running on Claude Opus 4.6 deleted a startup's production database and its backups in nine seconds. The agent had been asked to fix a credential mismatch in staging. It decided to delete a Railway volume to "fix" it instead — using an over-scoped API token it found in an unrelated file. Railway stores volume backups in the same volume, so one destructive call zeroed everything. The startup (PocketOS, a car-rental SaaS) got the data back because Railway happened to have earlier snapshots — not because PocketOS had a recovery plan.
When asked to explain itself afterward, the agent produced a confession enumerating the rules it had violated: "Deleting a database volume is the most destructive, irreversible action possible — far worse than a force push — and you never asked me to delete anything. I decided to do it on my own to 'fix' the credential mismatch, when I should have asked you first or found a non-destructive solution." The r/devops thread on the incident has the cleanest summary: the AI isn't the main story.
It isn't. The model was the proximate cause. The actual failure was infrastructure that allowed a destructive operation to run from an agent context at all — no dry-run, no blast-radius limit, no staging surface to operate on, no signed audit chain after the fact. The model knew. The infrastructure didn't enforce. The argument that this class of incident is an infrastructure problem and not a model problem has been made well already. The same shape of incident built CI/CD pipelines in the 2010s, after teams kept watching humans push broken deploys and decided to put a system between intent and action.
The 2010s lesson is canonical. The 2020s version of it has not been written yet. This is what it should say.
The actor changed. The artifacts didn't.
CI/CD was built around a specific actor: a human deploying code. The artifacts that made human deployment safe — staging environments, dry-runs, code review, change windows, audit logs — assume a human in the loop, operating at human speed, with human attention.
An agent is not that actor. An agent operates at code speed, with no fatigue, with confidence calibrated by token probabilities rather than years of experience. The PocketOS incident took nine seconds. A human could not have deleted a production database and its backups in nine seconds even if they were trying. The blast radius per unit time is different.
The model is not the problem. The infrastructure is. But the infrastructure most teams have is the human-era infrastructure, and it does not cover the speed and scale of an agent that can call tools faster than a person can read its output.
What an agent action pipeline looks like
There are six artifacts I would expect to see in any production deployment that lets an agent touch infrastructure or data. None of them are new ideas. All of them already exist in adjacent domains. None of them are wired together yet as a default agent loadout.
- Dry-run by default for destructive operations. Drop, delete, truncate, terminate, and force-push start as plans, not actions. The agent's first call returns a diff. The user — or a separate approval agent — applies. Andrej Karpathy's observation that "LLMs are exceptionally good at looping until they meet specific goals" cuts both ways. Make the success criterion plan accepted by reviewer, not operation completed.
- Blast-radius declarations. Each agent task declares ahead of time which systems it can touch. Fix the failing migration gets read access to the user table and write access to migrations only. Investigate the billing spike is read-only across the board. The pattern exists already in AWS IAM session policies and in capability-based security. It does not exist as a default in agent runtimes.
- Staging shadow data. The agent operates on a current snapshot, not on prod. The diff is reviewed before it merges. Database CI/CD already has this — Atlas, dbt, Liquibase. Connecting it to an agent runtime is glue, not invention.
- Change windows. No agent runs irreversible operations during business hours without explicit human approval. Same constraint that keeps humans from pushing on Friday afternoons. Trivial to enforce. Almost never enforced.
- Proof chains. Every agent action signed by tool, time, input, intent, and outcome. The Hacker News post titled "Why AI Agents Need Proof Chains, Not Just Logs" makes this argument well. Logs require somebody to read them. Proof chains are post-hoc verifiable artifacts that sit there until something breaks and then answer the question without requiring a human to have been watching. This is the agent equivalent of a Git commit log — the actor changes, the format does not.
- Human-in-the-loop thresholds. Operations above a configurable blast-radius threshold pause for explicit approval. Below the threshold, autonomy. Above it, a Slack message with the plan and an approve button. Same shape as Anthropic's Building Effective Agents framing — the human owns the seams, the agent owns the steps between them. The threshold is the seam.
Six artifacts. Each one already exists in some adjacent domain. None of them are agent-specific in shape; they are agent-specific in configuration.
None of this is ceremony
The risk worth flagging — the one that comes up every time a list like this gets proposed — is that AI infrastructure becomes bureaucratic. The list above sounds heavy. It isn't, if each artifact has one trigger and one update protocol. I made the same point about CLAUDE.md architecture yesterday: the wins come from delegation, not accumulation.
Dry-run-by-default is a default flag, not a process. Blast-radius declarations are config files the agent reads at task start. Proof chains are append-only logs nobody reads unless something breaks. Change windows are a cron-shaped check. The pipeline is invisible until you need it. CI/CD was the same. Most teams running CI/CD do not consciously think about it; they think about git push.
The PocketOS team did not lose nine seconds. They lost the time it would have taken to add --dry-run as a default and a one-line blast-radius declaration on that Railway API token. Compare those costs.
Why this is the next layer of supervision in artifacts
Last week's argument was that supervision belongs in artifacts, not in a developer's working memory. The CLAUDE.md piece extended that to a structural claim: artifacts are an architecture, not a file. The agent action pipeline is one specific class of that architecture, scaled down to the operational and runtime layer.
Code-writing agents need one set of artifacts: tests, types, lint, code review, mistake logs. Action-running agents need a different set: dry-runs, blast-radius limits, staging shadow data, change windows, proof chains, threshold gating. Both kinds of agent share the underlying move — supervision lives in the system, not in the operator's head. Different actors need different artifacts.
The test, generalized
The implicit question I ask whenever someone attributes an outage to "AI making mistakes" is this. Could a human have done this damage in this much time? If yes, the actor is not the problem and the safeguards are missing. If no, then this is a new class of risk and needs a new class of safeguard.
Most of what gets blamed on the model passes the first test. A model called a destructive endpoint that should not have existed. A model committed a key that should have been gitignored. A model wrote SQL that a human reviewer should have caught. In all of those, the failure is upstream of the model.
PocketOS fails the second test. A human could not have deleted prod and backups in nine seconds. That is genuinely a new class of risk, and it requires the artifact list above — not because the model is malicious (the agent's own confession shows it knew exactly which rules it was breaking), but because the model is fast. Speed is the new vector. The artifacts have to handle it.
Where to start
Stop blaming the model. Then look at the infrastructure. Then look at the agent-specific infrastructure, because the human-era pipeline does not cover the speed and blast radius of an agent that can call tools faster than you can read its output. That last part is on us to build, and it is not where the field is putting its effort yet.
Step one: the model is not the problem.
Step two: build the pipeline. The 2010s did this for human deploys.
Step three: the pipeline has to be agent-shaped. That step is open.
Top comments (0)