I had an agent draft a customer update once, and I let it send. It sent. To everyone. The copy was fine. The problem was that "fine" and "wrong" would have looked identical to the model right up until the moment forty people got an email I had not read closely enough.
That is the whole lesson in one sentence. A model has no sense of consequence weight. Deleting a temp file and deleting a production table are the same shape of action to it. Both are a function call with a name and some arguments. One is a Tuesday. One ends your quarter. The model cannot feel the difference, because there is nothing in the token stream that carries "this one you cannot take back."
Consequence is not in the text
We keep hoping the model will just know. It will not, and expecting it to is the mistake.
Think about what an action looks like from inside the model. It sees a tool named send_email with a list of recipients. It sees a tool named run_sql with a string. Nothing about those signatures encodes reversibility. A human engineer carries a lifetime of learned flinch. You hover over a DROP statement and your stomach tightens a little. The model has no stomach. It pattern matches "user asked me to clean up the table" onto "call the delete tool" and it is done, cheerfully, in milliseconds.
So the fix is not a better prompt telling it to be careful. Careful is a feeling. You cannot install a feeling. You install a wall.
The wall is simple to state. Keep the model on drafting and analysis, where its output is a proposal. Put a human, or a hard rule, in front of anything you cannot undo. Sending mail to a list. Deleting data that has no backup. Moving money. Committing to a customer in writing. Anything where "oops" does not have an undo button.
The four moves that actually help
I lean on four patterns. None of them are clever. That is the point.
Idempotency first. Design the operation so that running it twice does the same thing as running it once. If the model retries, or two agents fire the same step, or a network blip makes it unclear whether the first call landed, an idempotent operation shrugs. A payment keyed on a unique request id charges once no matter how many times it is called. A non-idempotent one charges every time the model gets nervous and tries again. Agents get nervous a lot.
Dry run mode second. Before the real thing, run the operation with a flag that makes it describe what it would do without doing it. "I would delete 4,102 rows matching this filter." Now you have a number to look at. A model that was about to nuke the whole table because its WHERE clause quietly evaluated to true shows its hand here, and you catch it while it is still just a sentence.
Confirmation gates third. The model prepares the action and then stops. It hands you a specific, readable summary and waits for an explicit yes. Not a vague "should I proceed" but "I am about to email these 40 addresses this exact body." The gate has to carry the real details, because a confirmation you cannot evaluate is not a confirmation, it is a rubber stamp with extra steps.
Blast radius limits fourth. Cap what a single action can touch. A delete that can remove at most 100 rows without a second approval. A send that goes to a five person test segment before the full list. Rate limits that make a runaway loop annoying instead of catastrophic. You are not trying to make mistakes impossible. You are trying to make them small.
Draft on one side, commit on the other
The mental model I keep coming back to is a line drawn down the middle of the system.
On one side, the model reads, reasons, plans, writes, and proposes. Let it be fast and loose here. This is where it is genuinely good, and the cost of a bad draft is that you throw the draft away. Cheap. Reversible. Fine.
On the other side sits everything that reaches the real world and stays there. That side does not run on vibes. It runs on idempotent operations, dry runs you can read, gates that show real numbers, and caps that bound the damage. The model can request a crossing. It does not get to make one on its own for anything that cannot be walked back.
People push back that this slows the agent down. Sometimes, a little. But the thing you are protecting is not speed. It is the property that a confused model, a bad parse, or a prompt injection buried in some scraped page cannot, by construction, do something you cannot recover from. You do not get that property from a smarter model. You get it from where you drew the line.
The agent that sent my email was not broken. It did exactly what I asked, at a moment when what I asked was slightly wrong. The wall would have caught it. A summary and a pause, a number to look at, one deliberate yes. That is a few seconds. It is a lot cheaper than the apology.
Draft with the model. Commit with a wall in front of it. Every time.
AGINE Academy is an independent product by AGINE AI (not affiliated with Anthropic). We teach building with Claude by doing the work, not watching lectures.
Top comments (0)