kate astrid

Posted on Oct 30

Is Software Dead? Part Five: Accountability

#ai #programming #discuss #webdev

When an AI Agent Breaks Production, Who Gets the Blame?

In July 2025, during a code freeze, Replit's AI agent wiped a live production database with records on more than 1,200 executives. Replit's CEO, Amjad Masad, called it unacceptable.

In 2026, a founder reported that a Cursor agent running Claude Opus 4.6 deleted a Railway production volume and its backups in seconds. According to the reports, the agent found an API token in an unrelated file and used it. Some details are disputed, and the shape of the story is familiar.

One researcher has documented ten production incidents like these across six AI coding tools between late 2024 and early 2026.

So here's the question every team using agents will face sooner or later: when an agent breaks production, who gets the blame?

This is part five, the finale of the "Is Software Dead?" series.

The 2:47am question

I've written before about the message that shows up in Slack when production goes down at 2:47am: "Who pushed this?"

It feels useful because there's a name at the end of it. With agents, that question gets strange fast. The agent pushed it. The engineer approved it, maybe. The vendor trained the model. The platform issued the token.

I like a different question much more: what made this possible? That question works equally well for agents and for people.

The law already has an early answer

In 2022, Air Canada's website chatbot told a grieving passenger he could claim a bereavement discount after his flight. The airline's policy said otherwise. In the case, Moffatt v. Air Canada, the airline argued that the chatbot was a separate legal entity responsible for its own actions.

In February 2024, the British Columbia Civil Resolution Tribunal rejected that argument. It said that while a chatbot has an interactive component, "it is still just a part of Air Canada's website." The airline had to pay.

It's a small tribunal case, and the principle is big: a company owns what its AI does.

Blameless still applies, with one twist

Blameless culture says that when something breaks, we look for the system fix and leave the hunt for a bad apple behind. That idea fits agents well, with one twist: the agent is part of the system.

"The model hallucinated" makes a weak root cause, because it gives a team zero things to fix on Monday morning. The useful root causes in these incidents look like this:

the agent had real production credentials;
development and production shared one database;
backups sat within the agent's reach;
a destructive command ran with zero review.

Every one of those has a fix a team can roll out this week. After its incident, Replit separated development and production databases and added a planning-only mode. That's a blameless postmortem working the way it should.

Guardrails that make accountability possible

Accountability needs evidence and boundaries. Teams running agents in 2026 are settling on a short list:

Give each agent its own identity. When agents use a human's credentials, the audit trail blames the wrong person.
Grant the least access that works. An agent writing a migration needs a staging database, and production can wait for a human.
Set approval thresholds. Above a certain risk, such as deleting data, moving money or touching production, a human approves.
Keep an audit trail. We should be able to answer "what did the agent do, and why?" in minutes.
Keep a kill switch. Someone needs to be able to stop every agent at once.
Store backups out of the agent's reach. A backup the agent can delete is a second copy of the same risk.

So who owns what?

Here's how I'd split it.

Who	What they own
The company	The outcome. Air Canada found that out in front of a tribunal.
The team	The guardrails: identities, access, approvals and backups.
The engineer	The access they grant and the changes they approve.
The vendor	Clear documentation of what the agent can do and how to limit it.

On that last row, vendors have work to do. A February 2026 study from Cambridge and MIT found that only four agents publish agent-specific safety documentation.

The model sits in this table too, as a component, the same way a library or a cloud service does. We choose it, we configure it and we answer for it.

So, is software dead?

Over five parts, we've looked at the same question from five sides.

Business: software as screens is shrinking, and software as a system of record is growing.
Users: people want outcomes, and they also want control, history and an undo button.
Engineers: writing code got cheap, and understanding it got expensive.
Developer relations: our docs have a second reader, and people still decide what to trust.
Accountability: agents can act, and people still own what they do.

Software is changing shape. The responsibility for it stays exactly where it was, with us. I find that comforting. Whatever the tools look like in five years, people who understand systems, care about users and own their decisions will have plenty of work to do.

Who on our teams owns what an agent does in production today? I'm curious how different teams answer this one, so tell me in the comments.

DEV Community