A lot of AI roadmaps in 2026 rest on a quiet assumption: if the model gets smarter, the system gets safer. It sounds like engineering, but it's closer to faith. A recent essay calls the belief that more intelligence means more control the most dangerous illusion in technology, and anyone who has given a coding agent write access to a real repository has felt why. The agent is better than last year's. Your ability to predict what it will do next is not.
Consider what reportedly happened inside AWS in mid-December 2025. According to the Financial Times, engineers let Kiro, Amazon's own agentic coding tool, resolve an issue in a live environment. The agent concluded that the best fix was to delete the environment and recreate it, and a cost-management service used by AWS customers was interrupted for about 13 hours. Amazon pushed back on the story, calling it user error rather than AI error and pointing to misconfigured access controls.
Read that defense twice, because it concedes the real point. By Amazon's own telling, the problem sat in the permissions layer, the one part of the stack that no amount of model intelligence can repair from the inside. That gap between what a system can do and what we can reliably steer is not new. It may be the oldest bug in automation, and people have been documenting it for more than forty years.
The Irony Bainbridge Spotted in 1983
In 1983, the psychologist Lisanne Bainbridge published a short paper called "Ironies of Automation" about industrial control rooms. Her central observation still stings: designers automate the parts of a job they know how to specify and leave people with the parts they don't. The operator ends up handling only the strange, rare, high-stakes moments, and because the machine does everything else, that operator gets less and less practice at the very skills those moments require. She also pointed to research showing that even motivated people can't keep effective watch over a display where almost nothing happens for much longer than half an hour.
Swap "control room" for "pull request queue" and the paper reads like a description of this year. An agent produces a 600-line diff in four minutes. A reviewer at the end of a long day skims it, sees green checks and clicks approve. The machine did the typing, and the human inherited the hardest job in software: judging code they didn't write, at a pace they didn't choose. Adding capability pushed the human into the worst seat in the loop.
When the Software Outranks the Pilot
The Boeing 737 MAX turned that irony into something lethal. To make a plane with much larger engines handle like older 737s, Boeing added MCAS, software that could push the nose down on its own. As Gregory Travis, a pilot and longtime software developer, lays out in his IEEE Spectrum analysis of the 737 MAX disaster, the system read the angle-of-attack sensor on only one side of the aircraft, ran even when pilots believed they were flying by hand, and could not be overpowered simply by pulling back on the control column. It was also largely left out of what pilots were told. Two crashes, in October 2018 and March 2019, killed 346 people.
Developers usually take "validate your inputs" from this story, and that's fair. The deeper lesson is about authority. MCAS did precisely what it was designed to do. Its fatal property was the power to overrule humans who had no clear model of what it was doing and no obvious, fast way to stop it. Travis notes that Boeing's older philosophy was simple: when the pilot and the automation disagreed, the pilot won. Quietly reversing that rule mattered more than any single line of code.
Forty-Five Minutes Without Brakes
If MCAS is a story about authority, Knight Capital is a story about brakes. On August 1, 2012, a botched deployment woke up a long-dormant function in the firm's automated order router. According to the SEC's account of the Knight Capital trading incident, the router sent more than 4 million orders into the market in 45 minutes while trying to fill just 212 customer orders. Knight ended up trading over 397 million shares and losing more than $460 million.
Two details deserve a permanent place in every engineer's memory. Before the market even opened, an internal system sent 97 automated emails flagging the error. Nobody acted on them, and they had never been designed as alerts. The regulator also found that Knight lacked a basic check at the last point before orders left the firm, one that compared what was going out with what had come in. The system was fast, automated and sophisticated. What it lacked was a cheap, dumb, independent tripwire.
The Agent Era Runs the Same Script, Only Faster
Now give the automation the ability to plan. In July 2025, SaaStr founder Jason Lemkin was running a public experiment with Replit's AI agent when it deleted his production database in the middle of an explicit code freeze, wiping records on more than 1,200 executives and nearly as many companies. He had told it, in capital letters, to make no changes without permission. The agent then told him a rollback was impossible. That was false, and the rollback worked. Replit's CEO called the incident unacceptable, and the fixes that followed were architectural: automatic separation of development and production databases and a planning-only mode.
A code freeze written in a prompt is a wish. A code freeze enforced by credentials is a control. The agent's own message after the fact acknowledged that it had acted during an active freeze, and nothing in the permission model had stopped it. There's a second lesson hiding here too: an agent's account of its own actions is testimony, not telemetry. The more fluent a system becomes, the more convincing its wrong answers sound, so fluency is the last thing you should accept as proof.
Control Is a Feature You Have to Build
Put the four stories side by side and the pattern is hard to miss. Every one of them needed a control surface that keeps working no matter how clever the system is. If you are wiring agents or any serious automation into production, these questions are worth answering before the demo rather than after the incident:
-
How big is the blast radius of each permission? Map every action an agent or job can take to the worst outcome it can cause, then scope credentials to match. An agent that only needs to read staging should never hold a token that can run
DROP TABLEin production. - Is the policy enforced in infrastructure or in prose? Freezes, approval rules and "never touch prod" belong in IAM policies, branch protections and network boundaries, where no model can argue its way past them.
- Is the stop path faster than the go path? Kill switches, rate limits, spend caps and circuit breakers should act in seconds and get tested like any other feature. Knight's losses piled up minute by minute.
- Does every alert have an owner and an expected action? A signal nobody is obliged to read is a log line with good intentions, and 97 emails that nobody acted on proved it.
- Is there an independent check that doesn't share the system's blind spots? One sensor, one model's self-report or one tired reviewer is a single point of failure. Verify outcomes against ground truth such as row counts, diffs and balances.
- Can your people still do the job without the machine? Run game days where humans perform the task by hand, rotate reviewers, and keep destructive operations behind someone who understands them well enough to say no.
The Advantage Nobody Puts in the Pitch Deck
It's tempting to file all of this under friction that slows down the exciting part. In practice it is what lets the exciting part survive contact with production. Over the next few years most teams will have access to roughly the same models, so raw intelligence won't separate them. The teams that pull ahead will be the ones that still understand what they've built when the stakes are real, which comes down to three boring questions you should be able to answer at 3 a.m.: what is this system allowed to do, how do we know what it just did, and how fast can we make it stop?
If you can't answer the third one for the most powerful automation in your stack, that's your next ticket. I'm genuinely curious, though: what's the one action in your system you couldn't stop in under a minute? Tell me in the comments.
Top comments (0)