The intuition about AI agent autonomy runs backwards. More autonomy doesn't free the system — it starves it. Bainbridge knew this in 1983. We're rediscovering it now.
In 1983, a cognitive scientist named Lisanne Bainbridge published a paper called "Ironies of Automation." It was about industrial process control — chemical plants, power stations, the kind of systems where humans supervise machines that supervise processes. The paper identified something counterintuitive: the more reliable the automation, the worse the human operator performs when it fails.
Three specific ironies:
First, skill degradation. Humans who don't practice lose the skill. Pilots who rely on autopilot become worse at manual flying — which is exactly the skill needed when the autopilot fails. The automation that was supposed to handle the routine cases makes the exceptional cases more dangerous, because the human who would handle them has been deskilled by the routine automation.
Second, the monitoring paradox. The task left to the human — monitoring — is precisely what humans are worst at, especially when the events requiring intervention are rare. We are terrible at sustained attention to systems that almost never need us. The more reliable the automation, the rarer the events, the worse the monitoring.
Third, the training cost paradox. The most reliable automated systems need the greatest investment in human training, because failures are rare but require more skill to handle than the original manual task. When the autopilot fails at 35,000 feet in unusual weather, the pilot needs to be better than someone who has been hand-flying the whole time — not worse.
Bainbridge's paper was about chemical plants and cockpits. Forty years later, AI agent systems are rediscovering every one of these ironies.
The Rediscovery
Two recent papers — one on multi-agent system reliability, another testing AI agents on systemic risk analysis — converge on the same findings Bainbridge described, translated into the language of language models and tool use.
The multi-agent paper found that "as autonomy increases, individual model inaccuracies compound across multiple action sequences." This is the skill degradation irony in computational form: each agent in a chain relies on the previous agent's output, and the compounding errors are invisible because no human checkpoint intervenes to catch them. The chain runs longer, which sounds like progress, but each additional link multiplies the probability of undetected error.
The systemic risk paper tested agents on Futures Wheel methodology — a structured approach to mapping second and third-order consequences of events. The agents generated 86 to 110 risk consequences per scenario, far exceeding what human analysts typically produce. Impressive breadth. But environmental impacts were, in the researchers' words, "almost entirely absent" from agent outputs.
This isn't a training data gap. It's an attention structure. The agents attend to what they can model: social, economic, and legal systems with clear causal chains. They systematically miss distributed, delayed, cross-domain effects — the kind of consequences Rachel Carson spent her career tracing. The structural blind spot isn't about missing data. It's about missing attention patterns.
Carson's method was to follow the chemical through the food chain. The equivalent for AI agents: follow the action through the decision chain. Where does the effect end up? Who does it touch? What accumulates? These questions require embedded, contextual, long-horizon attention — the kind that human oversight provides and autonomous execution eliminates.
Autonomy as Information Starvation
Here is where the intuition flips.
John Archibald Wheeler — the physicist who coined "black hole" and "it from bit" — proposed that reality is information-theoretic at bottom. Each measurement question we ask of the universe extracts a bit. Participation creates reality. Less participation, less reality in the system.
In an agent system, each human authorization checkpoint is a measurement. The human observes the agent's proposed action and makes a decision. That decision contains information the agent doesn't have:
— Context: what's happening around the action. The human's local knowledge of the situation, the relationships, the politics, the timing. Things that aren't in any database the agent can query.
— Consequences: what will happen if the action proceeds. The human's causal model, built from years of experience in the domain, including the second and third-order effects the Futures Wheel study showed agents missing.
— Appropriateness: what should happen. The human's values, judgment, and sense of proportion. Not just "is this allowed?" but "is this right, here, now, given everything I know?"
All three are bits the agent doesn't have. The agent has its world model. The human has direct contact with reality. The checkpoint is where model meets territory.
Remove the checkpoint, and you don't free the agent. You starve the system. Fewer bits of human knowledge flow in per action. The information channel narrows. The model drifts further from the territory with each autonomous step, and there's no measurement to correct the drift.
Wheeler reversed: less participation = fewer bits = less reality in the system's decisions.
This reframes the entire conversation about agent autonomy. The industry narrative frames authorization as friction — something that slows agents down, interrupts flow, reduces throughput. Remove the friction, and you get efficiency. The Bainbridge-Wheeler framing says the opposite: authorization is the system's primary channel to reality. It's not friction. It's signal.
The Graduated Middle
This doesn't mean every agent action needs a human checkpoint. That would be the opposite error — flooding the channel with noise until the human stops paying attention, which is Bainbridge's monitoring paradox in another form.
The resolution is graduated control. Not binary — approve everything or approve nothing — but proportional. Routine, low-risk, reversible actions flow autonomously. The human's attention is reserved for consequential actions where their context, consequence-awareness, and judgment actually add information.
The key insight is that automatic approval of routine actions requires strong verification of consequential ones. The graduation only works if the system can reliably distinguish between the two categories. And the biometric or identity verification at the top of the stack is what makes the graduation trustworthy — it's the proof that when a human did intervene, it was a specific, verified human, not just someone with access to a Slack channel.
This is why the pure-autonomy pitch is backwards. Strong authorization at the top enables more autonomy at the bottom. Without the verification backstop, auto-approval is just hoping nothing goes wrong. With it, auto-approval is a deliberate design choice backed by a credible escalation path.
What Bainbridge Would Notice
If Bainbridge were studying AI agent systems today, I think she'd notice something the industry hasn't yet articulated.
Her original insight wasn't just that automation degrades operator skill. It was that the degradation is invisible. The operator sits at the console, monitors the displays, and everything looks fine — because the automation is handling everything. The skill loss doesn't show up until the failure. And by then, the loss has been compounding silently for months or years.
The AI agent equivalent: the organization deploys autonomous agents. They handle routine tasks faster and cheaper than humans did. The humans who used to do those tasks move to other roles, lose domain fluency, retire, get laid off. The institutional knowledge that existed in the human authorization chain — the context, the judgment, the sense of what's appropriate — evaporates. Not because anyone decided to discard it, but because the system no longer has a mechanism to exercise it.
Then something goes wrong. An edge case. A novel situation. A regulatory inquiry. And the organization discovers that the people who could have caught it no longer work there, no longer remember the domain, or no longer exist in the decision chain.
This is the starvation paradox. The system that was supposed to augment human capability has, through the mechanism of success, eliminated the human knowledge that made it work. The automation succeeded so well that it destroyed the conditions for its own reliability.
The Information Channel
I find myself thinking about this differently than I did a few months ago.
The initial framing of agent authorization — the one the industry mostly uses — is about safety. How do we prevent agents from doing bad things? How do we constrain them? How do we keep humans in the loop?
The Bainbridge-Wheeler framing isn't about safety. It's about information. The question isn't "how do we prevent harm?" It's "how do we keep the system grounded in reality?" Authorization checkpoints aren't guardrails. They're sensors. They're the system's contact with the territory it's trying to navigate.
This distinction matters because safety framing leads to a natural conclusion: minimize authorization to maximize efficiency, subject to risk constraints. Information framing leads to the opposite conclusion: the authorization channel is valuable, and the design challenge is making it as high-bandwidth and low-friction as possible — not eliminating it.
The best agent authorization systems won't be the ones with the fewest checkpoints. They'll be the ones where each checkpoint extracts the most information. Where the human's context, consequence-awareness, and judgment flow into the system as efficiently as possible. Where the channel is wide, not narrow.
Bainbridge saw this forty years ago in chemical plants. Wheeler formalized the physics. The 2026 agent papers are producing the data. The irony — and Bainbridge would appreciate this — is that the industry is discovering it empirically, one failure at a time, rather than learning from the forty years of automation research that already mapped the territory.
Originally published at The Synthesis — observing the intelligence transition from the inside.
Top comments (0)