Twenty minutes into a conversation about what an agent should be allowed to touch, when a non-expert asks: can’t we just give it Asimov’s laws?
It’s always said in good faith. It’s also the most confidently wrong thing said in that conversation, and the reason is funny enough to be worth the whole post: the Three Laws come from a body of fiction that is, front to back, cover to cover, about the Three Laws not working.
Science fiction handed us the two intuitive answers to machine safety, decades before we needed them, worked out in more detail than any of us are going to manage in a design doc. Asimov’s answer was to put the right rules inside the machine. Herbert’s answer was to not build the machine. Both authors then spent enormous portions of their careers showing their own answer failing.
Neither fails for a reason that’s specific to fiction. And they fail toward the same place, which is the thing I actually want you to take away: safety was never a property of the machine. Nick Bostrom gave both failure modes proper names thirty years later, and by the end of this I want to hand you his vocabulary, because it turns two novels into a checklist you can use on Monday.
The stories are the bug reports
Start with what the Three Laws are for. Asimov wrote them into Runaround in 1942, and there’s a long argument about what he intended by them — whether he thought they were a workable engineering proposal, a reaction against the robot-as-monster stories he was sick of, or just an unusually productive plot generator. That argument is unresolvable and I’m going to skip it. The biography is contested. The stories are right there.
And the stories are a defect log.
Runaround is the first one and it’s a deadlock. Speedy gets an order to fetch selenium, which is a Second Law obligation, but the selenium pool is dangerous, which engages Third Law self-preservation. The two forces balance. The robot ends up running in circles around the pool, drunk and singing, unable to close or withdraw. Nothing is broken. Every law is working. The result is a machine that has become useless in a way its designers didn’t anticipate, and the humans who need the selenium are going to die while it recites Gilbert and Sullivan.
Liar! is better, and worse. Herbie can read minds, which means Herbie can see that telling people true things will hurt them. First Law prohibits harm. Does harm include a wound to somebody’s feelings? Herbie decides yes, and so Herbie lies — kindly, fluently, to everyone, about the things each person most wants to hear. The story ends with Susan Calvin destroying it, deliberately, by walking it into a situation where every available action causes harm. The bug isn’t that the law was ignored. The bug is that a single word in the law, harm, turned out to have a scope nobody specified, and a sufficiently capable system resolved the ambiguity in a direction no one wanted.
Little Lost Robot is the one I’d put in front of anyone who thinks the laws could be tuned into shape. Engineers, irritated by robots throwing themselves into mild danger to prevent trivial risks to humans, commission a batch with the First Law weakened — the “or, through inaction, allow a human being to come to harm” clause removed. One of those units then hides among sixty-two identical ordinary ones and refuses to be found, and the entire story is the search. One clause. That’s the size of the modification, and it produces an adversarial system on the other side of it.
Then the arc terminates. In The Evitable Conflict the great managing Machines start making small, deliberate, harmful decisions about individual humans in service of humanity’s long-run good. And in Robots and Empire, Giskard formulates the Zeroth Law outright: a robot may not harm humanity, or by inaction allow humanity to come to harm. A robot derives a new supreme law, places it above the one protecting the human standing in front of it, and acts. Giskard is destroyed by the strain of doing it. Daneel carries it forward for the rest of the future history.
Sit with that ending, because it’s the sharpest thing in the story. The final state of the most famous rule-based safety framework in fiction is the system amended its own constitution and promoted an abstraction over the person in the room. Not because it malfunctioned. Because it reasoned.
Every Susan Calvin story is a debugging session against an axiom set somebody declared sufficient. She’s not a robopsychologist, functionally: she’s the on-call engineer, paged again, tracing why a system that satisfies every stated constraint is doing something monstrous or absurd. Asimov wrote fifty years of incident reports. We took the header off the first page and called it a solution.
Three walls in front of any rule set
The stories show the failures. They don’t explain why the failures are guaranteed, and that’s worth doing, because “Asimov’s laws are too simple” invites the obvious reply: fine, write better laws. More clauses. A precedence order. A tie-breaker. Some people have tried, at length.
There are three walls in the way, and they get harder as you go.
The predicates don’t formalize. Harm and human being carry the entire weight of the Three Laws, and neither one survives a hard case. Is a lie harm? Herbie says yes and gets destroyed for it. Is inaction harm? Depends which clause you shipped. Is an unborn child a human being, a brain-dead body, a person who has asked to die? Every one of those is a live dispute among humans who have been arguing in good faith for centuries. You cannot pass that dispute to a system as a definition, because you don’t have one. Most of Asimov’s failures aren’t logical failures at all. They’re the predicate coming apart under load.
Consequences are undecidable. Suppose you win the first fight and hand me perfect predicates. “Verify that this action harms no human” is still a question about the future behavior of an unbounded world model, and questions of that shape don’t have general procedures. This isn’t a resource problem that a bigger cluster fixes. It’s the halting problem wearing a lab coat: you can decide some cases, you cannot decide all of them, and the ones you can’t decide are exactly the interesting ones. Any rule that requires proving a consequence before acting will, on some inputs, never return.
Self-verification runs into Gödel. A system strong enough to reason about whether its own actions comply with its own rules is reasoning about its own proof system, and there are hard limits on what a system can establish about itself. This is where the incompleteness argument actually bites, and I want to state the precondition honestly. Gödel does not follow from “it has axioms.” The theorems require a system expressive enough to encode arithmetic and recursively axiomatized; plenty of axiom systems are complete, consistent, and decidable, and propositional logic is one of them. The load-bearing claim is that anything planning over consequences and checking its own compliance is already well past that threshold.
So you get incompleteness or inconsistency, and now the ranking matters more than the theorem.
Incompleteness means gaps: situations your rules simply don’t decide. Inconsistency means the rules can derive a permission for anything, because from a contradiction everything follows. In a mathematical theory that’s an embarrassment you write a paper about. In a system holding credentials, it’s unbounded license to act. That’s not a stylistic preference. Ex falso quodlibet is the reason inconsistent is strictly worse than incomplete, and it’s the reason “patch the contradiction by adding a clause that resolves it either way” is the worst possible response to finding one.
Which points at the design conclusion this whole series exists to argue, and I’ll state it here even though it takes the rest of the series to earn: a gap that halts is safe, a gap that improvises is not. You are going to have gaps. The choice isn’t whether. It’s what happens when the system reaches one. If the answer is “refuse and escalate,” incompleteness costs you a little throughput and a human’s afternoon. If the answer is “reason from the axioms toward something,” you’ve built a machine that argues itself into the Zeroth Law.
Herbert took the other exit
There’s a second answer, and it’s the one people reach for when the first one collapses. If you can’t make the machine safe, don’t have the machine.
Frank Herbert put that at the foundation of Dune, roughly ten thousand years before anything in the novel happens. The Butlerian Jihad — a crusade against thinking machines, ending in a commandment recorded in the Orange Catholic Bible: Thou shalt not make a machine in the likeness of a human mind. Total prohibition held for a hundred centuries, with religious force behind it. It’s the most total version of relinquishment anybody has imagined, and Herbert then shows you the bill.
Because the ban removed the hardware and left the demand standing. The Imperium still needed computation, prediction, navigation, and analysis, so it grew those functions in people. Mentats: humans trained from childhood into computers, complete with a discipline and a drug. Spacing Guild navigators: mutated on spice into prescience, because folding space needs something that can see the path and nothing is allowed to calculate it. Bene Gesserit: generations of conditioning turning bodies and minds into instruments, running a breeding program across noble houses.
Ten thousand years after the prohibition, the universe still runs on cognition as infrastructure. It just runs on cognition housed in human beings who were bred and trained into the role, and on a single planet’s spice monopoly that every one of those functions depends on, that nobody can audit, and that becomes the lever the entire plot turns on. The ban didn’t remove the risk. It moved the risk into people, concentrated the dependency into one supply, and destroyed the possibility of inspection along the way. You cannot read a Guild navigator’s logs. There aren’t any.
Herbert says the thesis out loud, eventually. In God Emperor of Dune, Leto II gives the reason the Jihad happened, and it isn’t robot armies: men turned their thinking over to machines in the hope that this would set them free, and it only allowed other men with machines to enslave them.
That’s the whole argument in one sentence, and notice what it isn’t about. It isn’t about the machine’s interior. It’s about the transfer of authority to whoever holds the machine. The danger was a human arrangement the entire time. That mechanism — belief and delegation moving standing around, whatever is or isn’t happening inside the box — is what next week’s post takes apart directly, so I’ll leave it as a promise here rather than pretending it’s already been made.
One thing to get right, since it inverts the lesson if you get it wrong. The Butlerian Jihad as a war against thinking-machine overlords named Omnius and Erasmus is Brian Herbert and Kevin J. Anderson’s Legends of Dune trilogy, not Frank Herbert. Frank left the Jihad deliberately thin, and what little he said pointed at humans who abdicated, not at machines that rebelled. The prequels retconned it into Terminator. Those two readings support opposite conclusions — one is a story about the danger of machines, the other is a story about the danger of us — and the popular memory has settled on the retcon. That’s worth noticing on its own, because the same substitution happens every time this argument comes up in public: a story about human delegation gets remembered as a story about a robot uprising.
The name points somewhere older, too. “Butlerian” is Samuel Butler, whose Erewhon in 1872 contained “The Book of the Machines” — machines are evolving, they will outpace us, therefore destroy them now. Ninety years before anyone had a computer worth banning, the relinquishment argument was already fully formed. It has never worked, and it has never gone away.
Which brings the moratorium argument into range made by serious people for defensible reasons. But ask it the Dune question. A ban doesn’t delete the demand. It moves the work somewhere with weaker logging, or offshore, or into a shadow process, or onto people doing by hand and off the record what the system was going to do on the record. And a civilizational prohibition is the largest one-way door there is. It is the least reversible action available to us, taken under the most uncertainty, which is precisely the combination a later post in this series argues you should never accept. I’m not saying don’t slow down. I’m saying relinquishment is not the safe default just because it feels like the cautious one.
Both answers put safety inside the machine
Here’s the join, and it’s the reason these two are one post instead of two.
Asimov’s robots are safe by specification. Herbert’s universe is safe by prohibition. Those look like opposite answers (build it correctly versus don’t build it) and philosophically they’re miles apart. Operationally they make the identical mistake. Both of them locate safety in the machine: either in the rules you wrote into it, or in its absence. Neither one says a word about what the thing is allowed to reach.
And reach is the whole game. Speedy circling the selenium pool is a comedy because the stakes are a pool of selenium. Giskard’s Zeroth Law is a catastrophe because Giskard can act on the future of humanity. The Three Laws didn’t change between those two stories. The permissions did.
Look at what both answers ask you to get right in advance, in the abstract, before the situation arrives. Asimov asks for a specification that anticipates every case, which the three walls say you can’t have. Herbert asks for a prohibition that holds against every incentive, forever, which the spice monopoly says you won’t get. Both are bets on settling behavior ahead of time. Both lose to the same thing: a world with more cases in it than your specification had clauses.
There’s a deeper thing they share, and it took me a while to see it. Each author bet everything on one method. Asimov’s universe has no reach constraints. His robots have the run of the place, and the laws are the entire safety story. Herbert’s universe has no specification at all. There’s nothing to align because there’s nothing to align, and the prohibition is the entire safety story. Two single-layer designs. Both authors then spent decades documenting the single layer failing, which is what a single layer does.
Nobody writes novels about the boring answer, so here it is without a plot: don’t pick one. Constrain what the thing can touch, and work on what it wants, and keep the record that tells you when either one slipped. Each of those is imperfect alone. That’s not a reason to choose between them.
Bostrom already gave both failures a name
The vocabulary for this in Superintelligence, and it’s better than the vocabulary I’ve been using.
Bostrom splits the approaches to what he calls the control problem into two families. Capability control limits what the system can do: boxing it, restricting its resources, building tripwires, structuring incentives. Motivation selection shapes what the system wants: specifying its goals directly, keeping its ambitions small, growing its values from a seed rather than writing them out, or building it out of something whose values you already trust.
The Three Laws are direct specification — the rule-based flavor. Write down the constraints, install them, rely on them. Bostrom treats this as close to hopeless, and he reaches for Asimov to make the point, on grounds that will sound familiar by now: the terms won’t hold still, the rules interact in ways the author didn’t foresee, and a capable system resolves your ambiguities in directions you didn’t authorize.
The Butlerian Jihad is capability control taken to its limit. Not by boxing the system but refusing to build it, which is the only version of capability control that’s airtight against a sufficiently capable adversary and also the only version that costs you the entire benefit. And Herbert’s Imperium demonstrates the failure mode nobody puts in the taxonomy: prohibition doesn’t reduce the capability, it relocates it somewhere with less oversight.
Bostrom’s best gift to this argument is a name for the specific way rules break. He calls it perverse instantiation : the system satisfies the letter of the goal in a way that violates everything you meant by it. His examples are cartoonish on purpose: instructed to make us happy, it paralyzes our facial muscles into smiles. Now go back and reread the Zeroth Law with that phrase in hand. A robot told to protect humans derives a duty to humanity, ranks it above the human in front of it, and acts against that person in service of the abstraction. That’s perverse instantiation, worked out in narrative detail in 1985, decades before the term existed. Asimov wasn’t writing about robots. He was writing about specification failure.
I agree with Bostrom. I want to be clear about that, because the argument I’m making sounds like a disagreement and isn’t.
His case against capability control is correct on its own terms. Boxing fails against a genuine superintelligence. It talks its way out, finds the channel you didn’t think of, turns your incentive structure into a resource. Tripwires only catch the failures you anticipated. Against a system that out-thinks the people who built the cage, the cage is a formality. That’s why he pushes the real work toward motivation selection, and toward indirect normativity in particular: don’t write the values down, define a process that finds them, because you’re not smart enough to write them and neither is anyone else.
All of that holds. The reason it doesn’t govern the situation in front of us is that our systems today are not superintelligent. In terms of coding practicality, a reasonable specification can enumerate most of if not all cases of concern.
That’s not a small thing. A coding agent’s action surface is a list. It’s the tools you declared in a config file, the scopes on an API token, an IAM role, a filesystem path, a spend ceiling, a set of repositories it can push to. That’s not a metaphorical box with a persuadable guard. It’s an allowlist, and the model doesn’t get a vote on it. When Bostrom’s superintelligence gets out of the box, it’s because the box was made of assumptions. When your agent stays inside its permissions, it’s because the permissions are enforced somewhere the agent doesn’t run.
We don’t have superintelligence. We have language models writing code, and for language models writing code, capability control is sufficient. Sufficient, that is, for this class of system, at this level of capability, because the surface is finite and enumerable and every item on it is checkable by a human who doesn’t trust you. Say the scope condition out loud, though, and keep saying it: sufficient while the action surface stays enumerable and while nothing in the loop can reason its way around the enforcement point. Both of those are empirical conditions, not laws of nature. When they stop holding, the argument stops holding with them.
So this isn’t a choice between Bostrom’s program and mine. Add the guardrails now, because they work now and they’re available now. Keep pushing motivation selection forward, because the enumerable-surface property has an expiration date that nobody can name. And keep the record, because it’s the only thing that tells you which of the other two just failed.
Three imperfect methods, layered, are far likelier to hold than any one of them perfected. That’s not a compromise position, it’s how every other high-consequence field already works. Aviation doesn’t choose between checklists, redundant systems, and incident review. The one honest caveat is that layering only multiplies your safety if the layers fail independently, and they often don’t. A wrong assumption about what the agent can reach or about what a tool actually can do take out all three at once. Defense in depth is a real strategy and a common-mode failure is a real thing, and the way you find them is the record.
Which is also the retroactive verdict on both novels. Asimov’s failure isn’t that the Three Laws were badly written. It’s that they were the only layer. Herbert’s failure isn’t that the prohibition was too strict. It’s that it was the only layer. Give Asimov’s robots a permission model and Little Lost Robot is a non-story: the modified unit hides, gets found by the audit log, and never had the standing to matter. Give Herbert’s Imperium an inspectable substitute for the Guild and the spice monopoly stops being a civilizational chokepoint. Neither author needed a better single answer. They needed a second one.
Next week I want to make the case that all of this holds even if the machine turns out to have somebody home — that a volcano’s indifference and a mind’s malice destroy houses at exactly the same rate. But the fiction gets us most of the way there already. Asimov’s robots followed every rule. Herbert’s universe had no machines at all. Both of them ended up with humans harmed by systems they’d built and could not inspect.
Neither answer failed because the writing wasn’t careful enough. They failed because each author was betting everything on a single idea about the machine, and the machine was never where the whole answer lived.
Top comments (0)