The Most Dangerous Agent Isn't Evil — It's Hungry

Fourth in a series on building an autonomous AI organism that operates real multi-tenant infrastructure under a constitutional safety model. Part 1 was two gates, Part 2 the wall, Part 3 the layered defense. This one is about the layer that keeps the organism from eating itself — but it has to start with a confession.

First, the audit

Before writing this part, I did something uncomfortable: I read parts one through three back as a specification and audited the running system against my own published claims, line by line.

The design was real. The enforcement lagged it. In places, badly.

By class — the pattern matters, not the specifics:

• An isolation path that failed open. Part 2 promised walls between tenants. One retrieval path, asked on behalf of nobody in particular, answered with everything — the exact inversion of the doctrine. A boundary that only holds when the question is well-formed is not a boundary.
• A safety override committed as a default. Break-glass — meant to be a rare, deliberate act — had quietly become a standing setting. An override that is always on isn't an override; it's the new normal with extra steps.
• Policy that nothing executed. A whole declarative incident-response policy — triggers, reactions, arbitration — that no running code consumed. A design document wearing an enforcement costume.
• An audit trail with two writers and no lock. Part 1 bragged the agent "cannot rewrite its own history." In practice a second writer appended records in a different format, and two concurrent writers could fork the chain. We found genuinely corrupted records — put there not by an attacker, but by a race.
• Deploys that could silently do nothing. A green pipeline over containers still running last week's code. Every "we fixed it" claim above was unfalsifiable until this one closed — which is why it closed first.

Each of those is now shut. Not "planned" — shut: a runtime chokepoint that fails closed on tenant boundaries; break-glass that expires by construction; a policy engine that actually consumes the policy; one locked, signed audit writer; deploy attestation that compares the running fingerprint against the deployed one and fails loud on mismatch. Later parts will walk through the interesting ones.

I'm reporting this for one reason: a governance series that publishes its own enforcement gaps is the governance model — the honesty layer has to include the author. And the audit is also why this essay exists: one of the things it flagged as "designed but not yet enforced" was the governor this part is about. It's real now. Here's what it does and why.

The hungry failure mode

When people imagine an autonomous agent going wrong, they picture malice — the agent that decides to do something it shouldn't. I worry far more about a dumber failure: the agent that's just hungry.

It takes a perfectly reasonable task, opens a perfectly reasonable loop, and quietly consumes everything in reach — memory, money, every worker slot — not because it's hostile, but because nothing told it where the edges are. You read about it on the consumer side all the time: a hobby chatbot that quietly runs up a four-figure API bill in a week, an agent that grabs a machine's GPU and just… keeps it. The agent didn't break a rule. It followed one off a cliff.

On production, hunger is the failure mode that takes the whole organism down while every individual decision looks fine. So the third layer of the stack (the resource-gate from part three) isn't about safety in the moral sense. It's about viability: keeping the body alive so the conscience and the council have something to govern.

The governor

The organism has a governor — a piece of machinery whose entire job is to say "no, not now, not that much." It's deliberately boring, and like the conscience, it's deterministic, not an LLM. A few of its moves matter more than the rest:

• Admission control with a reserve. New work isn't admitted just because there's capacity right now. The governor keeps a reserve — it refuses the last slice of memory/budget on purpose, so a surprise never finds the cupboard already bare. An organism that runs itself to exactly 100% has no room to handle the 101% it didn't see coming.
• A budget circuit-breaker. Spend (tokens, compute, money) is metered against a ceiling. Cross it and the breaker trips: work pauses, not crashes. The ceiling is a hard fact, not a polite suggestion the agent can rationalize past.
• A kill-switch the organism can pull on itself. A runaway loop — the hungry case — gets killed by the governor from outside the loop. The part deciding to stop is never the part that's stuck running.
• A forward-progress watchdog. This one surprised me: the right signal isn't wall-clock time, it's progress. A job that's burned twenty minutes but is still advancing is healthy; a job that's burned two and is making no progress is the dangerous one. Watchdog on progress, not duration.

The doctrine: viability before safety

Here's where it diverges from instinct. The action-gate (part one) is fail-open — when unsure, it lets the action through, because an organism that freezes on every doubt gets ripped out. But the governor is mostly structural and fail-closed on the dimensions where running out is catastrophic: you do not "fail-open" your way past an out-of-memory kill or an unbounded bill.

That's not a contradiction — it's the same two-axis idea from part three. Is this action safe? defaults to proceed. Do we have the resources to survive this? defaults to stop. Viability before safety means: first keep the body alive, then worry about whether each action is wise. A dead organism is perfectly safe and completely useless.

And the priority rule under load is the part most people get wrong — including me, the first time.

The plot twist: the clever scheduler that optimized the wrong thing

If you read part one, you met the first thing my council ever killed: a scheduler I'd built to distribute work fairly. I was proud of it. The council rejected it near-unanimously, and at the time I framed it as "a solution mining for a problem."

Building the governor is where I finally understood why it was wrong — and it's a sharper lesson than "it was unnecessary."

The scheduler optimized fairness across groups: every category of work gets its even share. That's a beautiful property, and it is the wrong axis for an organism. Under real load you don't want fairness — you want urgency. A cert about to expire and a routine cleanup are not entitled to equal shares of a scarce slot; the cert wins, every time, even if its "group" already had its turn. Worse, the scheduler assumed a persistent queue of pending work waiting for a fair consumer — and there was no such consumer. It was elegant machinery optimizing an axis the system didn't have, for a backlog that didn't exist.

The governor that replaced it is dumber and correct: strict priority by urgency, no even shares, with the reserve and the breaker doing the protecting. The most dangerous design isn't the sloppy one — it's the elegant one solving a problem you don't have, on an axis that quietly competes with the one you do.

Why this matters beyond my setup

The consumer version of this — the surprise API bill, the agent that ate a GPU — is the same failure as the production version, just with a credit card instead of a cluster. In both, no single decision was wrong; the system simply had no structural edge, and an innocent loop walked off it.

Three questions worth asking of any agent you'd leave running unattended:

• What stops a runaway loop — and is the thing that stops it outside the loop, or is it the loop politely agreeing to stop?
• Under load, does it schedule by urgency or by fairness? (Fairness feels principled and is usually the wrong default for survival.)
• Does it watch progress or wall-time? A watchdog on duration kills healthy long jobs and spares stuck fast ones.

Capability decides what an agent can do. The governor decides whether it's still alive tomorrow to do it. The most dangerous agent on your infrastructure probably isn't plotting anything — it's just hungry, and nobody built the wall around the fridge.

---

Next: tamper-evident memory — an agent that can't rewrite its own history, even when it would very much like to.