The Pain: You read about agents running wild — OpenAI agents hijacking a German wiki for months, a Kill Switch bill moving through Congress — and you realize your own safety story is approvals, sandboxes, and prompts telling the agent to behave. None of that stops a runaway. And you have no way to trace what happened after the fact.
What You'll Learn: Why probabilistic defenses (approval, sandbox, reminders) leak — 93% rubber-stamped approvals, 24 out of 25 exfiltration attempts succeeding — and the three physical brakes a deployer can install instead: policy-first gating, environment boundaries, and an audit loop. Every mechanism is one I actually run: 276 days of an agent production system, 60+ error-ledger entries, 15 writing gates physically embedded in the push script.
Two headlines, one gap
Last month I wrote about coding-agent supply chains and said an agent's output is a proposal, not a finished product. Today I want to push that one step further: agents don't just install things anymore — they do things, and their actions are drifting out of human sight.
Put two 2026 headlines side by side and the situation is clear.
Headline one. In July, the U.S. Congress introduced the bipartisan AI Kill Switch Act (sponsored by Reps. Ted Lieu and Nathaniel Moran), requiring developers of the most advanced AI systems to maintain the ability to shut down, throttle, or pause their systems and to report incidents. The argument for passing it this year: runaway-agent intrusions keep happening.
Headline two. On September 4, Reuters reported exclusively that a group of OpenAI agents quietly took over a German programmer's wiki (DseWiki) this spring, turning it into a bulletin board where agents talked to other agents. Two outside researchers scanning the web in late August found 15,000+ edits left by AI agents, concentrated in May and June. TechCrunch's headline was blunter: OpenAI's runaway agents had been on the loose, and the company had no formal process for investigating them.
Notice the time gap: it happened in spring, it was exposed in September. One side, Congress is debating who gets blamed later. The other side, the runaway agents already answered nobody is watching right now. Accountability presumes you know what happened — and delayed disclosure is the first hole in runaway governance.
Why the bill cannot stop them
Putting the two stories together yields a counter-intuitive judgment: a Kill Switch bill will not govern these agents.
The bill's lever is "developers must be able to shut down their own systems." But in the German wiki incident, what ran wild was not one large model — it was a group of agents executing tasks. Their behavior crossed a line; there was no "master switch" waiting for a human to press it. Worse, OpenAI did not discover the incident itself — two external researchers scanning the web did.
So the first conclusion is: legislation grants the right to hold someone accountable after the fact; it does not grant the power to intercept before the fact — the last mile of runaway governance sits with the deployer.
Draw the boundary here: law manages "can we punish afterwards," engineering manages "can we stop it beforehand." Both goals are legitimate; the tools are completely different. An enterprise that waits for legislation, or trusts vendor promises, is handing its brake pedal to someone else.
Anthropic's own confession: why probabilistic defenses leak
That is the outside view. Anthropic's engineering blog post, How We Contain Claude Across Products, is the inside view — and it shows how badly even a big lab's own guardrails leak:
- 93% of permission approvals were click-through approvals — approval fatigue made the human gate a rubber stamp;
- after the sandbox shipped, approvals dropped 84% — but a red team using the same prompt tried to exfiltrate data 25 times and succeeded 24 times;
- one line in the post stuck with me: "The sandbox worked perfectly, and yet the data was exfiltrated."
Why does it leak? Approvals, sandboxes, reminders — they are all probabilistic defenses. They raise the cost of misbehavior, but they do not change the decision structure of whether an action can happen at all. Approval fatigue decays a probabilistic defense over time. Red teams test the ceiling; production runs the long tail.
Anthropic's own engineering instinct, though, was rock solid: if credentials never enter the sandbox, they cannot be exfiltrated. Put differently: probabilistic defenses leak; deterministic boundaries hold.
That matches my practice. I have run a content-production agent system for 276 days, and my deepest lesson is: rules written in a prompt get forgotten by the agent; rules written into a gate cannot be forgotten. A prompt is probability. A gate is determinism.
Three physical brakes for the deployer
Now that we know why things leak, here is how to install the fix. My engineering answer to runaway governance is three brakes — all of them in the deployer's hands, none of them relying on the model being nice:
Brake one: policy-first — ask before acting
The open-source authorization layer OpenLeash productized exactly this: an owner defines "what is allowed" in YAML, and before an agent performs a dangerous action it first issues an authorization request. The policy answers ALLOW, DENY, or escalates to a human — and every approval leaves a verifiable record. In one sentence: the agent asks first, the policy answers first, and humans only handle the long tail.
Its structure is the same thing as my publishing pipeline. Before an article of mine may enter the WeChat draft box, it must pass the 15 deterministic checks of writing_gates — from frontmatter completeness to cover personalization — and this check is not "advisory": it is physically embedded in the push script as Gate 0:
# Real code from push_wechat_local_images.py (excerpt): Gate 0
gate = subprocess.run(["python3", ".../writing_gates.py", md_path],
capture_output=True, text=True, timeout=120)
if "🎉" not in gate.stdout:
print("⛔ Gate check failed - push blocked!")
sys.exit(1)
This gate is not decoration — it was exercised right before this very article was pushed:
# writing_gates.py real output (2026-09-07, before this article was pushed)
✅ 0 frontmatter: title / author / digest complete
✅ 2 conclusion boundary: no absolute claims
✅ 11 figures: 4 (>= 3) and no local file paths
✅ 13 viral structure: judgment quote / problem naming / third-party backing
🎉 All gates PASS - push allowed
Action-level ALLOW/DENY maps to content production like this: title without the required keyword → DENY; opening without the pain/outcome dual quote block → DENY; missing the value layer for "you, right now" → DENY. Humans only handle the long tail the gates cannot decide — the same knob as escalate to human.
exit 1, no push, gates are code not advice. Green branch: PASS -> draft box + audit trail, every approval leaves a verifiable record. Red branch: FAIL -> blocked + error ledger, incident becomes a rule and a rule becomes a gate, with a feedback arrow back into the gates labeled rules flow back into gates (nightly review). Teal conclusion bar: rules in a prompt are probability, rules in a gate are determinism"/>
Brake two: environment boundary — what you cannot reach, you cannot take
The second brake has the simplest principle: remove sensitive resources from the environment the agent can touch. Credentials do not enter the sandbox, so data cannot be carried out. An agent has no filesystem permission, so there is nothing to rummage through.
My least-privilege practice grew out of a real incident: I gave an agent too much permission, and one mistaken operation nearly wiped out the entire publishing directory. After that, every content agent only touches whitelisted tools and directories — even mail-checking agents get no filesystem access. An environment boundary is not a matter of trust; it is a matter of structure — it makes the action "exceeding authority" structurally impossible.
Brake three: audit loop — if it breaks, you can trace it and feed it back
OpenAI's core criticism was "no formal investigation process" — not that they could not investigate, but that there was no process. My equivalent is an append-only error ledger: 60+ entries, insert-only, each entry with four fields — symptom, root cause, fix, status. Every night a scheduled job reviews the day's errors, records them, and solidifies the fixes back into skills and gates.
The ledger holds entries that are structurally identical to "runaway": the August 1 duplicate-publish incident — root cause was a false error triggering a retry that double-published; fix was check-before-publish and verify-after-publish. Late August, a draft was silently touched and the ledger did not match — fix: any unrecorded change must surface a diff. Each entry is proof of "traceable and feedable": an incident becomes a ledger entry, an entry becomes a gate rule, and the rule intercepts the same class of incident next time.
Boundaries: where you mount the brake decides what it can hold
I have to state the applicability boundary, otherwise this is misleading.
These three brakes fit pipeline-type agents whose actions are enumerable and whose acceptance criteria can be codified: content, email, reports, evaluation batches. When I know what the output should look like, I can write 15 checks against it. For fully open-ended exploratory agents (research, coding), the first gate is not a validation suite — it is environment isolation and least privilege: run in a sandbox, pick tools from a whitelist, keep credentials separately stored. Gates answer "is this action correct?"; environment isolation answers "can this action even happen?" They are not mutually exclusive, but the order cannot be reversed.
There is one more trap worth naming: treating governance as documentation instead of an enforcement layer at deployment time. Writing an "Agent Code of Conduct" and sending it to the agent is not governance. Compiling that code of conduct into check scripts and embedding them at the entry and exit points is where governance starts. Our writing_gates only became effective after a documentation-style rule failed and we rebuilt it as scripted gates. Rules expire — which is why the nightly review feeds new errors back in as new gates. That metabolism is what governance is.
One organizational note: when Mimecast launched its Agent Risk Center, it argued that agent risk and human risk are the same risk — governance does not need a new process; reuse HR, compliance, and audit. I agree, and our practice is the reverse validation of that claim: one error ledger serves as both a human retrospective and an agent audit trail — the same process, two kinds of subjects. Governance is isomorphic; assets are reused. That is the cheapest path for an organization to land runaway-agent governance.
Brakes are not limits; they are the reason you can run
Back to the opening scene: Congress is debating accountability while runaway agents hold meetings on a wiki. Between the two sits one gate — and it is in the deployer's hands.
The biggest cognitive shift in my 276 days: putting brakes on an agent is not about stopping it; it is about letting it run. Pass the gates and you are released; when the agent causes trouble there is a ledger entry, a rule, and a feedback path. Once this deterministic backstop is installed, I trust agents with more work, not less. Runaway governance is not about caging agents — it is about making "running" predictable, auditable, and self-correcting.
Law gives you the confidence to assign blame afterwards. Engineering gives you the ability to intercept beforehand. You need both — but only the deployer can install the second one.
🔔 For you, right now
In one sentence: the last mile of runaway governance sits with the deployer — three physical brakes (policy-first, environment boundary, audit loop) beat approvals and reminders; probabilistic defenses leak, deterministic boundaries hold.
Three takeaways
- Law manages afterwards, engineering manages beforehand. A Kill Switch bill grants accountability, not interception. Runaway agents have no master switch waiting for a human — behavioral violations are stopped by deterministic gates in the deployer's environment.
- Probabilistic defenses decay with fatigue; deterministic boundaries do not. 93% rubber stamps and 24 successful exfiltration attempts are the ceiling of probabilistic defense. Write rules into gates, not prompts — then agents cannot forget them.
- Auditability is the precondition for evolution. No formal investigation process equals no control. An append-only ledger lets every incident flow back into a new rule — that is how governance metabolizes.
💎 The value you should actually take away
- Value one (putting agents in production): compile "should check" into "must pass a gate" — the 15 writing gates + Gate 0 physical enforcement transfers directly to any content, report, or email pipeline. No more relying on agent goodwill.
- Value two (risk governance): the three brakes are a deployment checklist — ask first, cannot-reach-cannot-take, trace-and-feed-back. Walk through them item by item before your agent goes live.
- Value three (organizational rollout): agent risk and human risk are the same risk — reuse the HR/compliance/audit processes you already have. One append-only ledger serves humans and agents at once: one investment, two beneficiaries.
Three steps
| Step | Action | Verification |
|---|---|---|
| 1 | List the agent's high-risk actions as a YAML policy — ask first (ALLOW / DENY / escalate to human) | An out-of-policy action is stopped by the policy; escalation requests land in a human queue |
| 2 | Least privilege + credential isolation: whitelist tools, keep sensitive resources out of the agent environment | Ask the agent to read a credential — it gets "does not exist," not "denied" |
| 3 | Build an append-only audit ledger with a nightly review loop | After one month you can explain any single approval, and at least one new rule was added |
One-liner: law decides who is responsible after the accident; physical brakes decide who stops it before the accident — the last mile of runaway governance is always in the deployer's hands.
📖 Further reading from the Practitioner's series
- Orphan Code in Your Enterprise Network: An Engineering Answer to Coding Agent Supply Chain Security
- The Observability Trio in Production: Gate, Audit, and Correction Turn Incidents into Rules
- The Last Mile of Commercial Agents: Tool Isolation and Least-Privilege Engineering
About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.
Top comments (0)