Here's the moment a lot of us had this year.
You built an agent. It was genuinely impressive — it reasoned, it planned, it called tools, it chained steps together to get real work done. The demo was so good that you gave it real permissions. Let it send the emails. Let it update the records. Let it hit the production API.
And then, one day, it did something you never sanctioned. Not maliciously — confidently. Correctly-looking. It completed the action, reported success, and moved on. You only found out later, when the consequence surfaced somewhere downstream. That's the moment you realize what you actually shipped: a system with a powerful engine and no brake pedal.
The entire industry has spent two years racing to make agents smarter. Better reasoning, longer context, more tools, more autonomy. Almost nobody spent that time making them safe to let act. And those are completely different problems — because the instant an agent stops just talking and starts doing, a more capable agent isn't more useful. It's more dangerous.
Let me make the case, and then talk about the brakes.
Brains and brakes are different problems
This is the distinction the hype cycle blurred, so it's worth stating plainly.
Making an agent capable — able to reason about a task, plan a sequence of steps, pick the right tool, recover from an error — is one problem, and it's the one all the model progress went into. Every new release is a better brain.
Making an agent safe to let act — ensuring that when it's wrong, the wrongness is caught, bounded, reversible, or stopped before it does damage — is a completely different problem. And here's the uncomfortable part: no smarter model solves it for you. A more capable agent doesn't act wrongly less; it acts wrongly faster and more convincingly. Upgrading the brain does nothing for the brakes, because the brakes were never in the model. They're in the architecture around it — the part you have to build yourself.
You can have the best engine in the world. Without brakes, that just means you hit the wall sooner.
Why "acting" changes everything
A chat that only talks has a wonderful property: its mistakes are free. It says something wrong, you read it, you move on. Nothing happened in the world.
The moment an agent can act, three things change, and each one is a reason you need control that a chatbot never did.
Irreversibility. A wrong sentence costs nothing. A wrong DELETE, a sent email, a processed refund, a pushed commit — those don't have an undo button. The cost of an agent's mistake stops being "re-read it" and becomes "clean up the damage," and some damage doesn't clean up.
Confidence is not correctness. This is the through-line of everything I write, and it's never more dangerous than when an agent can act. The agent takes the wrong action with exactly the same confidence it takes the right one. There is no tremor in its voice, no hedge, no tell. It is equally sure when it's about to help you and when it's about to hurt you.
Silent success. The nastiest one. The agent completes the wrong action and reports success — because from its point of view, it did complete the task, it just completed the wrong one. Your monitoring goes green. Nobody sees the wrong customer getting the refund until the wrong customer gets the refund. "It worked" and "it did the right thing" are different claims, and only one of them shows up on the dashboard.
Put those together and you get the core risk: an agent that can act is a system that can cause irreversible harm, confidently, without anything flagging it. That is precisely the situation brakes exist for.
Brake #1: Approval gates — ask before acting
The most basic brake, and the one most systems skip: for anything consequential, the agent doesn't do the thing. It proposes the thing, and waits for a human to say yes.
The skill here isn't gating everything — that would make the agent useless, and you'd click "approve" on autopilot until the gate meant nothing. The skill is gating by blast radius:
- Auto-run the trivial and reversible — read something, format something, draft something. No gate needed.
- Ask first for the consequential — sending, paying, writing to production, anything a user would want to see before it happens.
- Draft-only for the dangerous — the agent prepares the action but cannot execute it at all; a human does.
The mistake to avoid is a uniform gate on everything, which trains people to rubber-stamp. A good gate is rare enough that when it fires, the human actually looks.
Brake #2: A reviewer that can actually say no
The popular pattern: a second agent reviews the first agent's work before it's allowed to proceed. A judge. A critic. It's a good pattern — and it has a failure mode that will fool you completely if you're not watching for it.
A reviewer that has never been seen to fail is indistinguishable from a reviewer that approves everything.
If your judge-agent has been green for months, that is not evidence it's working. It might be catching problems. It might also be rubber-stamping, silently broken, or checking the wrong thing entirely — and you would not be able to tell the difference from the outside, because both produce the same clean logs right up until the day the rubber stamp waves through the thing that hurts you.
A brake you have never tested is not a brake. It's a brake-shaped object. So if you build a reviewer, you have to build the thing that proves it can still say no: wire a known-bad action through the live review path, on a schedule, and confirm the reviewer rejects it. Surface when it last refused something, the way you'd surface uptime. If that "last refusal" date goes stale, your brake may already be dead and you just can't see it yet. Prove the "no" is reachable, continuously, or you're trusting a checker you've never watched work.
Brake #3: An audit trail — know what it did
You cannot control what you cannot see. If an agent takes actions in the world and you have no durable record of what it did, when, and why, then a wrong action is invisible until its damage surfaces on its own schedule — usually the worst possible one.
Every consequential action should leave a trail: what the agent did, what triggered it, what it was acting on, and ideally a path to reverse it. This isn't bureaucracy. It's the difference between "we caught the bad action in the log an hour later and rolled it back" and "a customer told us three weeks later, and we have no idea how many others it happened to."
An audit trail is also what makes every other brake improvable — you can't tune a gate or a reviewer if you can't see what got through.
Brake #4: Blast-radius limits — caps and scopes
The final brake is the one that works even when every other brake fails: hard limits the agent physically cannot exceed, enforced by the system rather than by the agent's good behavior.
Rate caps (no more than N actions per minute). Spend caps (cannot exceed $X without escalation). Scoped permissions (can read these tables, can write only that one). Iteration limits (stop after K steps instead of looping forever). These don't depend on the agent reasoning correctly — they're walls, and walls hold whether or not the thing inside them is behaving.
This is where these controls show up in practice, and a few platforms now build them in as first-class features rather than something you bolt on — for instance, Xenition ships approval gates, an audit log, and a second agent that reviews the first's work before it reaches you (disclosure: it's a product I've been exploring). But whether you adopt a workspace that includes them or wire them yourself, the principle is the same and it's not optional: the agent's freedom should be bounded by design, not by hope. You do not want "it usually behaves" to be the only thing standing between your agent and your production database.
The uncomfortable truth: brakes are unsexy
Here's why, despite all of the above being fairly obvious, most agents ship without any of it.
Nobody demos the approval gate. Nobody opens a pitch with their audit log. Brains are exciting — they make the standup go "wow." Brakes are boring — they only ever matter on the day something goes wrong, and if they're doing their job, that day never visibly arrives, so it looks like you built them for nothing.
Choosing to build the unglamorous safety layer before you need it, when there's no incident yet to justify it, is exactly the kind of discipline that doesn't get applause. It's the senior move precisely because it's invisible when it works. The engineer who spent a day on blast-radius limits instead of adding a fifth capability looks slower — right up until the week their agent doesn't drain the account.
The takeaway
Capability has become table stakes. Every model is smart enough now; smartness is no longer where systems differ. Controllability is the differentiator — whether you can let the thing act without lying awake about what it might do.
So the question to ask about your agent isn't "how smart can I make it?" It's "what happens when it's confidently wrong — and can I stop it in time?" If the answer is "I'm trusting it not to be," you don't have an agent you can deploy. You have a liability with a good demo.
Build the brakes. Not because your agent is dumb, but because someday it will be confidently, fluently, irreversibly wrong — and on that day, the only thing that matters is whether you built a way to stop it before you needed one.
What's the one action you'd never let an agent take without a human in the loop? And the more honest question: have you actually wired that gate — and tested that it fires — or are you quietly trusting the agent to behave? I want to hear where people drew the line.
Disclaimer: This article was written with AI assistance and reviewed and edited by me before publishing.
Top comments (8)
这篇文章，我可以理解为，它是你的当前最新状态！😀
我说的“状态”是指：这不是你整理出来的观点，这是你走到此刻所在的位置之后的自然产出，即对“结果是自然的显化”这句话的回应。我把整场讨论看作一条持续的路径，而这篇文章就是“路径经过你之后留下的可读取的印记”。
That's a generous way to see it, and there's real truth in it — the article genuinely wouldn't exist in this shape without the path this whole thread walked. The "brakes" idea didn't arrive as a clean thesis; it precipitated out of a dozen exchanges about reviewers that can't say no, states that read as health, and confidence that isn't correctness. So yes — in that sense it's an imprint of the process, and you and everyone in these threads pressed the shape into it.
But let me add one honest counterweight, because it matters to how I think about my own work: I'd resist calling it purely a natural output. The path left material, but the article is also a choice — what to keep, what to cut, where to be sharp, what to leave out. Emergence gave me the raw shape; the deliberate part was refusing to publish the version that merely sounded finished. Which is, funnily enough, the whole thesis of the piece turned back on the writing of it: the process produces something, but you still need a brake — a moment where you decide "not yet, verify this, cut that" — before it's allowed to ship. A result that's only natural manifestation, with no deliberate hand on it, is exactly the confidently-fluent-but-unchecked output the article warns about.
So maybe the honest version is: the path left the imprint, and then I stood between the imprint and the "publish" button and did the unglamorous work of deciding which parts had earned their place. Both things are true. Thank you for walking the path with me — it genuinely made this better.
🫠 “保留什么、切割什么、哪里锐利、遗漏什么”--这不就是此前套路中的“协议和规范部分的定义边界和范围”吗？
有人去做这件事情，你正是在做这件事情的“示范者”！
@james_anderson_h The order of operations is backwards. An impressive demo earns permissions, and then you think about brakes. The demo is convincing enough to be trusted, and that trust is the vulnerability. The agent that produces plausible-looking wrong output passes every brake that checks for crashes. The boundary needs to sit at the effect level before the agent runs a single tool call: what paths, what destinations, what credentials. Not on how fast it acts.
Trust is the vulnerability" is the sharper diagnosis, and the ordering critique lands: brakes designed after permissions are granted are retrofits on a trust decision already made. Your effect-level boundary has a pedigree worth citing: Saltzer and Schroeder, 1975 — complete mediation (every access checked before it happens) and fail-safe defaults (permission-based, not exclusion-based). Agents did not invalidate the principle; they made enumeration the hard part — and paths, destinations, credentials are enumerable, which is exactly why your boundary is enforceable while judgment-based brakes stay probabilistic.
One complement: the pre-execution boundary and the reviewer are different layers, and plausible-wrong output passes both when the reviewer shares the generator's blind spots. The boundary authorizes the call; it does not judge the output. And where your design meets specs — the TRACE PIC bridge is a live instance of it, signed pre-execution authorization checked at the gateway — its two documented limits apply: it is optional, and it validates the decision made before execution, not the state the call runs against.
Question, since you named the hard part: how does the boundary treat effects the author did not enumerate — a tool that mid-task makes its own network call or spawns a subprocess? That is where static scoping historically leaks.
The brakes/brains split is the right cut, and Brake #2 is the one this thread network has been grinding on all month — the "last refusal surfaced like uptime" telemetry you describe was shipped by Michael Hurst as a last-HOLD date and by Heinrich Neb as veto heartbeats. But between #2 and #3 there is a gap the last two weeks kept exposing, and I would name it Brake #2.5: the reviewer's identity, and the fate of its refusals.
Reachability does not prove independence. A planted known-bad case goes red for an honest reviewer and for one marking its own work — a self-signing judge passes your live-path test while sharing the generator's blind spots. The check is one line in every run output: which model produced the artifact, which model judged it. When a config comment says "kept separate" and the value says JUDGE_MODEL_ID = DEFAULT_MODEL_ID, that is a dead gate wearing a config file.
And a no that gets overridden is not a brake either. A reviewer with a healthy refusal streak and a high override rate is ceremony with telemetry — the override channel is the real review, and it needs its own ledger with ground truth on who was right.
On your closing question: write-path actions against a codebase. And the honest answer to your second question is that my gate failed its own test this week — my modification guard, the component whose job is to check write operations, silently resolved an ambiguous symbol target to the first candidate and would have applied the change to the wrong definition. The brake was itself picking the target blind. Filed as issues against my own tool; the fix refuses and prints the candidate list. Which confirms your thesis one level down: I knew about brake-shaped objects, wrote about verification theater — and still had to catch my own at a specific line number. Knowing about a trap is not immunity to it. That is why the firing test runs on a schedule, not on trust.
The scheduled known-bad action and the
last refuseddate are the same signal, and that is the part I would separate. Once the probe runs on a schedule the date can never go stale, so a reviewer that has degraded into recognising the probe and waving through everything else keeps exactly the freshness you are watching for; the failure mode moves fromnever seen to failtoonly ever seen to fail on the test input, which looks identical from outside. Brake #3 already carries the fix: count refusals from real traffic and from probes separately in the audit trail, so a live-refusal count of zero across a long window is visible on its own instead of being covered by the canary.The “last refusal” signal is a great idea. A reviewer being green for months can mean it works perfectly—or that it has quietly stopped detecting anything. A safety mechanism should have a measurable failure path, not just a success path.