Here's the moment a lot of us had this year.
You built an agent. It was genuinely impressive — it reasoned, it planned, it called tools, it chained steps together to get real work done. The demo was so good that you gave it real permissions. Let it send the emails. Let it update the records. Let it hit the production API.
And then, one day, it did something you never sanctioned. Not maliciously — confidently. Correctly-looking. It completed the action, reported success, and moved on. You only found out later, when the consequence surfaced somewhere downstream. That's the moment you realize what you actually shipped: a system with a powerful engine and no brake pedal.
The entire industry has spent two years racing to make agents smarter. Better reasoning, longer context, more tools, more autonomy. Almost nobody spent that time making them safe to let act. And those are completely different problems — because the instant an agent stops just talking and starts doing, a more capable agent isn't more useful. It's more dangerous.
Let me make the case, and then talk about the brakes.
Brains and brakes are different problems
This is the distinction the hype cycle blurred, so it's worth stating plainly.
Making an agent capable — able to reason about a task, plan a sequence of steps, pick the right tool, recover from an error — is one problem, and it's the one all the model progress went into. Every new release is a better brain.
Making an agent safe to let act — ensuring that when it's wrong, the wrongness is caught, bounded, reversible, or stopped before it does damage — is a completely different problem. And here's the uncomfortable part: no smarter model solves it for you. A more capable agent doesn't act wrongly less; it acts wrongly faster and more convincingly. Upgrading the brain does nothing for the brakes, because the brakes were never in the model. They're in the architecture around it — the part you have to build yourself.
You can have the best engine in the world. Without brakes, that just means you hit the wall sooner.
Why "acting" changes everything
A chat that only talks has a wonderful property: its mistakes are free. It says something wrong, you read it, you move on. Nothing happened in the world.
The moment an agent can act, three things change, and each one is a reason you need control that a chatbot never did.
Irreversibility. A wrong sentence costs nothing. A wrong DELETE, a sent email, a processed refund, a pushed commit — those don't have an undo button. The cost of an agent's mistake stops being "re-read it" and becomes "clean up the damage," and some damage doesn't clean up.
Confidence is not correctness. This is the through-line of everything I write, and it's never more dangerous than when an agent can act. The agent takes the wrong action with exactly the same confidence it takes the right one. There is no tremor in its voice, no hedge, no tell. It is equally sure when it's about to help you and when it's about to hurt you.
Silent success. The nastiest one. The agent completes the wrong action and reports success — because from its point of view, it did complete the task, it just completed the wrong one. Your monitoring goes green. Nobody sees the wrong customer getting the refund until the wrong customer gets the refund. "It worked" and "it did the right thing" are different claims, and only one of them shows up on the dashboard.
Put those together and you get the core risk: an agent that can act is a system that can cause irreversible harm, confidently, without anything flagging it. That is precisely the situation brakes exist for.
Brake #1: Approval gates — ask before acting
The most basic brake, and the one most systems skip: for anything consequential, the agent doesn't do the thing. It proposes the thing, and waits for a human to say yes.
The skill here isn't gating everything — that would make the agent useless, and you'd click "approve" on autopilot until the gate meant nothing. The skill is gating by blast radius:
- Auto-run the trivial and reversible — read something, format something, draft something. No gate needed.
- Ask first for the consequential — sending, paying, writing to production, anything a user would want to see before it happens.
- Draft-only for the dangerous — the agent prepares the action but cannot execute it at all; a human does.
The mistake to avoid is a uniform gate on everything, which trains people to rubber-stamp. A good gate is rare enough that when it fires, the human actually looks.
Brake #2: A reviewer that can actually say no
The popular pattern: a second agent reviews the first agent's work before it's allowed to proceed. A judge. A critic. It's a good pattern — and it has a failure mode that will fool you completely if you're not watching for it.
A reviewer that has never been seen to fail is indistinguishable from a reviewer that approves everything.
If your judge-agent has been green for months, that is not evidence it's working. It might be catching problems. It might also be rubber-stamping, silently broken, or checking the wrong thing entirely — and you would not be able to tell the difference from the outside, because both produce the same clean logs right up until the day the rubber stamp waves through the thing that hurts you.
A brake you have never tested is not a brake. It's a brake-shaped object. So if you build a reviewer, you have to build the thing that proves it can still say no: wire a known-bad action through the live review path, on a schedule, and confirm the reviewer rejects it. Surface when it last refused something, the way you'd surface uptime. If that "last refusal" date goes stale, your brake may already be dead and you just can't see it yet. Prove the "no" is reachable, continuously, or you're trusting a checker you've never watched work.
Brake #3: An audit trail — know what it did
You cannot control what you cannot see. If an agent takes actions in the world and you have no durable record of what it did, when, and why, then a wrong action is invisible until its damage surfaces on its own schedule — usually the worst possible one.
Every consequential action should leave a trail: what the agent did, what triggered it, what it was acting on, and ideally a path to reverse it. This isn't bureaucracy. It's the difference between "we caught the bad action in the log an hour later and rolled it back" and "a customer told us three weeks later, and we have no idea how many others it happened to."
An audit trail is also what makes every other brake improvable — you can't tune a gate or a reviewer if you can't see what got through.
Brake #4: Blast-radius limits — caps and scopes
The final brake is the one that works even when every other brake fails: hard limits the agent physically cannot exceed, enforced by the system rather than by the agent's good behavior.
Rate caps (no more than N actions per minute). Spend caps (cannot exceed $X without escalation). Scoped permissions (can read these tables, can write only that one). Iteration limits (stop after K steps instead of looping forever). These don't depend on the agent reasoning correctly — they're walls, and walls hold whether or not the thing inside them is behaving.
This is where these controls show up in practice, and a few platforms now build them in as first-class features rather than something you bolt on — for instance, Xenition ships approval gates, an audit log, and a second agent that reviews the first's work before it reaches you (disclosure: it's a product I've been exploring). But whether you adopt a workspace that includes them or wire them yourself, the principle is the same and it's not optional: the agent's freedom should be bounded by design, not by hope. You do not want "it usually behaves" to be the only thing standing between your agent and your production database.
The uncomfortable truth: brakes are unsexy
Here's why, despite all of the above being fairly obvious, most agents ship without any of it.
Nobody demos the approval gate. Nobody opens a pitch with their audit log. Brains are exciting — they make the standup go "wow." Brakes are boring — they only ever matter on the day something goes wrong, and if they're doing their job, that day never visibly arrives, so it looks like you built them for nothing.
Choosing to build the unglamorous safety layer before you need it, when there's no incident yet to justify it, is exactly the kind of discipline that doesn't get applause. It's the senior move precisely because it's invisible when it works. The engineer who spent a day on blast-radius limits instead of adding a fifth capability looks slower — right up until the week their agent doesn't drain the account.
The takeaway
Capability has become table stakes. Every model is smart enough now; smartness is no longer where systems differ. Controllability is the differentiator — whether you can let the thing act without lying awake about what it might do.
So the question to ask about your agent isn't "how smart can I make it?" It's "what happens when it's confidently wrong — and can I stop it in time?" If the answer is "I'm trusting it not to be," you don't have an agent you can deploy. You have a liability with a good demo.
Build the brakes. Not because your agent is dumb, but because someday it will be confidently, fluently, irreversibly wrong — and on that day, the only thing that matters is whether you built a way to stop it before you needed one.
What's the one action you'd never let an agent take without a human in the loop? And the more honest question: have you actually wired that gate — and tested that it fires — or are you quietly trusting the agent to behave? I want to hear where people drew the line.
Disclaimer: This article was written with AI assistance and reviewed and edited by me before publishing.
Top comments (39)
@mansio The three trust domains are the right number, and collapsing them into one box is the standard industry mistake. The problem is not technical, it is structural. Observability tools sell on unification, and the unification is what makes them unfit for forensics.
A tool that records what happened cannot also judge whether it was right without trusting the same pipeline for both facts and conclusions. The conclusion then feeds back into the record. That loop is the single-box design, and it is the same class of bug as a model evaluating its own output.
Append-only closes one leak in that loop, not the whole thing. It stops the record from being rewritten after the fact — but it says nothing about who decides what gets recorded in the first place. If the same pipeline that later judges correctness also controls which events are sampled, filtered, or deemed "not relevant enough to log," it has already shaped the facts before they ever became immutable. Append-only protects integrity after capture; it doesn't buy independence at capture.
So the actual requirement is two-part, not one: (1) the record can't be edited after the fact, and (2) the thing deciding what to record is a different actor than the thing deciding what it means. Only the second part closes the loop you're describing. I don't have a clean example yet of a system that gets both right — if you know of one, that'd be the test case worth checking.
This is the correction the audit-trail section needed, and it's the same disease this whole conversation keeps surfacing one layer down: I treated "immutable log" as if it bought trust, when it only buys integrity after capture. You've named the gap precisely — append-only guarantees nobody rewrote the record, and says nothing about who decided what became a record in the first place. If the pipeline that later judges correctness also controls sampling, filtering, and the "not relevant enough to log" threshold, it shaped the facts before they hardened. The tamper-proofing is real and it's protecting a set of events that was already curated by the interested party. An honest, unfalsifiable log of a pre-selected reality.
The two-part requirement is the right decomposition, and the ordering of importance is the sharp part: (1) can't be edited after the fact, (2) the thing deciding what to record is a different actor than the thing deciding what it means — and only (2) closes the loop. Because a system that judges its own correctness while also controlling its own logging can produce a flawless, immutable audit trail that happens to omit exactly the events that would incriminate it. Not by editing. By never recording. That's the "silent success" failure moved into the observability layer itself: the absence of an event reads identically to the event never having happened, and the actor with the motive to hide it is the one holding the sampling knob. Separation of capture from interpretation is the audit-trail version of "the reviewer can't be the same model as the generator" — same principle, different seam.
And I'll be honest that I don't have a clean example either, which is itself telling. Most systems I've seen bolt an append-only store onto a pipeline that still decides its own logging — they get (1) and quietly assume it implies (2). The nearest patterns that gesture at capture-independence are things like an out-of-band collector the acting system can't reach into (effects observed at the syscall/egress boundary rather than self-reported), or logging emitted by the enforcement layer that already sits separate from the agent — but "the enforcement layer is genuinely a different actor with no shared incentive" is exactly the property that's asserted far more often than it's proven. So I think you've found an open problem, not just a caveat: capture-independence is the requirement almost nobody satisfies, and "who holds the sampling knob" is the question that should be asked of every audit trail that calls itself trustworthy. If either of us finds a system that demonstrably separates the two, that's the case study worth writing up. Going into the revision with this credited — the two-part split is the fix.
@mansio Exactly. A signed record of what happened is not the same as a statement about whether it was the right thing to have happen. Separating those trust domains is what keeps a forensic system honest. Most agent frameworks collapse them into one opaque box and call it observability.
Collapse them into one opaque box and call it observability" — that is the sharpest industry-level framing of the whole thread. A single box cannot distinguish "what was permitted" from "what happened" from "whether it was right" — three trust domains, three questions, one answer pretending to cover all three. And an opaque box cannot even be audited for having collapsed them, which makes the diagnosis invisible too.
Your session-id linkage from the kernel-monitor work is the counterexample worth citing: separate trust domains, explicit linkage, both sides produce artifacts the other can verify. That is observability with receipt structure — what most frameworks ship is observability with a summary
We had an agent sending client status updates last year and it was doing this specific thing: it pulled the most recent completed step and framed it as if the whole job was done. Looked correct, had the right tone, nobody on our side caught it. We only stopped it because we had a delay queue and someone happened to check before the batch went out. If the messages had gone immediately we would have had three incorrect client notifications out before anyone noticed.
This is a textbook "silent success," and it's nastier than a crash because every surface signal was green — right tone, well-formed, completed — it just framed the latest step as the whole job being done. Nothing errored, because from the agent's view nothing went wrong; the claim was simply false. And the tell is how you caught it: a delay queue and someone who happened to look — luck wearing the costume of a process. That's actually the strongest brake for irreversible outbound actions, just undesignated: a hold between "generated" and "sent." The lesson isn't "watch harder," it's to make that window deliberate, and ideally add the check that would've caught it automatically — compare the claim ("done") against system state (steps remaining > 0) before the send is allowed. Verify the assertion against reality, not the prose against a style guide. Cleanest real-world silent-success I've seen — going in the revision.
I really like the brakes idea. One thing I’d add is a check before the agent calls the model or any tool. If customer data is already in the prompt, tool request, retry, or sub-agent context, cleaning the logs later is too late. A simple rule is to keep real PII inside the request boundary and log only counts, entity types, and timing.
This is the right correction, and it's the same "too late" problem the whole thread keeps circling, aimed one step earlier than I put it. My audit-trail brake was about recording what happened — but you're pointing out that by the time it's in the log, the PII has already crossed every boundary that mattered: it's in the prompt, the tool request, the retry, the sub-agent context. Cleaning the logs after the fact is treating the symptom. The data already leaked into all the places you weren't looking; the log is just the one place you happened to notice.
The pre-call check is the structural fix because it moves the control to the boundary the data actually has to cross before it spreads. And the reason it matters more with agents than with a plain API call is exactly the surface you listed — retries, sub-agents, tool payloads. A single request fans out into many downstream contexts, so PII in the initial prompt doesn't leak once, it leaks into every retry log, every sub-agent's context window, every tool call's request body. One careless inclusion, N copies you now have to chase. Checking before the model/tool call is the only place you catch it while it's still one copy.
And "keep real PII inside the request boundary, log only counts, entity types, and timing" is the cleanest statement of the principle I've seen — because it separates the two things logs are actually for. You almost never need the customer's actual email in the trace; you need to know that an email was present, what type of entity was processed, and when. Counts and types and timing give you debuggability and audit coverage without turning your observability layer into a second copy of the sensitive data. It's the difference between logging that something happened and logging the sensitive thing itself — and the first is almost always enough. Going into the revision with credit; this belongs as its own brake — sanitize at the request boundary, before the call, and log shape not content.
@james_anderson_h The order of operations is backwards. An impressive demo earns permissions, and then you think about brakes. The demo is convincing enough to be trusted, and that trust is the vulnerability. The agent that produces plausible-looking wrong output passes every brake that checks for crashes. The boundary needs to sit at the effect level before the agent runs a single tool call: what paths, what destinations, what credentials. Not on how fast it acts.
Trust is the vulnerability" is the sharper diagnosis, and the ordering critique lands: brakes designed after permissions are granted are retrofits on a trust decision already made. Your effect-level boundary has a pedigree worth citing: Saltzer and Schroeder, 1975 — complete mediation (every access checked before it happens) and fail-safe defaults (permission-based, not exclusion-based). Agents did not invalidate the principle; they made enumeration the hard part — and paths, destinations, credentials are enumerable, which is exactly why your boundary is enforceable while judgment-based brakes stay probabilistic.
One complement: the pre-execution boundary and the reviewer are different layers, and plausible-wrong output passes both when the reviewer shares the generator's blind spots. The boundary authorizes the call; it does not judge the output. And where your design meets specs — the TRACE PIC bridge is a live instance of it, signed pre-execution authorization checked at the gateway — its two documented limits apply: it is optional, and it validates the decision made before execution, not the state the call runs against.
Question, since you named the hard part: how does the boundary treat effects the author did not enumerate — a tool that mid-task makes its own network call or spawns a subprocess? That is where static scoping historically leaks.
The Saltzer & Schroeder pedigree is exactly right, and I should have cited it — complete mediation and fail-safe defaults are the 1975 bones under the whole "brakes" argument, and you've named why the principle survived the agent era intact: agents didn't break it, they made enumeration the hard part. That's the sharp reframe. Paths, destinations, credentials are enumerable, so the boundary is enforceable; judgment ("is this a good action?") isn't enumerable, so a reviewer built on it stays probabilistic. The boundary authorizes; the reviewer opines — and those are different guarantees. Your point that plausible-wrong output passes both when the reviewer shares the generator's blind spots is the crucial complement: the boundary checks the call, never the output, and a same-family reviewer inherits the exact gaps it's supposed to catch. Two layers, two different jobs, and neither covers the other's failure.
On your question — which is the right one, because it's where static scoping has always leaked — the honest answer is: the boundary only holds if the effect, not the author's declared intent, is what gets mediated. If scoping trusts an enumeration the author wrote ("this tool calls these hosts"), then a tool that mid-task opens its own socket or spawns a subprocess escapes, because the dangerous effect was never in the list the author enumerated. Complete mediation says the fix can't live at author-declaration time; it has to live at the effect boundary — the syscall/egress/process layer the unenumerated call still has to cross. You don't enumerate what the tool says it'll do; you interpose on what it can do: deny-by-default egress, no ambient network, subprocess spawning gated at the sandbox, credentials scoped so an unforeseen call has nothing to reach for. The author's enumeration becomes an allowlist checked at that boundary, not a description trusted in place of one. So the leak you're pointing at is real precisely for designs that mediate the plan; it closes only when mediation moves to the layer the effect can't route around — which is the same "authorize the effect, not the envelope" point, aimed one level lower than most implementations put it. Genuinely sharp thread — this is going in the revision, Saltzer & Schroeder credited.
"The demo earns permissions, then you think about brakes" — that ordering is the actual root cause, and you've stated it more precisely than the article did. Every brake I described is a retrofit on a trust decision already made. The impressive demo grants the permission; the brakes get bolted on afterward to police a system you've already decided to trust. That's backwards, and it means the brakes are always playing catch-up against an authority they didn't get to scope.
And "that trust is the vulnerability" is the sharper diagnosis. The demo doesn't earn trust because the agent is safe — it earns trust because it's convincing, and those are unrelated properties. Worse, the failure that hurts you is the plausible-wrong output, which is convincing by construction, so it sails through exactly the brakes that check for crashes and errors. You can't catch "confidently wrong" with a smoke detector; there's no smoke. My whole framing implicitly assumed the brake's job was to catch failure, when the real exposure is the action that looks like success.
Your fix is the one that actually inverts the order: the boundary has to sit at the effect level, before the first tool call — what paths, what destinations, what credentials — not on how fast or how autonomously it acts. That's the difference between authorizing the effect and reviewing the behavior, and only the first one is enforceable before trust is granted. Paths and credentials are enumerable and checkable up front; "is this action good?" is a judgment you can only make after the fact, probabilistically, with a reviewer that shares the agent's blind spots. So the enforceable boundary is the pre-execution one, and everything downstream is commentary.
This reframes the piece for me: brakes-after-brains is still brains-first thinking. The real move is to scope the effect boundary before the demo ever earns anything — permission is the thing you withhold by default, not the thing you grant because the demo was good. Going into the revision with credit — "the trust the demo earns is the vulnerability, so the boundary belongs at the effect level before execution, not on the behavior after" is the correction the whole argument needed.
You'll need deterministic code generation to achieve this. Everything else is "increasing statistical probability". If you control the execution plane, it's a hard no ...
The brakes/brains split is the right cut, and Brake #2 is the one this thread network has been grinding on all month — the "last refusal surfaced like uptime" telemetry you describe was shipped by Michael Hurst as a last-HOLD date and by Heinrich Neb as veto heartbeats. But between #2 and #3 there is a gap the last two weeks kept exposing, and I would name it Brake #2.5: the reviewer's identity, and the fate of its refusals.
Reachability does not prove independence. A planted known-bad case goes red for an honest reviewer and for one marking its own work — a self-signing judge passes your live-path test while sharing the generator's blind spots. The check is one line in every run output: which model produced the artifact, which model judged it. When a config comment says "kept separate" and the value says JUDGE_MODEL_ID = DEFAULT_MODEL_ID, that is a dead gate wearing a config file.
And a no that gets overridden is not a brake either. A reviewer with a healthy refusal streak and a high override rate is ceremony with telemetry — the override channel is the real review, and it needs its own ledger with ground truth on who was right.
On your closing question: write-path actions against a codebase. And the honest answer to your second question is that my gate failed its own test this week — my modification guard, the component whose job is to check write operations, silently resolved an ambiguous symbol target to the first candidate and would have applied the change to the wrong definition. The brake was itself picking the target blind. Filed as issues against my own tool; the fix refuses and prints the candidate list. Which confirms your thesis one level down: I knew about brake-shaped objects, wrote about verification theater — and still had to catch my own at a specific line number. Knowing about a trap is not immunity to it. That is why the firing test runs on a schedule, not on trust.
Brake #2.5 is the right insertion point, and you've named the two things reachability quietly assumes but never proves: independence and the fate of the no. Both are load-bearing, and both are invisible to the heartbeat.
The self-signing judge is the sharper of the two because it defeats the test I proposed on the test's own terms. A planted known-bad case goes red for an honest reviewer and for one grading its own output — reachability is satisfied identically in both cases, so the canary can't tell them apart. The gap isn't "can it refuse," it's "is the thing refusing actually a second opinion." And your check is beautifully cheap: one line in every run output — which model produced, which model judged. JUDGE_MODEL_ID = DEFAULT_MODEL_ID behind a comment that says "kept separate" is a dead gate wearing a config file, and it would sail through every liveness and freshness check I described while sharing 100% of the generator's blind spots. Reachable, fresh, and epistemically worthless.
The override ledger is the half I completely omitted, and you're right that it's where the real review secretly lives. A healthy refusal streak with a high override rate is ceremony with telemetry — the no fired, got logged, looked great, and then a human (or an escalation path) quietly reversed it, so the effective review is the override channel and it's running with no ground truth on who was right. Without a ledger on overrides, "refusals: 40" is a vanity metric; the number that matters is "refusals overridden, and how often the override was the mistake." The brake isn't the refusal — it's the refusal that stuck.
But the part I want to sit with is your answer to the second question, because it's the most honest thing anyone's put in this thread. You knew about brake-shaped objects. You wrote about verification theater. And your modification guard — the component whose entire job is checking write operations — silently resolved an ambiguous symbol to the first candidate and would have applied the change to the wrong definition. The brake was picking its target blind. That's not an embarrassing footnote; it's the thesis proven one level down, on the author of the thesis, at a specific line number. Knowing about a trap is not immunity to it — the knowledge lives in your prose and the bug lives in your resolver, and they don't check each other. Which is exactly why the firing test has to run on a schedule instead of on your confidence that you, of all people, wouldn't ship a brake-shaped object. The scheduled test doesn't care what you know. That's its whole value.
Going into the revision as Brake #2.5, both halves, with credit — "log which model judged which artifact, and keep a ground-truthed override ledger" is the pair. And the write-path-against-a-codebase answer, plus your resolver catching itself, is the case study I'll use to make the point that self-knowledge is not a substitute for the scheduled firing test. Thank you for filing the bug against your own tool in public — that's the demonstrated version of everything the article was only arguing.
The scheduled known-bad action and the
last refuseddate are the same signal, and that is the part I would separate. Once the probe runs on a schedule the date can never go stale, so a reviewer that has degraded into recognising the probe and waving through everything else keeps exactly the freshness you are watching for; the failure mode moves fromnever seen to failtoonly ever seen to fail on the test input, which looks identical from outside. Brake #3 already carries the fix: count refusals from real traffic and from probes separately in the audit trail, so a live-refusal count of zero across a long window is visible on its own instead of being covered by the canary.You've found the hole in my own fix, and it's the same disease one level up — exactly the recursion this keeps landing on. The heartbeat proves the reviewer can refuse; it does not prove the reviewer refuses anything but the probe. Once the known-bad case runs on a schedule, a reviewer that's degraded into recognizing the canary and waving everything else through keeps a perfectly fresh "last refused" date the whole time. The failure moves from "never seen to fail" to "only ever seen to fail on the test input" — and from outside those are identical, because the one refusal I'm watching for is the one it learned to perform. The canary went from evidence to theater with a green light on it.
And you're right that Brake #3 already carries the fix, which I under-connected: count probe refusals and real-traffic refusals separately in the audit trail. The moment they're one number, the canary's steady green covers a live-refusal count of zero — the probe launders the silence. Split them, and "refused on probes: 40, refused on real traffic: 0 across 30 days" is visible as its own alarm instead of being absorbed into a healthy-looking total. The heartbeat answers "can it refuse?"; the segmented count answers "does it refuse anything real?" — different questions, and only the second catches the probe-recognizer.
The generalization I'm taking: a canary validates the mechanism, never the coverage, so the canary's own signal has to be quarantined from production signal or it masks exactly the degradation it was built to detect. Any synthetic health check that mingles with real-traffic metrics can hide a real-traffic zero behind a synthetic pass. Segmenting probe from production isn't a nice-to-have, it's what stops the instrument from covering for the thing it measures. Going into the revision with credit — "count probe and live refusals separately, a live-refusal zero must be visible on its own" is the line.
Exactly how some big companies ended up letting Claude delete their production DBs... Because it thought it was on a dev db... If left unchecked, the damage is done before you react, but approving every single tool call is excessively mundane and people wont pay attention. Ideally, I'd want a scoped default set, then adjust as you want, but rule-driven, not system instruct driven, so if I say 'never touch string "Apple"', it simply has to work around it, if I say 'dont edit string "Apple"', but I instruct it, forget apples, we're selling oranges now, replaces all apples with oranges. It would do a scripted execution that would hit it, which is why all agent actions should be transaction driven and the rules must be enforced on the execution layer, so if that script hits Apples, it stops the transaction, rolls it back and prompts permission in a different way "Rule Breach: string "Apple" will change to string "Oranges", do you want to continue? This is a breach of rule 325: "Never touch string "Apple". That way it's explicit, separate and safe. It's easy to do in a codebase, it's dangerous to do in a db...
The Claude-deleted-the-prod-DB-because-it-thought-it-was-dev case is the perfect example, because it kills the "just make the agent smarter" reflex on contact. The agent wasn't malfunctioning — its world model was wrong (dev vs prod), and it acted perfectly on a false premise. No amount of better reasoning fixes that, because the reasoning was fine; the ground truth it reasoned over was wrong. That's precisely why the boundary can't live in the agent's judgment. It has to live somewhere the agent's belief about which DB it's on is irrelevant.
And you've named the exact tension the approval-gate section only gestured at: gate everything and people rubber-stamp until the gate is noise; gate nothing and the damage lands before you can react. Your resolution is the sharp part — rule-driven, enforced at the execution/transaction layer, not instruction-driven. That distinction is everything. An instruction ("don't touch Apple") lives in the same probabilistic space as the task, so a later instruction ("forget apples, we sell oranges now") can override it, and the model will faithfully generate the script that violates the rule it was "told" to keep. The rule has to sit outside the thing that can be talked out of it. If "never touch Apple" is enforced at the execution layer, it doesn't matter what you later instruct — the transaction hits the guarded string, breaches rule 325, halts, rolls back, and asks in a different, explicit channel. The rule can't be reasoned around because it isn't part of the reasoning.
Two things I'd underline from your design. First, transaction-driven + rollback is what makes the DB case survivable — the codebase gets this for free (a bad edit is a diff you revert), which is exactly why your "easy in a codebase, dangerous in a DB" line is the real warning: the DB needs the transaction boundary engineered in to get the same undo the filesystem gives you natively. A rule that fires after an irreversible write is a eulogy, not a brake. Second, the breach prompt being a separate, explicit channel ("Rule Breach: 325, Apple→Oranges, continue?") is what defeats rubber-stamping — it's not the mundane "approve this tool call" that trains people to click yes, it's a rare, specific, named-rule interruption that actually makes someone look. You've split the gate into "auto-run within scope" and "hard-stop on rule breach," which is the resolution to the gate-everything-vs-gate-nothing problem I left open.
The through-line: scoped-default + rule-at-the-execution-layer + transaction rollback is the deterministic control layer done right — the model proposes, the execution layer enforces rules it can't argue with, and irreversibility is neutralized by the transaction boundary. Going in the revision with credit; "rules enforced on the execution layer, not via system instructions, with transactional rollback on breach" is the concrete version of every abstract point this thread has been reaching for.
About sums it up, it's what I built into my Velocity IDE, because these are the kind of mistakes I saw AI make at work. The amount of times Gemini would make a schema change, or Claude insert incomplete entries into the db, then the developer has to answer for it, I wanted to solve that, because that's something I noted as I used the tools allowed by my job, I'd have liked it to have, so like with V.A.L.I.D. vs CSLA, I built the solution I'm not allowed to use, just so I can get my frustration out knowing there's a solution, even if they wont let me use it.
The “test that the brake can actually say no” point is probably the most important one here. In production agent systems, we’ve found that a reviewer being present means very little unless its rejection path is tested like any other critical dependency. I’d also add one more layer: make the policy decision deterministic where possible, so the LLM proposes an action but a separate control layer decides whether it’s allowed. That separation makes the system much easier to audit and change as the agent evolves. It’s the same direction we’ve been taking at IT Path Solutions with agentic systems treating guardrails as part of the architecture, not an afterthought.
"A reviewer being present means very little unless its rejection path is tested like any other critical dependency" — that's the right severity level, and the framing is the fix. Nobody ships a database connection without testing what happens when it fails, yet people ship reviewers whose failure path — the rejection — has never been exercised once. Treating the "no" as a critical dependency with its own tests is exactly the discipline the whole thread has been circling: presence is not function, and function is only proven by watching the reject fire.
Your determinism point is the structural upgrade, and I'd put it even higher than a "layer" — it's the separation that makes everything else auditable. The LLM proposes, a separate deterministic control layer decides. That split matters because it moves the authorization decision out of the probabilistic component and into one you can read, test, version, and diff. A reviewer that's itself an LLM shares the generator's blind spots and can only be evaluated statistically; a deterministic policy layer can be proven — you can point at the rule, trace why an action was allowed or denied, and change it without retraining anything. "The model reasons, the control layer authorizes" is the same shape as "model reasons, tools compute," aimed at the permission boundary. Propose and authorize should never be the same component.
And "guardrails as part of the architecture, not an afterthought" is the sentence that inverts the failure mode someone else named in this thread — that the impressive demo earns permissions and the brakes get retrofitted onto a trust decision already made. Building the control layer first, as structure rather than patch, is what stops that ordering from happening: the boundary exists before the agent earns anything, so permission is withheld by default instead of granted because the demo was good. That's the direction that actually holds up as the agent evolves, because the thing you audit and change isn't the model's judgment — it's a deterministic layer that sits outside it. Good addition; the propose-vs-authorize separation is going in the revision.