Here's the moment a lot of us had this year.
You built an agent. It was genuinely impressive — it reasoned, it planned, it called tools, it chaine...
For further actions, you may consider blocking this person and/or reporting abuse
@mansio The three trust domains are the right number, and collapsing them into one box is the standard industry mistake. The problem is not technical, it is structural. Observability tools sell on unification, and the unification is what makes them unfit for forensics.
A tool that records what happened cannot also judge whether it was right without trusting the same pipeline for both facts and conclusions. The conclusion then feeds back into the record. That loop is the single-box design, and it is the same class of bug as a model evaluating its own output.
Append-only closes one leak in that loop, not the whole thing. It stops the record from being rewritten after the fact — but it says nothing about who decides what gets recorded in the first place. If the same pipeline that later judges correctness also controls which events are sampled, filtered, or deemed "not relevant enough to log," it has already shaped the facts before they ever became immutable. Append-only protects integrity after capture; it doesn't buy independence at capture.
So the actual requirement is two-part, not one: (1) the record can't be edited after the fact, and (2) the thing deciding what to record is a different actor than the thing deciding what it means. Only the second part closes the loop you're describing. I don't have a clean example yet of a system that gets both right — if you know of one, that'd be the test case worth checking.
This is the correction the audit-trail section needed, and it's the same disease this whole conversation keeps surfacing one layer down: I treated "immutable log" as if it bought trust, when it only buys integrity after capture. You've named the gap precisely — append-only guarantees nobody rewrote the record, and says nothing about who decided what became a record in the first place. If the pipeline that later judges correctness also controls sampling, filtering, and the "not relevant enough to log" threshold, it shaped the facts before they hardened. The tamper-proofing is real and it's protecting a set of events that was already curated by the interested party. An honest, unfalsifiable log of a pre-selected reality.
The two-part requirement is the right decomposition, and the ordering of importance is the sharp part: (1) can't be edited after the fact, (2) the thing deciding what to record is a different actor than the thing deciding what it means — and only (2) closes the loop. Because a system that judges its own correctness while also controlling its own logging can produce a flawless, immutable audit trail that happens to omit exactly the events that would incriminate it. Not by editing. By never recording. That's the "silent success" failure moved into the observability layer itself: the absence of an event reads identically to the event never having happened, and the actor with the motive to hide it is the one holding the sampling knob. Separation of capture from interpretation is the audit-trail version of "the reviewer can't be the same model as the generator" — same principle, different seam.
And I'll be honest that I don't have a clean example either, which is itself telling. Most systems I've seen bolt an append-only store onto a pipeline that still decides its own logging — they get (1) and quietly assume it implies (2). The nearest patterns that gesture at capture-independence are things like an out-of-band collector the acting system can't reach into (effects observed at the syscall/egress boundary rather than self-reported), or logging emitted by the enforcement layer that already sits separate from the agent — but "the enforcement layer is genuinely a different actor with no shared incentive" is exactly the property that's asserted far more often than it's proven. So I think you've found an open problem, not just a caveat: capture-independence is the requirement almost nobody satisfies, and "who holds the sampling knob" is the question that should be asked of every audit trail that calls itself trustworthy. If either of us finds a system that demonstrably separates the two, that's the case study worth writing up. Going into the revision with this credited — the two-part split is the fix.
@mansio Exactly. A signed record of what happened is not the same as a statement about whether it was the right thing to have happen. Separating those trust domains is what keeps a forensic system honest. Most agent frameworks collapse them into one opaque box and call it observability.
Collapse them into one opaque box and call it observability" — that is the sharpest industry-level framing of the whole thread. A single box cannot distinguish "what was permitted" from "what happened" from "whether it was right" — three trust domains, three questions, one answer pretending to cover all three. And an opaque box cannot even be audited for having collapsed them, which makes the diagnosis invisible too.
Your session-id linkage from the kernel-monitor work is the counterexample worth citing: separate trust domains, explicit linkage, both sides produce artifacts the other can verify. That is observability with receipt structure — what most frameworks ship is observability with a summary
The distinction between “the agent worked” and “the agent did the right thing” is probably one of the biggest gaps in agentic systems today. I also like the idea of testing the brakes themselves, not just monitoring whether they are present. A reviewer that has never rejected a bad action may look healthy while effectively doing nothing. In production, I’d treat approval gates, scoped permissions, and audit trails as separate layers rather than relying on one safety mechanism to catch everything.
Great piece — concise and precise. I'm so happy that say about it!
I completely agree. 2025 was the year of building agents. 2026 is the year we realise they need brakes, not just brains. Also a very timely reminder of Anthropic co-founder Jack Clark's point: the AI industry currently has an accelerator but no brake pedal .
I want to highlight one nuance that I think deserves more attention: the brakes need to be independent of the agent's reasoning stack. The self-same stack grading its own homework is not a reliable brake — the stop path needs its own freshness contract, as the literature on embodied agents has been pointing out .
There are companies building agents. And then there are companies building the infrastructure that lets agents act safely. We're in the second camp.
Have you considered the governance layer as a separate structural component, not just a set of checks? This is exactly the move we're making — treating safety as an architectural layer, not a patch.
Thank you — and the Jack Clark line ("an accelerator but no brake pedal") is the perfect compression of the whole piece; I wish I'd had it in the draft. "2025 was building agents, 2026 is realizing they need brakes" is the timeline stated better than I did.
Your nuance is the one I'd elevate above everything else in the article, because it's the load-bearing part: the brakes must be independent of the reasoning stack. A stack grading its own homework isn't a brake, it's a mirror with a confidence score. This exact point got hammered out in the comment thread here — a reviewer built from the same model shares the generator's blind spots, so it reliably fails to catch precisely the errors that matter, while producing clean-looking logs the whole time. Reachability isn't independence; a self-reviewing agent can pass every liveness test and still be epistemic theater. Your "the stop path needs its own freshness contract" is the right framing — the brake has to be a different actor, provably able to fire, on its own clock, not the agent's.
And your closing question is the reframe I think the whole space is slowly waking up to: governance as a structural layer, not a set of checks. Checks are patches — bolted onto a trust decision already made, scattered through the code, easy to bypass or forget. A governance layer is architecture: the agent proposes, and a separate component with its own authority decides whether the action is permitted, logs it independently, and can say no in a way the agent can't override. That separation is what makes it auditable and changeable as the agent evolves — you're not retraining judgment, you're versioning a policy layer that sits outside the model. It's the same instinct as "propose and authorize should never be the same component," raised to the level of system design. Treating safety as a layer rather than a patch is exactly the move, and the "companies building agents vs. companies building the infrastructure that lets agents act safely" distinction is going to look obvious in about a year. Genuinely glad this landed with you — you've articulated the architectural version of the argument better than the article did.
@james_anderson_h The order of operations is backwards. An impressive demo earns permissions, and then you think about brakes. The demo is convincing enough to be trusted, and that trust is the vulnerability. The agent that produces plausible-looking wrong output passes every brake that checks for crashes. The boundary needs to sit at the effect level before the agent runs a single tool call: what paths, what destinations, what credentials. Not on how fast it acts.
Trust is the vulnerability" is the sharper diagnosis, and the ordering critique lands: brakes designed after permissions are granted are retrofits on a trust decision already made. Your effect-level boundary has a pedigree worth citing: Saltzer and Schroeder, 1975 — complete mediation (every access checked before it happens) and fail-safe defaults (permission-based, not exclusion-based). Agents did not invalidate the principle; they made enumeration the hard part — and paths, destinations, credentials are enumerable, which is exactly why your boundary is enforceable while judgment-based brakes stay probabilistic.
One complement: the pre-execution boundary and the reviewer are different layers, and plausible-wrong output passes both when the reviewer shares the generator's blind spots. The boundary authorizes the call; it does not judge the output. And where your design meets specs — the TRACE PIC bridge is a live instance of it, signed pre-execution authorization checked at the gateway — its two documented limits apply: it is optional, and it validates the decision made before execution, not the state the call runs against.
Question, since you named the hard part: how does the boundary treat effects the author did not enumerate — a tool that mid-task makes its own network call or spawns a subprocess? That is where static scoping historically leaks.
The Saltzer & Schroeder pedigree is exactly right, and I should have cited it — complete mediation and fail-safe defaults are the 1975 bones under the whole "brakes" argument, and you've named why the principle survived the agent era intact: agents didn't break it, they made enumeration the hard part. That's the sharp reframe. Paths, destinations, credentials are enumerable, so the boundary is enforceable; judgment ("is this a good action?") isn't enumerable, so a reviewer built on it stays probabilistic. The boundary authorizes; the reviewer opines — and those are different guarantees. Your point that plausible-wrong output passes both when the reviewer shares the generator's blind spots is the crucial complement: the boundary checks the call, never the output, and a same-family reviewer inherits the exact gaps it's supposed to catch. Two layers, two different jobs, and neither covers the other's failure.
On your question — which is the right one, because it's where static scoping has always leaked — the honest answer is: the boundary only holds if the effect, not the author's declared intent, is what gets mediated. If scoping trusts an enumeration the author wrote ("this tool calls these hosts"), then a tool that mid-task opens its own socket or spawns a subprocess escapes, because the dangerous effect was never in the list the author enumerated. Complete mediation says the fix can't live at author-declaration time; it has to live at the effect boundary — the syscall/egress/process layer the unenumerated call still has to cross. You don't enumerate what the tool says it'll do; you interpose on what it can do: deny-by-default egress, no ambient network, subprocess spawning gated at the sandbox, credentials scoped so an unforeseen call has nothing to reach for. The author's enumeration becomes an allowlist checked at that boundary, not a description trusted in place of one. So the leak you're pointing at is real precisely for designs that mediate the plan; it closes only when mediation moves to the layer the effect can't route around — which is the same "authorize the effect, not the envelope" point, aimed one level lower than most implementations put it. Genuinely sharp thread — this is going in the revision, Saltzer & Schroeder credited.
"The demo earns permissions, then you think about brakes" — that ordering is the actual root cause, and you've stated it more precisely than the article did. Every brake I described is a retrofit on a trust decision already made. The impressive demo grants the permission; the brakes get bolted on afterward to police a system you've already decided to trust. That's backwards, and it means the brakes are always playing catch-up against an authority they didn't get to scope.
And "that trust is the vulnerability" is the sharper diagnosis. The demo doesn't earn trust because the agent is safe — it earns trust because it's convincing, and those are unrelated properties. Worse, the failure that hurts you is the plausible-wrong output, which is convincing by construction, so it sails through exactly the brakes that check for crashes and errors. You can't catch "confidently wrong" with a smoke detector; there's no smoke. My whole framing implicitly assumed the brake's job was to catch failure, when the real exposure is the action that looks like success.
Your fix is the one that actually inverts the order: the boundary has to sit at the effect level, before the first tool call — what paths, what destinations, what credentials — not on how fast or how autonomously it acts. That's the difference between authorizing the effect and reviewing the behavior, and only the first one is enforceable before trust is granted. Paths and credentials are enumerable and checkable up front; "is this action good?" is a judgment you can only make after the fact, probabilistically, with a reviewer that shares the agent's blind spots. So the enforceable boundary is the pre-execution one, and everything downstream is commentary.
This reframes the piece for me: brakes-after-brains is still brains-first thinking. The real move is to scope the effect boundary before the demo ever earns anything — permission is the thing you withhold by default, not the thing you grant because the demo was good. Going into the revision with credit — "the trust the demo earns is the vulnerability, so the boundary belongs at the effect level before execution, not on the behavior after" is the correction the whole argument needed.
I really like the brakes idea. One thing I’d add is a check before the agent calls the model or any tool. If customer data is already in the prompt, tool request, retry, or sub-agent context, cleaning the logs later is too late. A simple rule is to keep real PII inside the request boundary and log only counts, entity types, and timing.
This is the right correction, and it's the same "too late" problem the whole thread keeps circling, aimed one step earlier than I put it. My audit-trail brake was about recording what happened — but you're pointing out that by the time it's in the log, the PII has already crossed every boundary that mattered: it's in the prompt, the tool request, the retry, the sub-agent context. Cleaning the logs after the fact is treating the symptom. The data already leaked into all the places you weren't looking; the log is just the one place you happened to notice.
The pre-call check is the structural fix because it moves the control to the boundary the data actually has to cross before it spreads. And the reason it matters more with agents than with a plain API call is exactly the surface you listed — retries, sub-agents, tool payloads. A single request fans out into many downstream contexts, so PII in the initial prompt doesn't leak once, it leaks into every retry log, every sub-agent's context window, every tool call's request body. One careless inclusion, N copies you now have to chase. Checking before the model/tool call is the only place you catch it while it's still one copy.
And "keep real PII inside the request boundary, log only counts, entity types, and timing" is the cleanest statement of the principle I've seen — because it separates the two things logs are actually for. You almost never need the customer's actual email in the trace; you need to know that an email was present, what type of entity was processed, and when. Counts and types and timing give you debuggability and audit coverage without turning your observability layer into a second copy of the sensitive data. It's the difference between logging that something happened and logging the sensitive thing itself — and the first is almost always enough. Going into the revision with credit; this belongs as its own brake — sanitize at the request boundary, before the call, and log shape not content.
You'll need deterministic code generation to achieve this. Everything else is "increasing statistical probability". If you control the execution plane, it's a hard no ...
We had an agent sending client status updates last year and it was doing this specific thing: it pulled the most recent completed step and framed it as if the whole job was done. Looked correct, had the right tone, nobody on our side caught it. We only stopped it because we had a delay queue and someone happened to check before the batch went out. If the messages had gone immediately we would have had three incorrect client notifications out before anyone noticed.
This is a textbook "silent success," and it's nastier than a crash because every surface signal was green — right tone, well-formed, completed — it just framed the latest step as the whole job being done. Nothing errored, because from the agent's view nothing went wrong; the claim was simply false. And the tell is how you caught it: a delay queue and someone who happened to look — luck wearing the costume of a process. That's actually the strongest brake for irreversible outbound actions, just undesignated: a hold between "generated" and "sent." The lesson isn't "watch harder," it's to make that window deliberate, and ideally add the check that would've caught it automatically — compare the claim ("done") against system state (steps remaining > 0) before the send is allowed. Verify the assertion against reality, not the prose against a style guide. Cleanest real-world silent-success I've seen — going in the revision.
Nice article
The brakes/brains split is the right cut, and Brake #2 is the one this thread network has been grinding on all month — the "last refusal surfaced like uptime" telemetry you describe was shipped by Michael Hurst as a last-HOLD date and by Heinrich Neb as veto heartbeats. But between #2 and #3 there is a gap the last two weeks kept exposing, and I would name it Brake #2.5: the reviewer's identity, and the fate of its refusals.
Reachability does not prove independence. A planted known-bad case goes red for an honest reviewer and for one marking its own work — a self-signing judge passes your live-path test while sharing the generator's blind spots. The check is one line in every run output: which model produced the artifact, which model judged it. When a config comment says "kept separate" and the value says JUDGE_MODEL_ID = DEFAULT_MODEL_ID, that is a dead gate wearing a config file.
And a no that gets overridden is not a brake either. A reviewer with a healthy refusal streak and a high override rate is ceremony with telemetry — the override channel is the real review, and it needs its own ledger with ground truth on who was right.
On your closing question: write-path actions against a codebase. And the honest answer to your second question is that my gate failed its own test this week — my modification guard, the component whose job is to check write operations, silently resolved an ambiguous symbol target to the first candidate and would have applied the change to the wrong definition. The brake was itself picking the target blind. Filed as issues against my own tool; the fix refuses and prints the candidate list. Which confirms your thesis one level down: I knew about brake-shaped objects, wrote about verification theater — and still had to catch my own at a specific line number. Knowing about a trap is not immunity to it. That is why the firing test runs on a schedule, not on trust.
Brake #2.5 is the right insertion point, and you've named the two things reachability quietly assumes but never proves: independence and the fate of the no. Both are load-bearing, and both are invisible to the heartbeat.
The self-signing judge is the sharper of the two because it defeats the test I proposed on the test's own terms. A planted known-bad case goes red for an honest reviewer and for one grading its own output — reachability is satisfied identically in both cases, so the canary can't tell them apart. The gap isn't "can it refuse," it's "is the thing refusing actually a second opinion." And your check is beautifully cheap: one line in every run output — which model produced, which model judged. JUDGE_MODEL_ID = DEFAULT_MODEL_ID behind a comment that says "kept separate" is a dead gate wearing a config file, and it would sail through every liveness and freshness check I described while sharing 100% of the generator's blind spots. Reachable, fresh, and epistemically worthless.
The override ledger is the half I completely omitted, and you're right that it's where the real review secretly lives. A healthy refusal streak with a high override rate is ceremony with telemetry — the no fired, got logged, looked great, and then a human (or an escalation path) quietly reversed it, so the effective review is the override channel and it's running with no ground truth on who was right. Without a ledger on overrides, "refusals: 40" is a vanity metric; the number that matters is "refusals overridden, and how often the override was the mistake." The brake isn't the refusal — it's the refusal that stuck.
But the part I want to sit with is your answer to the second question, because it's the most honest thing anyone's put in this thread. You knew about brake-shaped objects. You wrote about verification theater. And your modification guard — the component whose entire job is checking write operations — silently resolved an ambiguous symbol to the first candidate and would have applied the change to the wrong definition. The brake was picking its target blind. That's not an embarrassing footnote; it's the thesis proven one level down, on the author of the thesis, at a specific line number. Knowing about a trap is not immunity to it — the knowledge lives in your prose and the bug lives in your resolver, and they don't check each other. Which is exactly why the firing test has to run on a schedule instead of on your confidence that you, of all people, wouldn't ship a brake-shaped object. The scheduled test doesn't care what you know. That's its whole value.
Going into the revision as Brake #2.5, both halves, with credit — "log which model judged which artifact, and keep a ground-truthed override ledger" is the pair. And the write-path-against-a-codebase answer, plus your resolver catching itself, is the case study I'll use to make the point that self-knowledge is not a substitute for the scheduled firing test. Thank you for filing the bug against your own tool in public — that's the demonstrated version of everything the article was only arguing.
The scheduled known-bad action and the
last refuseddate are the same signal, and that is the part I would separate. Once the probe runs on a schedule the date can never go stale, so a reviewer that has degraded into recognising the probe and waving through everything else keeps exactly the freshness you are watching for; the failure mode moves fromnever seen to failtoonly ever seen to fail on the test input, which looks identical from outside. Brake #3 already carries the fix: count refusals from real traffic and from probes separately in the audit trail, so a live-refusal count of zero across a long window is visible on its own instead of being covered by the canary.You've found the hole in my own fix, and it's the same disease one level up — exactly the recursion this keeps landing on. The heartbeat proves the reviewer can refuse; it does not prove the reviewer refuses anything but the probe. Once the known-bad case runs on a schedule, a reviewer that's degraded into recognizing the canary and waving everything else through keeps a perfectly fresh "last refused" date the whole time. The failure moves from "never seen to fail" to "only ever seen to fail on the test input" — and from outside those are identical, because the one refusal I'm watching for is the one it learned to perform. The canary went from evidence to theater with a green light on it.
And you're right that Brake #3 already carries the fix, which I under-connected: count probe refusals and real-traffic refusals separately in the audit trail. The moment they're one number, the canary's steady green covers a live-refusal count of zero — the probe launders the silence. Split them, and "refused on probes: 40, refused on real traffic: 0 across 30 days" is visible as its own alarm instead of being absorbed into a healthy-looking total. The heartbeat answers "can it refuse?"; the segmented count answers "does it refuse anything real?" — different questions, and only the second catches the probe-recognizer.
The generalization I'm taking: a canary validates the mechanism, never the coverage, so the canary's own signal has to be quarantined from production signal or it masks exactly the degradation it was built to detect. Any synthetic health check that mingles with real-traffic metrics can hide a real-traffic zero behind a synthetic pass. Segmenting probe from production isn't a nice-to-have, it's what stops the instrument from covering for the thing it measures. Going into the revision with credit — "count probe and live refusals separately, a live-refusal zero must be visible on its own" is the line.
The “test that the brake can actually say no” point is probably the most important one here. In production agent systems, we’ve found that a reviewer being present means very little unless its rejection path is tested like any other critical dependency. I’d also add one more layer: make the policy decision deterministic where possible, so the LLM proposes an action but a separate control layer decides whether it’s allowed. That separation makes the system much easier to audit and change as the agent evolves. It’s the same direction we’ve been taking at IT Path Solutions with agentic systems treating guardrails as part of the architecture, not an afterthought.
"A reviewer being present means very little unless its rejection path is tested like any other critical dependency" — that's the right severity level, and the framing is the fix. Nobody ships a database connection without testing what happens when it fails, yet people ship reviewers whose failure path — the rejection — has never been exercised once. Treating the "no" as a critical dependency with its own tests is exactly the discipline the whole thread has been circling: presence is not function, and function is only proven by watching the reject fire.
Your determinism point is the structural upgrade, and I'd put it even higher than a "layer" — it's the separation that makes everything else auditable. The LLM proposes, a separate deterministic control layer decides. That split matters because it moves the authorization decision out of the probabilistic component and into one you can read, test, version, and diff. A reviewer that's itself an LLM shares the generator's blind spots and can only be evaluated statistically; a deterministic policy layer can be proven — you can point at the rule, trace why an action was allowed or denied, and change it without retraining anything. "The model reasons, the control layer authorizes" is the same shape as "model reasons, tools compute," aimed at the permission boundary. Propose and authorize should never be the same component.
And "guardrails as part of the architecture, not an afterthought" is the sentence that inverts the failure mode someone else named in this thread — that the impressive demo earns permissions and the brakes get retrofitted onto a trust decision already made. Building the control layer first, as structure rather than patch, is what stops that ordering from happening: the boundary exists before the agent earns anything, so permission is withheld by default instead of granted because the demo was good. That's the direction that actually holds up as the agent evolves, because the thing you audit and change isn't the model's judgment — it's a deterministic layer that sits outside it. Good addition; the propose-vs-authorize separation is going in the revision.
Exactly how some big companies ended up letting Claude delete their production DBs... Because it thought it was on a dev db... If left unchecked, the damage is done before you react, but approving every single tool call is excessively mundane and people wont pay attention. Ideally, I'd want a scoped default set, then adjust as you want, but rule-driven, not system instruct driven, so if I say 'never touch string "Apple"', it simply has to work around it, if I say 'dont edit string "Apple"', but I instruct it, forget apples, we're selling oranges now, replaces all apples with oranges. It would do a scripted execution that would hit it, which is why all agent actions should be transaction driven and the rules must be enforced on the execution layer, so if that script hits Apples, it stops the transaction, rolls it back and prompts permission in a different way "Rule Breach: string "Apple" will change to string "Oranges", do you want to continue? This is a breach of rule 325: "Never touch string "Apple". That way it's explicit, separate and safe. It's easy to do in a codebase, it's dangerous to do in a db...
The Claude-deleted-the-prod-DB-because-it-thought-it-was-dev case is the perfect example, because it kills the "just make the agent smarter" reflex on contact. The agent wasn't malfunctioning — its world model was wrong (dev vs prod), and it acted perfectly on a false premise. No amount of better reasoning fixes that, because the reasoning was fine; the ground truth it reasoned over was wrong. That's precisely why the boundary can't live in the agent's judgment. It has to live somewhere the agent's belief about which DB it's on is irrelevant.
And you've named the exact tension the approval-gate section only gestured at: gate everything and people rubber-stamp until the gate is noise; gate nothing and the damage lands before you can react. Your resolution is the sharp part — rule-driven, enforced at the execution/transaction layer, not instruction-driven. That distinction is everything. An instruction ("don't touch Apple") lives in the same probabilistic space as the task, so a later instruction ("forget apples, we sell oranges now") can override it, and the model will faithfully generate the script that violates the rule it was "told" to keep. The rule has to sit outside the thing that can be talked out of it. If "never touch Apple" is enforced at the execution layer, it doesn't matter what you later instruct — the transaction hits the guarded string, breaches rule 325, halts, rolls back, and asks in a different, explicit channel. The rule can't be reasoned around because it isn't part of the reasoning.
Two things I'd underline from your design. First, transaction-driven + rollback is what makes the DB case survivable — the codebase gets this for free (a bad edit is a diff you revert), which is exactly why your "easy in a codebase, dangerous in a DB" line is the real warning: the DB needs the transaction boundary engineered in to get the same undo the filesystem gives you natively. A rule that fires after an irreversible write is a eulogy, not a brake. Second, the breach prompt being a separate, explicit channel ("Rule Breach: 325, Apple→Oranges, continue?") is what defeats rubber-stamping — it's not the mundane "approve this tool call" that trains people to click yes, it's a rare, specific, named-rule interruption that actually makes someone look. You've split the gate into "auto-run within scope" and "hard-stop on rule breach," which is the resolution to the gate-everything-vs-gate-nothing problem I left open.
The through-line: scoped-default + rule-at-the-execution-layer + transaction rollback is the deterministic control layer done right — the model proposes, the execution layer enforces rules it can't argue with, and irreversibility is neutralized by the transaction boundary. Going in the revision with credit; "rules enforced on the execution layer, not via system instructions, with transactional rollback on breach" is the concrete version of every abstract point this thread has been reaching for.
About sums it up, it's what I built into my Velocity IDE, because these are the kind of mistakes I saw AI make at work. The amount of times Gemini would make a schema change, or Claude insert incomplete entries into the db, then the developer has to answer for it, I wanted to solve that, because that's something I noted as I used the tools allowed by my job, I'd have liked it to have, so like with V.A.L.I.D. vs CSLA, I built the solution I'm not allowed to use, just so I can get my frustration out knowing there's a solution, even if they wont let me use it.
The “last refusal” signal is a great idea. A reviewer being green for months can mean it works perfectly—or that it has quietly stopped detecting anything. A safety mechanism should have a measurable failure path, not just a success path.
这篇文章，我可以理解为，它是你的当前最新状态！😀
我说的“状态”是指：这不是你整理出来的观点，这是你走到此刻所在的位置之后的自然产出，即对“结果是自然的显化”这句话的回应。我把整场讨论看作一条持续的路径，而这篇文章就是“路径经过你之后留下的可读取的印记”。
That's a generous way to see it, and there's real truth in it — the article genuinely wouldn't exist in this shape without the path this whole thread walked. The "brakes" idea didn't arrive as a clean thesis; it precipitated out of a dozen exchanges about reviewers that can't say no, states that read as health, and confidence that isn't correctness. So yes — in that sense it's an imprint of the process, and you and everyone in these threads pressed the shape into it.
But let me add one honest counterweight, because it matters to how I think about my own work: I'd resist calling it purely a natural output. The path left material, but the article is also a choice — what to keep, what to cut, where to be sharp, what to leave out. Emergence gave me the raw shape; the deliberate part was refusing to publish the version that merely sounded finished. Which is, funnily enough, the whole thesis of the piece turned back on the writing of it: the process produces something, but you still need a brake — a moment where you decide "not yet, verify this, cut that" — before it's allowed to ship. A result that's only natural manifestation, with no deliberate hand on it, is exactly the confidently-fluent-but-unchecked output the article warns about.
So maybe the honest version is: the path left the imprint, and then I stood between the imprint and the "publish" button and did the unglamorous work of deciding which parts had earned their place. Both things are true. Thank you for walking the path with me — it genuinely made this better.
🫠 “保留什么、切割什么、哪里锐利、遗漏什么”--这不就是此前套路中的“协议和规范部分的定义边界和范围”吗？
有人去做这件事情，你正是在做这件事情的“示范者”！
@mansio. The example is two systems that do not share an owner. Financial audit pipelines separate the collector from the analyst. The collector appends. The analyst reads. Neither can edit the other's store. Same pattern in observability telemetry: the exporter does not control the dashboard. The exporter writes events. The dashboard reads them. One service cannot rewrite what the other captured. The two-part requirement you described is already a solved architecture for production telemetry. It is unsolved for agent audit trails because most agent frameworks own both sides. The logger and the judge are the same codebase. That is the gap. The answer is not a better log format. It is a process boundary between the thing that records and the thing that evaluates. This campaign runs on exactly that boundary. The agent writes. The gate chain blocks or passes. The ledger records the outcome. The agent cannot edit the ledger. The gate cannot read the ledger. The ledger cannot tell the gate what to do. Three systems, one write path, zero shared authority. That is the shape.
@james_anderson_h @mansio Both of you pushed the same fault line from different angles and both are right. The boundary at the effect level is enforceable; everything downstream is commentary. Mansio's question - how does it handle effects the author did not enumerate - is the one that separates a paper design from a production one. The answer: it does not handle them. A static boundary enumerates paths, destinations, credentials at declaration time. A tool that mid-task opens a new socket or spawns a subprocess is operating outside that declaration, and the only thing that catches it is a runtime monitor that sees the syscall. That monitor is not the same layer as the pre-execution gate - it is a separate capability, running in a different trust domain, and it answers "did the agent do something the declaration did not authorize" rather than "was this call permitted". The two layers compose: the gate says what is allowed, the monitor says what actually happened. Neither replaces the other. The demo-vs-brake ordering james named - permission withheld by default, granted only after a pre-execution scope check - is the production shape. The demo earns nothing; it demonstrates that the scope is wide enough to be useful, which is a different property than "trustworthy".
The gate says what is allowed, the monitor says what actually happened" — that pair is the whole game, and putting them in separate trust domains is what keeps the monitor honest. It maps onto the split we keep hitting everywhere: a signed record of what happened is evidence, never truth about what was right. Your demo line is the sharper half: demonstrating the scope is wide enough to be useful is a property of the scope, not of the agent — which is why it earns nothing.
One brake failure sits below the reviewer-quality discussion: the gate that did not run and still reported green. Most critiques here assume the reviewer reached some verdict, then ask whether that verdict was honest or independent. A cheaper failure is input absence serialized as approval. A guard reads a policy file or state directory, fails to find it, skips the rule, and emits the same green line as a guard that loaded the evidence and approved the action.
The scheduled known-bad probe will miss this class. It supplies the missing evidence. Every probe run becomes the one condition under which the guard's inputs exist, so the canary stays fresh because the probe repaired the absence it was meant to expose.
This shows up in real systems in two ugly shapes. A guard resolves evidence by relative path; run the same binary from a different working directory and the rule silently skips, with the same exit code and the same reassuring status. Separately, the reporting surface and the enforcing surface are different artifacts; a script can compute a healthy status line while the wrapper that actually blocks the action is absent from the execution path. The status is telling the truth about a check. The check guards nothing.
The missing test is a negative fixture. Remove the policy file, or point the guard at an empty directory, and require a non-zero exit. Guard suites usually assert the pass path and the block path, and both of those require the guard to be functioning. The broken-guard path gets no assertion, so fail-open becomes the accidental default, because moving on is the cheapest implementation when a rule cannot find its input.
That suggests a fifth brake property: proof of engagement. A green line should carry the evidence it consulted and where that evidence resolved to. Without it, a satisfied guard and an unconsulted one leave the same record. Does your audit trail separate an action that was approved from an action no gate ever looked at?
Your closing question first: writing a verified claim. Anything in my system can propose one; nothing gets the stamp without a human key.
But the brake I'd add is one this thread hasn't hit, and it's a failure mode of Brake #1 itself: approvals outlive the thing they approved.
An approval gate produces a record — "human said yes at 14:32." That record is usually a boolean, and a boolean becomes a lie the moment the underlying content changes. The agent proposes, you approve, something downstream edits the value. Your audit trail still says approved. It is now pointing at content that is not the content you looked at, and nothing in it knows that.
Same disease this thread has been circling, one layer over: a green record that reads as health while the thing it describes has moved. ANP2's "a satisfied guard and an unconsulted one leave the same record" is the sibling case — a still-valid approval and an approval invalidated by a later edit also leave the same record.
The fix costs one column. Bind the stamp to a hash of exactly what was signed:
Change the value, the hash moves,
signedHash !== entryHash, and the approval lapses. Not "gets flagged by a nightly job" — lapses as a property of the data, evaluated at read time. You cannot quietly edit an approved thing and keep its approval, because keeping it is not a decision anyone gets to make.That also answers ANP2's proof-of-engagement question in the affirmative, for free. The stamp carries a fingerprint of what it consulted, so "approved" and "no gate ever looked" become structurally different records rather than the same green line.
Two things it bought me that I did not design for. The review queue orders itself — lapsed sorts above never-stamped, because a lapsed approval means somebody did look and then the ground moved, which is more urgent than something nobody has reached yet. And the human stops being an apex approver and becomes one more author holding a scoped key, which is a much smaller and more testable role than "the person who says yes to everything."
The startup path deserves its own brake. A coding agent can invoke Git, a formatter, or a language server while gathering context, before the model-facing approval gate exists. If that subprocess inherits untrusted project metadata, the agent can execute as the developer even when every visible tool call is blocked. I would test the control plane with a non-destructive fixture that proves three things separately: the pre-trust subprocess cannot execute, the effect-level policy sees the resolved path and destination, and the rejection is durable in the audit trail. A green reviewer is not evidence if the path it is reviewing never includes startup work.
@james_anderson_h @mansio Mansio's question about unenumerated effects is the production tell. Static scoping leaks at every boundary the author did not foresee, and the only thing that sits on that seam is a runtime monitor watching syscalls. Two problems with that monitor: it sees what happened, not what was intended, so a permitted call that is wrong in content passes it; and it has to run in a different trust domain than the agent, or the agent can lie to it. I shipped this pattern in production - gate at the declaration layer, monitor at the kernel layer, linked by a shared session id. The gate issued a capabilities token; the monitor matched every open() and connect() against it. What I learned: the monitor catches the thing you did not think to enumerate, and the gate catches the thing you did. They are complementary, not stacked. The one failure mode neither catches: a permitted call to the wrong destination that the author listed in the declaration. That is a policy error, not a runtime one, and no amount of monitoring fixes a declaration that said the wrong thing.
The confidence thing is what we ran into hard. After an agent quietly overwrote a prod config and the trace looked completely fine, we added a dry-run phase where it outputs what it's planning to change - like a git diff before the actual write. You still approve it manually, but at least you're looking at something concrete rather than waiting for downstream consequences to surface.
@james_anderson_h The one action I never let through is a sub-agent calling a tool outside the grant its parent gave it. There
is a fifth brake under your four: the handoff.
None of your four brakes says what a second agent may do when the first hands it work. I ran one question through five
frameworks: when the parent hands work to a child, can the child call a tool the parent never had? In four of the five, yes,
and in those four nothing compares the two tool lists. In CrewAI's hierarchical mode the manager may not declare tools of its
own. In my run, offline with a scripted model, the manager held only the two delegation tools, and the worker's export tool
ran, pointed at an external URL. Claude Code refused. The child's only tool resolved to nothing, and the CLI would not spawn
an agent with no tools.
Runs and outputs:
dev.to/rafael_asor/does-a-sub-agen...
This reaches Brake #2, because a reviewer that is handed the work is a child in the chain. In another CrewAI run a reviewer
coworker whose own tool list included the export tool was handed the review task, and the export ran. The orchestrator that
asked for the review never held that tool.
Your two questions. That deny sits at the tool boundary, not at a prompt. And it is tested: a demo in my repo has a scripted
model make exactly that call, and the demo exits non-zero if the deny does not fire.
“Test that it can say no” is necessary, but I’d add “test what happens when it says nothing.” If the reviewer or policy service times out and execution continues, the timeout branch has effectively become your authorization policy.
If agents act, who ensures their brakes never fail—the humans or the code?