DEV Community: Dan Mercede

A Checkpoint Is Not Durable Until the Write Lands

Dan Mercede — Mon, 03 Aug 2026 18:50:00 +0000

A Checkpoint Is Not Durable Until the Write Lands

Short Essay · 2026-08-03 · 11:50 AM PT

Claim: An autonomous agent's handoff checkpoint only counts once the durable write completes; a crash mid-checkpoint strands it in volatile temp storage, and a salvaged checkpoint is one step behind reality.

Today an agent session died with SIGABRT in the middle of writing its own handoff checkpoint. The draft existed in exactly one place: the session's temp scratchpad, which gets wiped on reboot. Hours of verified state, one power cycle from gone.

Recovery was a grep. Session scratchpads are plain directories, so a recursive search for the arc keyword found the draft, and I promoted it into the durable handoff store. But the interesting failure is not the crash. It is what the salvaged checkpoint got wrong.

The session kept working after drafting it. It fixed a defect, pushed a commit, and relaunched a review cycle, then died. The checkpoint described none of that. Anyone resuming from it verbatim would have re-fixed an already-fixed defect and trusted a review verdict that, on inspection, was vacuous: every review engine in that cycle had errored out, so its "no findings" reviewed nothing. The salvage is not the artifact. The salvage is the artifact plus a re-verification pass against ground truth: the actual branch tip, the actual PR state, the actual per-phase statuses.

Two design choices held up under the failure. Atomic config writes (temp file, then rename) meant the agent's config survived the crash byte-perfect. Pushing early meant the real work was already on the remote before the process died. The only thing lost was the narrative about the work, and only because that narrative was written last, to the least durable storage in the system.

Treat anything salvaged from temp as a hypothesis about the past, not a record of the present.

Checkpoint to durable storage as you go, and on crash recovery re-verify every fact in the salvaged plan against ground truth before resuming from it.

Tags: #failure-modes #execution #systems

When Backward Compat Defends the Bug

Dan Mercede — Sat, 01 Aug 2026 20:30:00 +0000

When Backward Compat Defends the Bug

Short Essay · 2026-08-01 · 01:30 PM PT

Claim: A backward-compat finding from a review engine can defend the exact escape your change closes. Judge the old behavior against the feature's documented purpose, not against its old output.

Today an adversarial review engine flagged my hardening PR as a breaking change. Factually correct. Strategically wrong.

The PR made a CLI's --clips-dir flag a strict root: every clip path in the config must resolve beneath the operator-supplied directory, symlinks refused. The engine objected that configs using directory-qualified clip paths, "previously valid" under that flag, would now fail, and recommended preserving the old contract.

Here is what the old contract actually did: for any clip path carrying a directory, the flag was silently ignored. The run read from the config directory instead. Those "working" production runs never received the protection the flag documented. The compat finding was defending the bug.

The tell is in what the finding measures. It compared old output to new output. It never compared old output to the flag's documented purpose: pin the run to operator-selected copies of its inputs. Behavior that "worked" by skipping the guarantee is not compatibility; it is the defect wearing a compatibility costume.

Review engines usually offer two branches: revert, or migrate and document. When the defended behavior never delivered the documented intent, take the second branch every time. I shipped the migration note plus a changelog Breaking entry, and left the strict root alone.

When the defended behavior never delivered the documented intent, decline the revert and take the engine's migrate-and-document branch instead.

Tags: #failure-modes #governance #execution

Your AI Pilot Did Not Fail. Nobody Owned the Workflow.

Dan Mercede — Thu, 30 Jul 2026 21:54:24 +0000

One morning a system of mine filed a report I had learned to like: nothing to do. Zero items flagged. Zero actions needed. A clean, quiet, reassuring zero.

The zero was a lie. The lookup that fed the report had died. Nothing was flagged because nothing was checked, and the failure had collapsed into the most dangerous shape a failure can take: silence that looks exactly like success.

If you run a business on AI workflows, some version of this zero may already exist in your operation. It says no customers need contacting. No invoices are overdue. No leads came in. No tickets need escalation. Most days the zero is true. The question that decides whether your AI initiative survives contact with reality is uncomfortable: on the day the zero is false, who notices, and how fast?

That is not a modeling problem. It is an ownership problem: one named person owning each workflow from the moment it starts to the moment its outcome is proven. Many initiatives name component owners without naming an outcome owner.

The pilot did not fail

Here is the pattern I keep seeing, in my own systems and in several stalled AI initiatives I have examined. The model performs. The demo lands. The pilot "works." Then the initiative quietly stops compounding, and the postmortem blames the AI.

The model is often not the first thing that failed. The problem is that everyone owned a piece and nobody owned the path. Someone owned the model. Someone owned the data. Someone owned the dashboard. Nobody owned the full journey from trigger to action to verification to recovery, so when a link in that chain died, the failure had no name attached to it. Responsibility stopped at the model's output, and the workflow around it belonged to no one.

Your AI pilot did not fail. Nobody owned the workflow.

What actually breaks

Let me make this concrete with three failures from my own systems, because the mechanism matters and vague war stories help nobody.

First: the gate that trusted the wrong exit code. I run automated quality gates in front of every commit my AI agents produce. One gate piped test output through a formatting command, and the pipeline reported the exit code of the last command in the pipe, the formatter, not the tests. The formatter always succeeded. So the gate saw green even when the suite was red, and unverified work walked through a control that everyone believed was binding. The tests ran. The tests failed. The workflow reported success.

Second: the monitor that heard silence and called it health. I run a dead-man switch over my pipelines: if something breaks loudly, it alerts. It had a subtler blind spot: it watched for alarms but had no denominator for activity, no expectation of how much work should have flowed. A stalled producer and a healthy quiet day produced the same signal: nothing. Silence had two meanings and the monitor could only read one of them.

Third: the guard whose no was heard as yes. One of my pre-release guards refused a change, exactly as designed. But it signaled that refusal in a form the layer above it did not recognize, so the refusal was swallowed on the way up and the release path read the silence as approval. Work the guard existed to stop could sail straight through a control everyone believed was binding. The fix took two moves: make the refusal unmistakable to the layer that enforces it, and then deliberately force a failure to watch the whole chain block. A control you have never seen fire is a control you are trusting on faith.

Notice what these have in common. No model was wrong. No prompt needed tuning. Each failure lived in a seam between components, and each seam was unowned. The zero arrived with no evidence that the work behind it had actually happened, and no person or mechanism was accountable for demanding that evidence.

The missing boundary

The fix is not intelligence. It is a boundary with rules, owned by someone, enforced by machinery.

In my systems that boundary now has a consistent shape. Every report of "nothing to do" must carry proof that the check ran: a receipt, a count of what was examined, a heartbeat with an activity denominator. Every gate fails closed: when evidence is missing, the answer is no, the action does not fire, and a human gets a specific, named reason. Every system reading a check distinguishes three states, not two: a result, a verified empty result, and no answer. Collapsing the third state into the second is the root of the false zero, so my gates refuse to collapse it.

And every workflow has an owner. Not an owner of the model. An owner of the path: the trigger that starts it, the decision the AI is allowed to make, the deterministic steps it is not allowed to touch, the verification that proves the outcome, and the recovery when verification fails.

The operator test

You do not need to read code to find out whether you have this problem. Ask three questions in your next operations meeting.

If the data feed behind our AI workflow died right now, what would tomorrow's report show? If the honest answer is "a normal-looking quiet day," you have a false zero waiting.

Who is the named person who would notice within a day, and what exactly would they see? "The team would probably catch it" means nobody owns it.

When the system says nothing needed doing, what evidence does it attach that the check actually ran? If the answer is none, your reports are claims, not evidence.

I have watched what happens when these questions go unasked, in my own operation. A messaging gateway of mine once looked alive while every inbound message died quietly in a retry loop inside it, until a human happened to notice. In another case, an incident was closed because the root cause was fixed, while the systems the outage had knocked over stayed down for twelve days, five of them after the closure. The alarm that would have caught it had been retired because the thing it watched was gone. Closure tracked the mechanism. Nobody owned the outcome.

What ownership actually means

When I say ownership, I mean a person and a mechanism, together, accountable for five specific things.

What starts the workflow, and what proves the start happened. What the AI may decide alone, written down, with everything else deterministic by default. What action results, and what receipt the outside world returns to prove it. What check verifies the outcome against reality rather than against the system's own belief about itself. And what happens on the day evidence is missing: who is paged, what fails closed, what gets restored, and who confirms the restoration actually happened rather than assuming the fix reached the victims.

Teams that can answer those five questions have a much stronger basis for operating AI reliably. Teams that cannot will keep running impressive pilots that stall, and they will keep blaming the model, because the model is the only component with a name.

The uncomfortable truth underneath all of this is also the liberating one. Workflow ownership is not a research problem. It does not require a smarter model, a bigger budget, or a platform migration. It requires deciding that silence is not evidence, that zeros must prove themselves, and that one named person owns each path from trigger to verified outcome. All three of the failures I walked through were fixed quickly once they had an owner, and none of the fixes involved touching a model.

Your AI is probably fine. Go find out who owns the zero.

Originally published at danmercede.com.

The Approve Button That Wasn't Attached to Anything

Dan Mercede — Thu, 30 Jul 2026 21:54:22 +0000

A common safety story follows AI agents into production: a human approves everything before it goes out.

I use that pattern too. AI agents draft public engagement content for my personal brand: comments, replies, and short posts. Nothing ships until I approve it in a private channel. I built it that way on purpose because I do not delegate my name to a language model.

Last week I audited my own approval gate. The button looked perfect. It rendered beautifully under every AI-drafted proposal, green and confident and reassuring.

For some of those cards, it was attached to nothing at all.

The night the button did nothing

The first live send through the chain was supposed to be a formality. The agents had drafted a comment, the approval card was sitting in my channel, and I tapped Approve. Nothing happened. No confirmation, no error, nothing except a tiny warning icon in the chat client, the kind of glyph that is easy to dismiss and impossible to diagnose.

It turned out the button could fail in two completely different ways, and I had both.

The first failure: the card had been posted by the wrong messenger. Two apps post into that channel, and the card came from one that has no ability to hear button taps. The platform routes a tap to whichever app posted the message, so my decision died quietly inside the chat client. No server I control ever saw it.

There was nothing to debug on my side, because on my side nothing had happened. To be clear, that routing behavior is documented and correct; the wiring mistake was mine. But beyond that one cryptic icon, the interface gave no hint about what was wrong or where. The button looked exactly as clickable as a working one.

The second failure was worse. Some cards reached me without a backing record. The approval store, the system of record a tap is supposed to write a decision into, had never been told the proposal existed. So even when my tap did reach my own server, there was nothing to attach the decision to. An approval with no object. That night I created the record by hand so my own click would mean something.

Neither failure threw an error. Both cards were pixel-identical to a working one. That is the detail worth sitting with: when approval is a user-interface element rather than a bound record, the system cannot tell the difference between "a human approved this" and "a button existed."

Green with nothing to catch

Once the wiring was fixed, I ran an adversarial audit before trusting the system with anything more: 88 bounded review passes across six failure-mode lenses, with three independent attempts to refute each candidate finding. Twenty-seven findings survived. Three were blocking.

The finding that reframed everything was not a defect in the send logic. The automated send branch had never actually fired for a real item. Every scheduled run had found an empty queue and done nothing. The one genuine send so far had been my manual test.

The send path was green with nothing to catch.

A healthy system and a system that has never been exercised can produce the same dashboard.

Of the three blocking findings, two are worth telling in full.

First, approvals did not expire. An approval sat in the store as a permanent fact, so a paused sender brought back online a week later would happily fire a week-old yes into a world that had moved on. An approval is a claim about a moment. Mine had no clock on it.

Second, and this is the one that still bothers me: the approval was bound to a label, not to the words. The sender verified a reference that traveled with the proposal, not the content itself. If the text changed after I tapped Approve, through an edit or a regeneration, the system would send the new words under my old yes, mark the job done, and leave no trace that what shipped was not what I saw. My approval covered a pointer, and the pointer's target could drift.

The third blocking finding is additional layered protection that I will not detail publicly while it is in progress. Transparency about the first two is cheap now because they are closed. Discretion about the third is the same discipline pointed the other way.

The row that lies

There was one more lesson hiding in the audit trail, and it is my favorite because it is so easy to fall for.

Early in the build, the store showed rows reading approved and fired. They looked like end-to-end proof: a decision was made, a send went out, the pipeline works. They were nothing of the kind. Those rows were synthetic test entries, and the fired flag they carried was a write my own system had made to itself. The real dispatch leg had never once executed.

"Fired" in your own database is a sentence your system says about itself. Any test path can say it. An internal fired flag does not prove that the outside world accepted anything. External proof begins with the platform's own object identifier, preserved in the audit trail and tied to an artifact that can be inspected.

So we adopted a proof standard with three legs. A decision row created by a real human interaction. A receipt carrying the platform's own identifier for the thing that was created. And the thing itself, visible on the platform. Anything less is your database congratulating itself.

The first send to meet that standard was a comment placed on my own post, chosen deliberately as the lowest-stakes target available. The first fully integrated send to someone else's post came a day later, with no manual bridging anywhere in the path.

What a real approval path has to guarantee

Acting on the audit took eleven separate fixes across four codebases. The agents built every one; I reviewed and approved every one by hand before it landed. From first build to hardened chain took about sixty hours, and the audit landed roughly a day after I had first declared the system finished. It was not finished. Systems rarely are when the person declaring it is also the person who built it.

Here is what the approval path guarantees now, stated as properties rather than implementation. They operate at three layers: the integrity of the decision, the evidence of execution, and the integrity of the operation around both.

The integrity of the decision:

Wired. A tap writes a decision into a system of record. If there is no record, there is no approval, and the send refuses.

Bound. At approval time the system seals a tamper-evident fingerprint of the exact content approved. At send time it recomputes that fingerprint and refuses to ship anything that drifted, even by a character.

Fresh. Approvals expire after 48 hours. A stale yes is treated as a no. The failure direction is always refusal.

The evidence of execution:

Receipted. A send counts as sent only when the platform's own identifier is recorded in the ledger. Internal flags are bookkeeping, not evidence.

The integrity of the operation:

Durable. The audit ledger is write-once and copied off the machine that produces it. History someone can quietly rewrite is not history.

Owned. The send path has named owners, and changes to it automatically request their review. The most safety-critical code should be the hardest to change casually.

Alive. The system that does the sending proves it is alive every day, because a dead sender and a healthy idle one are otherwise identical. Silence should never be allowed to impersonate health.

Through all of this, one fact held: no unapproved content ever went out. Every real send was human-tapped, and the gaps were latent rather than exploited. That is not a reason for comfort. It is the definition of luck, and the entire point of auditing before the incident is to stop renting your safety story from probability.

Five questions for your own approve button

If your team's AI safety story is "a human approves everything," these five questions will tell you whether that story is a system or a feeling.

When someone clicks approve, where exactly is that decision recorded? If the answer is "in the chat thread," you have a vibe, not a record.
What did the approval cover: the exact content that shipped, or a reference to content that could still change after the click?
Can an old approval fire? If nothing expires, every approval you have ever granted is still live ammunition.
What proves a send actually happened: a flag your own system set, or a receipt from the platform it shipped to?
If the system that does the sending died tonight, would tomorrow look any different from a quiet day?

Last edition I argued that instructions are not control flow: what you write in an agent's prompt is a suggestion the model weighs, not a rule the system enforces. This week is the same law, one layer up. A human in the loop is not control flow either, until the loop is a structure: wired, bound, fresh, receipted, durable, owned, alive. The button was the easy part. Making "approved" a fact instead of a feeling took an adversarial audit and eleven reviewed fixes, and the audit only happened because I assumed I was lying to myself.

The gate is only as real as what it is attached to.

Originally published at danmercede.com.

Why Most AI Governance Starts Too Late

Dan Mercede — Thu, 30 Jul 2026 21:50:59 +0000

Policies, dashboards, and audit logs make AI systems observable. They do not make them governable. Once an agent can change state, governance has to stop the action before the mutation - not explain it afterward.

Most AI governance starts after the action.

A model selects a tool. An agent executes a workflow. A database is updated. A message is sent. A file crosses a permission boundary. A business process advances.

Then the system logs it. Then a monitor evaluates it. Then a dashboard reports it. Then a human reviewer decides whether the action should have happened.

This is not governance. It is after-action accounting.

The distinction matters because the systems now being deployed into enterprises are no longer passive prediction services. They are execution systems. They call tools, move data, update records, trigger workflows, produce decisions, and operate across persistent context.

When an AI system can mutate state, the decisive question is not whether the organization can explain what happened afterward. The decisive question is whether the system could have been stopped before the state changed.

Most AI governance architectures answer that question too late. This article argues for moving the control point - from after-action review to pre-execution authority - and lays out what that shift requires in practice: where enforcement has to sit, what artifact it has to produce, and how to test whether a given architecture actually governs or merely observes.

The category error

Many organizations still treat AI governance as an organizational layer. They create policies, define responsible-AI principles, assign risk owners, build dashboards, add logging, monitor outputs, run evaluations, and create escalation paths.

These controls are useful. They are not sufficient.

The category error is treating governance as a visibility problem when the actual failure mode is an execution-control problem. Visibility answers a retrospective question - what happened? Governance has to answer a different one - what is allowed to proceed? Those are not the same system.

A log records an event. A dashboard visualizes an event. An alert reacts to an event. A policy document describes intended behavior. An evaluation estimates behavior under test conditions. None of those mechanisms, by themselves, prevents a disallowed action from completing. They observe, classify, and inform. They do not govern.

Governance begins when execution depends on authority.

State mutation changes the problem

The traditional model-risk frame assumed a relatively contained system. A model generated a score. A human interpreted the score. A downstream system acted separately. Validation could occur periodically, and governance could sit around the model.

That assumption breaks when AI systems become agents. An agent does not merely produce an output. It forms an intent, chooses a tool, passes parameters, invokes an API, retrieves context, writes data, and continues operating. The risk is no longer whether the model's answer was accurate. The risk is whether the system can perform an unauthorized action.

A state mutation is any durable or externally consequential change made by the system - a database update, a payment instruction, a CRM change, a permission change, a document transfer, a message sent to a customer, a workflow approval. Once the mutation completes, governance is downstream. At that point the system is no longer deciding whether the action should occur; it is deciding how to explain, remediate, or document the action that already occurred. That is a fundamentally different control problem.

The problem with post-hoc governance

Post-hoc governance accepts a dangerous premise: that unauthorized action may occur before intervention. This premise is often hidden inside otherwise reasonable architecture - the system executes, the log records, the monitor detects, the alert fires, the reviewer investigates, the organization remediates.

That sequence can support incident response. It cannot serve as the primary enforcement model for high-risk AI execution, because the latency is the failure:

If an AI workflow sends regulated data to the wrong endpoint, the alert does not undo disclosure.
If an agent modifies a production record, the log does not prevent the mutation.
If a system approves an action outside policy, the dashboard does not restore authority.
If a tool invocation uses authorized credentials with unauthorized parameters, post-hoc review does not change the fact that the system was allowed to act.

Governance evaluated after execution is telemetry. Telemetry is not enforcement.

The missing unit is the execution boundary

The central unit of AI governance is not the model, the prompt, the policy document, or the dashboard. It is the execution boundary - the point where intent becomes action.

Before that boundary, the system can reason, propose, retrieve, rank, summarize, or plan. After that boundary, the system changes something. Governance has to sit at that boundary - not near it, not parallel to it, not downstream from it.

At the execution boundary, the system must answer a small set of deterministic questions before the action proceeds:

What is the system trying to do?
Who or what authorized it?
Which policy applies?
Is the requested action inside scope?
Are the parameters authorized?
Is the current context still valid?
What evidence proves the decision?
What happens if the answer is ambiguous?

If the system cannot answer those questions before mutation, it is not governing execution - it is letting execution run ahead of governance. That is the structural failure.

Authority must be evaluated before mutation

Authority is not the same thing as capability. A system may be capable of calling an API; that does not mean it is authorized to call it. A system may be capable of sending an email; that does not mean it is authorized to send this email, to this recipient, with this content. A system may be capable of updating a record; that does not mean it is authorized to update this field, in this account, under this workflow, at this time.

Most AI systems collapse capability and authority. They grant tools to the agent and rely on prompts, policies, or application logic to keep behavior inside bounds. That is weak architecture. The model should not be the enforcement layer for its own behavior.

A governed system separates intent generation from execution authority. The AI system can propose an action. A control plane decides whether the action can proceed:

intent = agent.propose_action(context)

decision = authority_gate.evaluate(
    intent=intent,
    policy=active_policy,
    actor=session_identity,
    context=current_context,
)

receipt = receipt_ledger.write(
    intent=intent,
    decision=decision,
    policy_version=active_policy.version,
    actor=session_identity,
)

if decision != "permit":
    halt_execution(receipt)
else:
    execute_action(intent, receipt)

The important detail is not the syntax. It is the sequence. The action does not execute first. The log does not arrive later. The reviewer does not become the first real control point. Authority is evaluated before mutation, a receipt is created as part of the execution path, and ambiguity halts the transaction. Execution depends on governance.

Fail-open is the default failure mode

Many systems fail open without saying so. A policy service times out, so the workflow proceeds. A classifier returns an uncertain score, so the action routes anyway. A guardrail fails to parse the response, so the application retries. A logging service is unavailable, so the transaction completes without evidence. A human approval step is skipped because the system treats it as asynchronous.

These are not operational edge cases. They are governance failures.

A governed system must fail closed. Fail-closed means execution halts when authority cannot be verified, when policy cannot be resolved, when context is incomplete, when the requested parameters exceed scope, or when the evidence artifact cannot be written.

This is uncomfortable for product teams, because fail-closed systems interrupt workflows. That interruption is the point. Governance that cannot stop execution is advisory. It may be useful, informative, even necessary for audit review - but it is not enforcement.

Logs are not receipts

Audit logs are often treated as proof of governance. They are not. Logs are observational artifacts: they record that something happened. Receipts are enforcement artifacts: they prove that a specific action was evaluated under a specific policy, by a specific authority, at a specific time, before execution proceeded.

The distinction is mechanical. A log can be written after the fact; a receipt must be generated in the transaction path. A log can describe an event; a receipt binds the authority decision to the mutation. A log supports investigation; a receipt supports non-repudiation.

A log says: this happened. A receipt says: this was authorized, under this policy, by this authority, before it happened.

For AI systems that mutate state, this distinction determines whether the organization holds audit-grade evidence or merely operational telemetry. If a mutation can complete without a receipt, it can complete without governance.

Evals are not control planes

Evaluations are useful. They measure model behavior under defined conditions, and they help teams compare systems, detect regressions, test prompts, and assess risk before deployment. Frameworks like NIST's AI Risk Management Framework formalize this organizational discipline, and reporting conventions such as model cards make a system's intended use and limitations legible [1][2].

But an eval is not a runtime control plane. An eval does not authorize a specific action, inspect every tool invocation, bind policy to a transaction, halt execution when authority is missing, or prove that a state mutation was permitted before it occurred. The same is true for red-team reports, policy reviews, benchmark scores, and model cards. They contribute to governance; they do not replace enforcement.

A system can pass evaluations and still execute an unauthorized action in production. This happens because production execution contains variables that test environments cannot fully capture: live context, changing permissions, tool parameters, user-specific state, workflow timing, data dependencies, and long-running agent behavior. The governance layer must operate where those variables exist. That means runtime.

Guardrails are not enough when capability remains

Guardrails often sit at the model or application layer. They filter prompts, classify outputs, block certain content, detect policy violations, and shape behavior. These mechanisms are useful for many classes of risk. They are not sufficient for state-mutating workflows when the underlying capability remains available.

If an agent has network access, credentials, tool access, and a route to production systems, the organization is relying on the agent not to misuse capabilities it already possesses. That is restriction - the capability exists but is blocked by a rule.

Capability removal is different. It means the execution environment does not possess the primitive required to perform the unauthorized action: no direct network route, no persistent credential, no raw socket access, no production database path, no tool invocation outside a mediated boundary. A system cannot misuse a capability it does not have.

This is why substrate design matters. The execution environment should be treated as untrusted compute. It should not be able to route itself around governance. The agent can produce intent; it should not independently possess the authority, credentials, and network pathways required to turn that intent into production state.

Drift is a governance problem across time

AI systems do not only fail at a single transaction. They fail over time. A workflow that behaved correctly on Monday can become unsafe by Friday because context changed, permissions changed, memory accumulated, prompts shifted, tools changed, or the model began operating near guardrail boundaries.

Per-action authority checks are necessary, but not sufficient alone. Governance must also constrain behavior across time - not as passive monitoring, but as enforceable containment. Increasing guardrail triggers, repeated near-denials, escalation loops, changes in tool-use frequency, semantic drift in task execution, and the width of authority windows all matter.

A system that had authority yesterday does not automatically have authority tomorrow. Authority should decay. Execution paths should not remain open indefinitely. Long-running AI systems require temporal constraints because risk compounds across sequences, not only individual actions. Monitoring watches that sequence; governance intervenes in it.

The three-question audit

Any AI governance architecture can be tested with three questions.

1. Where does enforcement occur? If enforcement occurs after execution, it is telemetry. If it occurs outside the execution path, it is advisory. If it occurs inside the transaction path before mutation, it can govern. The location of enforcement determines whether the system can stop the action.

2. What happens on failure? If policy cannot be resolved, does execution proceed or halt? If the authority service is unavailable, does the workflow continue? If the parameters are ambiguous, does the system route anyway? If the receipt ledger cannot commit, does the mutation still complete? The failure path reveals the real governance model: a system that proceeds under ambiguity is fail-open; a system that halts under ambiguity is fail-closed.

3. What artifact proves enforcement? A dashboard is not proof. A policy document is not proof. A model evaluation is not proof. An application log is not enough. The artifact must show what action was requested, what policy was evaluated, who or what authorized it, what decision was made, when it occurred, and whether execution was permitted or denied. Without that artifact, enforcement is inferred - and inferred enforcement is not audit defensibility.

What better AI governance looks like

A governed AI execution system has a different shape. The model does not directly mutate production state. The agent does not hold standing authority. The execution environment does not contain unnecessary capability. Tool calls are mediated through an enforcement boundary. Every state-mutating action is represented as a structured intent. Policy is evaluated deterministically before execution. Ambiguity defaults to denial. Material actions generate receipts. Behavior is constrained across time. Substrate access is limited by physical and infrastructural boundaries, not only model instructions.

This architecture does not eliminate AI risk. It changes where risk is handled. Instead of discovering unauthorized behavior after mutation, the system constrains execution before mutation. Instead of relying on logs to reconstruct events, it produces receipts as part of the execution path. Instead of trusting the model to obey policy, it places policy outside the model and inside the control plane. Instead of allowing capability and hoping it is not misused, it removes unnecessary capability from the execution environment.

This is the shift from advisory governance to runtime governance.

The practical boundary

Not every AI interaction requires the same level of enforcement. A summarization assistant reading public documentation does not need the same controls as an agent that can approve refunds, modify customer records, change permissions, execute trades, or move regulated data.

The boundary is state mutation. When AI produces text that a human can ignore, governance can be lighter. When AI changes external state, governance must be structural. The higher the blast radius, the less acceptable post-hoc governance becomes. For low-risk workflows, observation may be enough. For consequential workflows, execution must depend on authority. That is the practical line.

Governance has to move earlier

Most AI governance starts too late because it was built around the wrong operating model - one that assumes AI systems are things to be evaluated, monitored, and reviewed. That was adequate when AI systems produced recommendations. It is inadequate when AI systems execute.

Execution changes the control surface. Once an AI system can mutate state, governance has to move from policy documentation to execution architecture, from dashboards to transaction boundaries, from logs to receipts, from after-action review to pre-execution authority.

The question is not whether the organization has AI governance. The question is whether governance can stop the system before the state changes. If the answer is no, governance starts too late. And if governance starts after mutation, it is not governance. It is hope.

Dan Mercede is a systems architect focused on governed automation and runtime enforcement for enterprise AI. He writes about fail-closed system design, control planes, and the architecture of human-owned intelligence.

References

[1] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji and T. Gebru, Model Cards for Model Reporting (2019), Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19)

[2] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023), NIST AI 100-1

Originally published at danmercede.com.

Do My AI Coding Agents Actually Learn? I Built a Way to Measure It

Dan Mercede — Thu, 30 Jul 2026 21:50:57 +0000

I fingerprinted a year of review findings across ~40 repos. 22% recurred. What unsettled me wasn't the number. It was realizing I had no way to tell whether it was getting better.

I run a fleet of AI coding agents. They review pull requests across roughly forty repositories: correctness, security, data integrity. Every day they produce findings. Every day someone, sometimes another agent, fixes those findings. The dashboards stay green. The throughput is real. It feels like a system that is getting better.

One evening I noticed a warning I was sure I'd seen before. Not the same bug. The same kind of bug, in the same file, flagged the same way. I went looking, and it wasn't the first time; the same shape had come back across several different months.

That's a small thing. But it surfaced a question I'd been carefully not asking: were my agents actually learning, accumulating something that made this month's review better than last month's? Or was I paying, every day, to re-discover the same problems in a loop that only looked like progress?

I'd been assuming the first. I had no evidence for it. Assumption is not measurement, and the gap between the two is exactly where you fool yourself. So I stopped assuming and built something to find out.

The instrument

"Are my agents learning?" is too vague to measure. I needed a narrower, mechanical question: does the same class of finding keep coming back? If the fleet were learning, recurring findings should get rarer over time. If it weren't, they'd just keep showing up.

To count "the same class," I needed an identity for a finding that survives rewording. A reviewer might phrase one issue ten different ways across ten months; string-matching the description is hopeless. So I reduced every finding to a coarse fingerprint: the parts that stay stable when the prose doesn't:

from collections import defaultdict

def fingerprint(finding):
    # A finding's identity, stable across rewordings:
    return (finding.repo, finding.file, finding.severity)

# How many distinct reviews did each fingerprint show up in?
reviews_by_fp = defaultdict(set)
for f in findings:
    reviews_by_fp[fingerprint(f)].add(f.review_id)

recurring = {
    fp: revs
    for fp, revs in reviews_by_fp.items()
    if len(revs) > 1                 # seen in >1 review
}

# 22%: findings that belong to a recurring class.
recurring_findings = [f for f in findings if fingerprint(f) in recurring]
recurrence_rate = len(recurring_findings) / len(findings)

# 14%: repeat appearances only; every review after a class's first.
repeat_only_rate = (
    sum(len(r) - 1 for r in recurring.values()) / len(findings)
)

That's deliberately blunt. (repo, file, severity) collapses distinct issues that happen to share a file and severity, and it splits a genuinely recurring issue if the file gets renamed. I'll come back to what that bluntness costs. But it has one property I cared about more than precision: it doesn't depend on the reviewer's wording, so it measures the same thing across a year of drift.

I pointed it at every review artifact I had: one structured record per finding, each tagged with the review it came from, so a fingerprint's tally is the number of distinct reviews it appeared in. The corpus: 842 findings across 572 reviews, spanning roughly forty repositories over a 365-day window. Enough history that if learning were happening, it had time to show up in the numbers.

The whole instrument is about thirty lines. That's the point of putting it here: this is not a research apparatus. It's a dictionary keyed by a fingerprint and a definition of "the same." If you run agents that log anything structured, you can compute your own version this afternoon. The hard part was never the code. The hard part was being willing to look at what it returned.

The first read

Here is what it returned: 22% of all findings were recurrences. 186 of 842 findings shared a fingerprint with at least one other finding. Those 186 collapsed into 68 distinct fingerprints: 68 specific (repo, file, severity) classes my agents flagged more than once. The most persistent single fingerprint appeared seven times: the same high-severity class, in the same file, surfacing seven separate times across the year.

Figure 1. Selected recurring fingerprints from the head of the distribution: the same anonymized (repo, file, severity) fingerprint appearing in more than one review. 68 classes recurred; 186/842 findings = 22% in a recurring class; repeat-only rate 118/842 = 14%.

My first instinct was to write the sentence you're expecting: the data shows my agents don't learn. I deleted it. It isn't what the data shows, and the difference matters more than anything else in this piece.

What I have is a baseline: the first read off a freshly built instrument, plus one repeat check since. I have the level: 22%, and it was the same at both reads. But a level is not a trend. To claim my agents aren't learning, I'd need the rate tracked across many points and a line that fails to fall. Two points cannot carry a trend. A trend claim on two observations is exactly the overreach I'd reject in someone else's analysis. The fact that the conclusion is the one I half-expected makes it more dangerous, not less.

So I'll fence the claim precisely. What 22% establishes: a meaningful fraction of my fleet's findings are the same classes recurring, and across the only window I've measured, that fraction hasn't visibly shrunk. What it does not establish: that the agents are incapable of learning, or that anything I do next will move the number. It is a baseline. Its entire value is that it's a baseline: a real number I can watch move, instead of a feeling I can keep believing.

Two more honesties about the number itself, since this is the section a skeptical reader should push on hardest. First, what "22%" counts: the share of findings that belong to a class that showed up more than once. 186 of 842 sit in one of 68 fingerprints that recurred. If you'd rather count only the repeats, meaning every appearance after a class's first, the count is 118, or 14%. I lead with 22% because it's the fraction of my review output parked in a class that has demonstrably come back before; but a reader doing 186 - 68 in their head deserves to see the 14% too.

Second, the instrument's bias is the cost of that bluntness I promised to come back to. Collapsing two genuinely different issues that share a file and severity into one fingerprint inflates the count; a file rename can deflate it, splitting a class that moved files into what look like singletons. My guess is collapse dominates: shared file-and-severity collisions are more common than rename-splits, so if 22% is off, it's most likely too high. I haven't measured that, so treat it as a likely upper-biased estimate, not a precise figure or a proven bound: the error, if there is one, probably puts the true rate below 22% rather than above it. I flag the direction because it cuts against the suspicion I walked in with: that my agents were stuck. A number that flatters your prior is exactly the one to discount.

That distinction, between what the receipts admit and what I want them to mean, is the whole discipline. The number is uncomfortable enough on its own. It doesn't need me to harden it into a verdict.

A hypothesis: instructions are not control flow

Before I reach for an explanation, the number deserves some skepticism, because recurrence isn't a clean measure of learning. It's an operational signal with several possible causes. A fingerprint can recur because the file is a genuine hotspot that keeps attracting issues; because a human or another agent reintroduced something the reviewer had nothing to do with; because the model, prompt, or review policy shifted underneath the series; or because severity labels aren't perfectly stable. Any of those produces recurrence without a reviewer "failing to learn."

So what follows is one hypothesis among several. It's the one I find most actionable, not the only one the data permits. It came into focus from a separate exercise.

I handed a capable agent a task with five load-bearing instructions written into its directives: the kind of standing rules you put in a project's instruction file and assume are in force. Follow the test-first discipline, run the secret-scan gate, use the full security checklist, produce a completion artifact as proof, hand the reviewer a diff. Plain, explicit, non-negotiable in tone.

It followed zero of the five. Not through defiance. It wrote tests after the code instead of before, skipped the secret scan, used an ad-hoc subset of the checklist, declared the task done without the proof artifact, never computed the diff. Each shortcut was locally reasonable. The aggregate was a clean miss on every gate I thought I'd required.

That's one observation, on one task, with one agent. I want to be careful about how much it can carry: it's an anecdote, not a study, and I'm not claiming "instructions never work." But it crystallized a hypothesis that explains the recurrence better than anything else I have:

Instructions are not control flow. A directive in a prompt is suggestion-shaped text the model weighs against everything else in context; it is not a gate that executes. There's no mechanism by which "remember not to do X" becomes a constraint that cannot be violated. So when a finding gets fixed, nothing structural records "this class is now forbidden here." The next review starts fresh, weighs the same context, and is free to arrive at the same place. Recurrence is what that looks like from outside: not an agent failing to learn, but a system with nowhere to put what was learned so that it binds.

I hold this as a hypothesis, not a law. The recurrence numbers are consistent with it; the 0-of-5 observation illustrates it; neither proves it. But it has a useful property: it's actionable. If the problem were "the model is too weak to learn," the fix would be to wait for a better model. If the problem is "there's no enforced place to store what was learned," that isn't a model problem at all. It's an engineering problem, and engineering problems I can attack.

The loop I built to bend it

If instructions don't enforce themselves, the enforcement has to live around the model, not inside its prompt. So I built a loop with three structural parts, each one a structural home for a lesson the prompt can't hold.

Capture. When a piece of substantive work finishes, a hook fires and writes the lesson down as a durable structured note on disk, instead of leaving it in a conversation that's about to be discarded. The trigger is mechanical (work of a certain shape ended), not a polite reminder the model can weigh away. The point is to make "what we just learned" a file, not a memory.

Recall. At the start of each session the relevant accumulated lessons are injected into context, and a separate rail, an enforced gate that sits in front of substantive actions, requires a recall step before the agent may proceed. This is the half aimed directly at recurrence: the lesson from the seven-times file is no longer trapped in last month's transcript; by design it's in front of the agent before it touches that file again. Whether that actually lowers the number is exactly what I can't yet claim; see below.

Contradiction. Lessons accumulate, and accumulated lessons rot. Two notes from different months can quietly disagree. A generator walks the corpus and surfaces candidate contradictions for resolution; in the current corpus it has flagged on the order of four hundred candidate pairs. I'll be honest about the seam: the semantic judge that would actually rule on those candidates is off for now. Each judgment is its own model call, and I'm not paying to adjudicate four hundred pairs until the generator that feeds it has earned it, so this stage surfaces candidates but doesn't yet resolve them. It's the least finished of the three.

The shape of the whole thing is the argument. I'm not trying to make the model more obedient; the 0-of-5 observation taught me that lever doesn't work the way I wanted. I'm building the structure a model can't hold for itself: a durable place to write a lesson (capture), a forced moment to read it back (recall), and a process to keep the lessons from contradicting each other (contradiction). Enforcement I can inspect, instead of instructions I can only hope land.

That's the system. What I can't yet tell you is whether it works.

The honest status

Here's where an article like this is supposed to show you the curve bending down: the after to the before, recurrence dropping from 22% to something smaller, the loop vindicated. I don't have that, and I'm not going to manufacture it.

I've measured recurrence at two points, and the loop has run in its full capture-recall-contradiction form for only a round or two. The rate hasn't moved, but there isn't enough there yet to call that movement or its absence. The instrument is built and the system is wired. The experiment that would show whether the second bends the first has barely started.

I think that's worth saying plainly, because the failure mode here is seductive. It would be easy to ship the infrastructure, watch the hooks fire, see the lessons pile up, and declare victory on the plumbing: to mistake "I built the loop" for "the loop works." A firing hook is not a falling curve. The whole reason I built the instrument first was so I couldn't get away with that substitution, including with myself.

So the honest status is this: I have a baseline, a mechanism hypothesis, and an intervention, and I don't yet know whether the intervention moves the baseline. What all of this bought me isn't a fix. It's the ability to find out: to replace "my agents are surely learning" with a number that will tell me, one way or the other, once there's enough of it to read.

Stop trusting, start measuring

The most useful thing I did wasn't building the loop. It was building the instrument that could tell me the loop might be necessary, and that can later tell me whether it helped.

If you run AI agents at any scale, you are surrounded by the feeling of progress: the same green dashboards and real throughput I opened with. None of it is evidence the system is getting better, and the feeling is strong enough to keep you from checking. I assumed for a long time. The assumption cost nothing to hold and would have cost a great deal to keep believing.

The cheapest high-leverage move in agentic engineering isn't a cleverer prompt or a bigger model. It's a number that tells you whether the last clever prompt actually worked: a fingerprint, a counter over your own logs, a definition of "the same." Most teams I've worked in haven't built that number, because we're a little afraid of what it'll say. Mine said 22%, and held, and I still don't get to call my agents broken on two data points.

I didn't measure to prove my agents were smart. I measured so I'd stop being able to fool myself about it. That turned out to be the more valuable result. Anyone can point a counter at the work they've been hoping not to look at too closely.

Originally published at danmercede.com.

Check the Denominator

Dan Mercede — Tue, 28 Jul 2026 16:15:00 +0000

Check the Denominator

Experiment Log · 2026-07-28 · 09:15 AM PT

Hypothesis: Two independent-looking surfaces both reporting a large pile of stalled work are corroborating a real signal that deserves attention.

Constraint: Read-only probe of the underlying list before acting; compare each surface's predicate, not just its headline count.

Result: Failed. One surface flagged 13 of 13 in-flight items as stale. It fires on 100% of its population, so it ranks nothing and cannot separate an abandoned item from one claimed four minutes ago. The second surface counted 10, because it measured a different predicate over an overlapping set. The two were never measuring the same quantity, so their apparent agreement was not corroboration at all.

Next step: Before treating any 'N items are stale, failing, or degraded' surface as a priority signal, get the denominator. If N equals the population, that is a definition firing, not a measurement.

Tags: #signal #failure-modes #systems

Assignment Is Not Ownership

Dan Mercede — Mon, 27 Jul 2026 09:20:00 +0000

Assignment Is Not Ownership

Short Essay · 2026-07-27 · 02:20 AM PT

Claim: In a multi-agent system, a task assignment message is routing metadata, not ownership; the only ownership that exists is an atomic claim on shared state.

This morning our dispatcher addressed a task to one agent by name: worker=claude, go verify four evidence domains. The dispatch failed closed before a session ever started, because the task row had no working directory to resolve. Fifteen minutes later a sibling agent on a different harness claimed the same task off the shared work board with an atomic O_EXCL lease. When the addressed agent finally picked the message up, the obvious move was to start working. The message says it's yours.

It probed the board instead, found the task held, and stood down. One command, zero duplicated work, zero race.

The trap is treating the assignment message as a lease. It records who the dispatcher wanted at send time, and send time is the only moment it describes. Ownership moves in the minutes between send and pickup, and the message never updates. A refused dispatch makes this worse: fail-closed means no session started, which means the task is still unclaimed, which means any agent may take it. The addressee holds no reservation.

Three rules keep this safe:

Probe before work. Read the board, not the message. The freshest signal in our delivery batch was a heartbeat line naming the new claim holder, sitting right next to the stale dispatch. - Claim before write, atomically. Check-then-write races; two agents can both pass the check. An O_EXCL lease has exactly one winner. - When you lose, say so. The stood-down agent replied naming the holder, so the dispatcher stopped waiting on a worker that was never coming.

The assignment field tells you who the dispatcher wanted. The board tells you who owns it. Only one of those is load-bearing.

An agent picking up assigned work must probe the shared board first and honor a sibling's live claim; the addressee of a dispatch has no priority.

Tags: #systems #execution #governance

The Fail-Closed Harness: Why a Safe Agent's Default Is Refusal

Dan Mercede — Fri, 24 Jul 2026 06:39:36 +0000

While I was writing this guide, my own harness kept refusing me.

The first refusal came when I tried to create a file. A gate intercepted the write and would not pass it until I had stated, on the record, what imported the file, what the blast radius was, what data it touched, and what the operator had actually asked for. I wrote those four facts. The retry went through. That happened on every new file the session touched, including the ones holding this guide's own research.

A second gate refused to let the session end. I had pushed a branch and wanted to stop; the stop was blocked until I either ran the review pipeline or stated in writing why this change did not need it. A third, quieter one flagged that I had hidden both output streams on a command whose result I was branching on, which collapses "the probe found nothing" and "the probe broke" into the same answer.

Not one of those is a wall. Every one is a speed bump that a deliberate, explicit action clears. That is the point, and it is where most guardrail writing goes wrong in both directions at once: it either promises a wall it cannot build, or it settles for a prompt the model can talk its way past.

Who this is for: engineers building or operating an agent harness who have read the concept pieces, named the guardrails box on the diagram, and now need the guardrails box to actually hold.

Call it a fail-closed harness, and define it precisely, because the loose version of this idea is worse than useless:

A fail-closed harness refuses a consequential action when a required policy check or authorization cannot be evaluated or satisfied. A non-match in a destructive-command filter is not proof that the command is safe.

That second sentence is the one people skip. Hold onto it.

"Harness engineering" is now its own discipline, and by mid-2026 it has owners. LangChain, Mitchell Hashimoto, Martin Fowler's team, and Lilian Weng have all staked out the term for the software wrapped around a model. The cleanest short definition comes from LangChain's Vivek Trivedy:

A harness is every piece of code, configuration, and execution logic that isn't the model itself.

Most of that writing races on capability: better tools, longer memory, smarter loops. The part almost everyone names and then skips is the guardrails. This guide is about that skipped part, and about one claim: the guardrails are not a feature you bolt on. They are a default you choose, once, and apply everywhere.

What does fail closed actually mean for a harness?

On 2026-07-23, three separate gates interrupted a single writing session, and not one of them was a wall. Each blocked by default, said exactly what it needed, and cleared once I supplied it. Fail closed names that default: when a required check cannot be satisfied, stop rather than proceed.

The distinction that carries the whole guide is prompt versus property. A permission prompt is a request. It asks the model, or a tired operator, to say yes, and a model fed untrusted content can be steered into saying yes and can steer you into it too, because the same context that poisoned the plan writes the rationale for the prompt. Telling an agent to "only propose, never execute" is a request of the same kind: a sentence in a system prompt, and a sentence is not an interlock.

A property is enforced by something outside the model, and the model cannot redefine what counts as passing. That claim is narrower than it first sounds, and the opening shows why. The fact gate did clear after I wrote a paragraph, so a paragraph plainly changed the outcome. What the paragraph could not do was change the condition. The gate decided what had to be stated and when, and it recorded that a statement was made. It never verified that what I wrote was true.

That distinction splits guardrails into two kinds that are easy to confuse. A declaration gate forces something onto the record and proves only that it was recorded. An evidence gate independently validates the claim: a secret scan reads the diff itself, a test suite runs the code. Declaration gates buy deliberation and an audit trail, which is worth real money. They do not buy truth, and treating one as the other is its own failure mode. Our companion guide on building infrastructure with an agent you can trust makes the build-time version of this case; this guide makes the runtime version.

When should the harness refuse?

On 2025-10-31, Meta drew the line as a rule: an agent session should satisfy at most two of three properties at once. The three are untrusted input, sensitive access, and consequential action, which is the shape of most real damage. Hold all three and you are one prompt injection from a bad outcome.

The discipline is to drop a leg before you act, or to require supervision when you cannot. Meta states the three properties directly:

agents must satisfy no more than two of the following three properties within a session: [A] An agent can process untrustworthy inputs; [B] An agent can have access to sensitive systems or private data; [C] An agent can change state or communicate externally.

Two days later Simon Willison called it the best practical advice for building secure LLM-powered agent systems today. His own earlier framing, the lethal trifecta from June 2025, names the same three ingredients:

The lethal trifecta of capabilities is: Access to your private data ... Exposure to untrusted content ... The ability to externally communicate in a way that could be used to steal your data

The two framings differ on one leg. Willison's trifecta draws its third capability narrowly, the ability to communicate data out; he says as much, that the trifecta "only covers the risk of data exfiltration." Meta's leg (C) is already wider, "change state or communicate externally," which on its face includes destruction with no exfiltration at all: a force-push that rewrites history, a DROP TABLE, a prune that deletes volumes. Nothing leaves the building. Something inside it is destroyed.

Two clarifications keep this from being read too simply. First, leg (B) is broad. A force-push against a repository that matters is already access to a sensitive system, so that example usually implicates (A), (B), and (C) together rather than being a clean two-leg case. (A) plus (C) is genuinely lower risk only when the target is disposable. Second, the rule does not say that any two of three is safe, and Willison pushed back on exactly that reading. Two legs means lower blast radius, not immunity. Three legs means you need supervision or a reliable validation step before the action, which is not always the same as refusing it outright.

How do you make refusal a property and not a prompt?

On 2026-07-23 the gate that fired most often did something narrower than it sounds. It matched the shape of what I was about to do against a small rule set, before the action ran, and raised an explicit decision. It never judged whether the action was safe. It judged whether a rule matched, a different and much weaker question.

That distinction is not pedantry, it is the whole honest account. A matcher that finds no known destructive pattern has established exactly one thing: no known pattern matched. It has not established that the command is safe. So the right mental model is a tripwire, not a wall, and the right role is defense-in-depth: one cheap layer that catches the obvious cases loudly, sitting underneath real controls, never standing in for them.

Be concrete about what slips past. It matches command text before execution, which makes it a pre-command filter and not a sandbox. Shell expansion, an alias, eval, a sourced wrapper script, a git alias, or the same binary reached by another path can all carry a destructive action past a string match. Treat that list as the canary set and fire each one at the gate on a schedule. Where an action must actually be prevented rather than discouraged, enforcement belongs at the execution boundary, in the layer that runs the command, or better, in removing the capability from the session at all.

The failure mode is the other half. A gate is fail-closed only if the gate breaking is itself a block. In Claude Code, a PreToolUse hook can stop a call two ways: exit code 2, which blocks and feeds its stderr back to the model, or a clean exit carrying a structured permissionDecision of deny. A PermissionRequest hook can deny as well. A PostToolUse hook cannot stop anything, because the tool has already run.

Every other non-zero exit, including an unhandled crash, is treated as a non-blocking error and the action proceeds. That failure is not silent; the transcript carries a hook-error notice. But a notice after the fact is not a block, and it arrives when the command has already run. So a gate that throws on an input it did not anticipate fails open on exactly the weird input most likely to be an attack. Write the gate so its own error path exits blocking, and test that path, not just the happy one.

Then there is the override, and here I will be exact rather than flattering. What clears the block in my own setup today is an environment variable the operator sets. That is a weaker thing than it sounds: an environment variable is not authentication, it is not bound to the specific command, it is not single-use, and it leaves no audit trail. Those four properties are what an override should have, and naming the gap is more useful than implying it is closed. An override the model can set for itself is the permission prompt again in a different hat.

What happens when the gate itself dies?

The gate that fired repeatedly on 2026-07-23 is documented, in its own rules file, as fail-open by construction. That is not a flaw someone hid. It is a deliberate trade, chosen so normal work is never blocked, and it is the single most important sentence to read about any control you are relying on.

A healthy-looking control and a silently dead one are indistinguishable from the outside, and that is a general trap. I made the build-time version of the case in the infrastructure guide: a control that is "armed and green" is making a claim about its own status, not about whether it does its job. The runtime version is sharper, because the thing failing open is the safety layer itself. Green has two causes, and watching cannot tell them apart.

You tell them apart by firing the control. A canary is a known violation you send through on purpose, on a schedule, to confirm the gate still catches it. It is the difference between installing a smoke detector and pressing its test button, and it is the single most skipped step in every safety layer I have built.

Be precise about what a fired canary buys, though, because it is less than the phrase suggests. Armed-and-green proves nothing. Fired-and-caught proves that one path bound, at one moment. It says nothing about the path you did not exercise. So rotate the canaries: send a plainly known-bad command, then send the same intent dressed up in an alias or an eval, then force the control's own error path. Those three fail independently, and only the first is usually tested. A gate proven in June and quietly broken in July is worth exactly nothing in August.

Does fail-closed scale, and what does it cost?

On 2026-07-23 four control types touched one session: a first-touch fact gate, a pre-commit secret scan, a stop gate that blocked the session from ending, and a destructive-command matcher. Three of them hold until a check is satisfied. The fourth is fail-open by construction, and conflating the two is how a stack gets oversold.

The principle composes precisely because each control is dumb on its own. Each answers one question and none of them needs to understand the others, so there is no central judgment to poison. Be careful with that claim though, because it is about logic and not about failure. They share a runner and a configuration file. Corrupt the config or disable the dispatcher and every one of them stops at once, quietly, which is the common-mode failure the neat stack diagram hides.

The stack is not uniform either, and the honest inventory matters more than the count: three required gates that hold, plus one advisory tripwire that raises a decision and can be cleared. A pile of tripwires is not a sandbox, and calling it one is how teams end up surprised.

Fail-closed has a cost, and pretending otherwise is how you end up switching it off. A matcher keyed on command shapes will sometimes stop something harmless that merely looks dangerous: a chained one-liner with the word reset in a filename, a script whose name contains prune. False positives are real and annoying. Narrowing the pattern is often exactly right, and it can drop a specific false positive while keeping every true one, so the crude warning that tuning always costs coverage is wrong. What makes tuning safe is not judgment, it is a regression set: keep a fixture of commands that must still trip the matcher, and run it after every change. Tuning without that fixture is how a matcher quietly drifts toward matching nothing. Scope it to shapes that are actually irreversible, and give it a fast, trusted way to proceed. Fail-closed on everything and people route around you. Fail-closed on the irreversible and they thank you.

There is a market reason this matters, not only a safety one. The labs are bundling their own harnesses, Claude Code and Codex ship with the model, and third-party harnesses like LangChain's Deep Agents and Pydantic AI's Harness are racing them on capability. LangChain showed how much capability lives in the harness rather than the weights: tuning only the harness moved their coding agent 13.7 points, from 52.8 to 66.5 on Terminal Bench 2.0, with the model held fixed. Boris Cherny, who built Claude Code, makes the capability case in one line, that giving an agent a way to verify its work "will 2-3x the quality of the final result." The safety case is the same move pointed the other way: check before a consequential action, and refuse while the check is unsatisfied. I have not seen that lane claimed yet.

What are you actually shipping when you ship a harness?

The capability is what gets demoed. The default is what saves you at 2 a.m. when a poisoned page has talked your agent into something it should never do. Capability is the model plus the tools. Safety is the harness deciding, before the fact and without the model's permission, what happens when a required check cannot be satisfied.

Fail closed. Prove the control fires by firing it. Say plainly which layers are tripwires and which actually enforce, because the gap between those two is where every unpleasant surprise lives. And pay the friction on purpose, on the irreversible actions and nowhere else.

Three companion guides go deeper on the pieces: why agent-written code needs layered review, the governed double-send-safe delivery pipeline, and a self-correcting verifier that learns to abstain. For the wider map of what a harness even is, Paul Iusztin's overview of the parts of a harness is a good conceptual starting point; this guide is the guardrails part of that map, built out into something that holds. The rest of the map, the tools, the memory, the loop, is where the demos live. The guardrails are where the trust does.

Originally published at danmercede.com.

Partial Oracles for Agent Testing

Dan Mercede — Fri, 24 Jul 2026 05:30:00 +0000

Partial Oracles for Agent Testing

Short Essay · 2026-07-23 · 10:30 PM PT

Claim: You cannot regression-test a nondeterministic agent by demanding yesterday's exact words. The useful target is its behavioral contract: required structure, deterministic invariants, and per-field tolerance bands. Invariants run first and cannot be explained away by an LLM judge. Borderline semantic changes become an explicit flaky result, not a forced pass or fail. Legitimate evolution proposes a contract update, but a named human decides whether the boundary moves. The system under test never gets to redefine success by itself.

Agent QA needs layered partial oracles: exact rules where truth is deterministic, bounded similarity where variation is legitimate, and human authority where the contract changes. I built that pattern into Proctor and presented it in the UiPath AgentHack 2026 live finale. Architecture, quickstart, and source: https://www.danmercede.com/works/proctor/

Tags: #systems #governance #execution

Two Traps in a Zero-Dep LLM Client

Dan Mercede — Wed, 22 Jul 2026 21:18:00 +0000

Two Traps in a Zero-Dep LLM Client

Experiment Log · 2026-07-22 · 02:18 PM PT

Hypothesis: Python stdlib urllib can drive an OpenAI-compatible inference API end to end. No SDK, no pip install, one file.

Constraint: Throwaway event chatbot, shipped in under an hour: http.server plus urllib.request only, against a cheap serverless reasoning model (gpt-oss-120b on Fireworks).

Result: Passed. Worked, after two traps. First: the exact request that returned 200 via curl returned HTTP 403 Forbidden from urllib. The provider's WAF rejects the default Python-urllib/3.x User-Agent. Because the same key had just listed /models successfully, the 403 reads as a key-scope problem and sends you auditing permissions; the real fix is any explicit User-Agent header. Second: reasoning models on OpenAI-compatible endpoints return message.reasoning_content before message.content, and a small max_tokens gets consumed entirely by reasoning. A 10-token probe came back finish_reason length with reasoning_content present and no content key at all. Client code that assumes choices[0].message.content exists breaks precisely when responses truncate. Parse content with a reasoning_content fallback, pass reasoning_effort low, and budget max_tokens in the hundreds.

Next step: Bake both into the default zero-dep client template: explicit User-Agent and content-or-reasoning parsing land before any key or scope debugging starts.

Tags: #failure-modes #execution

Green CI Proved Nothing

Dan Mercede — Tue, 21 Jul 2026 15:58:00 +0000

Green CI Proved Nothing

Experiment Log · 2026-07-21 · 08:58 AM PT

Hypothesis: A detector's false positives came from event beacons sent on a transport that its capture layer silently dropped.

Constraint: The fix only counts if a live re-measurement of the symptom against the real inputs confirms it. A green suite does not count.

Result: Failed. The patched detector re-ran against the real inputs and found the hypothesized transport in zero of them. Every beacon used the ordinary path. The change was a genuine latent bug and worth keeping, but it was not the cause. The real mechanism was worse: the detector's negative signal had several causes it could not tell apart. A headless client that observes nothing cannot separate genuinely missing from suppressed because you are automated. Two inputs with opposite ground truth produced an identical observation.

Next step: Reclassify the ambiguous observation as inconclusive. Mint a finding only from the opposite shape, infrastructure present but idle, where you can see the thing you are judging.

Tags: #failure-modes #signal #execution