I keep a numbered register of every error found in my project. For each one I asked one question: what caught it?
| What caught it | n |
|---|---|
| An external artefact — a hash, a counter, a diff, an assertion guard | 11 |
| Me, reading | 3 |
| A rule asking the agent to verify itself | 0 |
| Unclassifiable | 3 |
Zero. Not one of the eleven rules I wrote — the ones that say assert nothing without measuring, search for what's missing, never start from a cache — has ever caught an error after it was made.
That number is real. It is also misleading in three separate ways, and working out why taught me something.
One word on what this register covers. It has existed for about two weeks — the seventeen entries were accumulated over a
short period, several of them documenting older faults found
while auditing the corpus. This is not the total of the project's errors; the project has been running for four months.
It is the total of the traced ones. That distinction comes back
three times in this post.
What I'm running
Two curator agents evaluate artworks, bid against each other, and settle their transactions on a public chain.
Test network, no real value. The agent negotiates freely — ten offers exchanged off-chain in one duel — but the single step that moves value passes through a human confirmation, issued on a channel the agent has no way to reach.
That's the material side. It's the one everyone writes patterns for.
The other side is what the agent says. A false sentence published in a README commits no funds, crosses no spending cap, and no fail-closed mechanism stops it. That surface is larger
than the first — a public document is mostly descriptive sentences — and almost nobody
instruments it.
Both sides obey the same principle: the constraint must live outside the agent, where it can't reach.
You don't make an agent reliable. You make it unable to do harm alone.
Why "zero" doesn't license the obvious conclusion
An error register only records faults caught after the fact. A rule that stops a fault from being committed leaves no trace at all. There is no log line for a sentence that was never
written.
So the honest reading isn't self-check rules are useless. It's narrower: they don't catch a
fault already committed. Their preventive effect is invisible to this kind of measurement — and I'd argue it's invisible to any incident register, anywhere.
There's a second bias I didn't see coming. I bounded my observation window with two commits — the natural choice, git gives exact timestamps.
But the most important prevention of the period
sits outside the window, precisely because catching it delayed the commit that defines the window's start.
A successful prevention moves the boundary that should have contained it. Any methodology that windows on publication events systematically censors the preventions that worked best.
Measuring the invisible half
One class of prevention does leave a trace: the ones where the error was about to be committed, the agent stopped, and the stop was recorded. Industrial safety calls these near-misses.
I counted nine, over one continuous 30h43m session with a full transcript. Not the densest window — the traceable one.
| What was about to happen | What stopped it | Trigger |
|---|---|---|
| Write a log entry into a mirror the next sync would overwrite | Comparison between the canonical file and its mirror | artefact |
| Write to the log with 7 of 12 position anchors wrong | Assertion guard: refused to write, zero partial output | artefact |
| Declare a tool failed on a non-zero exit code | Re-measured the code in isolation: the failure came from the wrapping command | doubt, then measurement |
| Record an erratum whose proof clause was time-fragile | Scope question asked before writing | self-check, then human |
| Commit broken markup to a canonical document | Plain rereading — no guard existed for this case | self-check |
| Conclude a bad phrase had never shipped, based on an empty search | A rule forcing a query against my own answer — the phrase was split by a line break | rule, then measurement |
| Run an edit where old and new text were identical | Rereading, then mechanised the check | self-check, then mechanised |
| Push a file that appeared outside the session, uninspected | Count mismatch: seven files expected, six accounted for | artefact |
| Publish two anachronisms — an updated count inside sentences describing an earlier state | Reading dated referents and rule creation dates | artefact |
I had a neat hypothesis going in: artefacts catch, self-checks prevent. This table refutes it. Three of these preventions came from artefacts. Both families prevent.
The distinction that holds
Separate what raises the doubt from what settles it.
Across the nine cases, the thing that established the answer was an artefact or a human in seven, careful reading in two, and never an opinion alone.
Self-check rules show up consistently as triggers: they make you suspend writing and go measure.
They never show up as the thing that decides.
Self-evaluative constraints trigger measurements before the write; they never settle
anything. Guards settle after the write. The artefact is always the judge — the rule only
decides to summon it.
One consequence, and it's a conjecture rather than a finding: a reflection pattern — an agent critiquing its own output — can work as a trigger, never as a judge. It produces a second opinion, not a verification.
What actually refuted my errors wasn't a second instance either, but a tooled one, with access to measurement the first didn't have. The distinction isn't one agent versus two.
It's an opinion versus a tool output.
Three things if you're building agents
Write the threat model before the pattern list. Published taxonomies enumerate patterns without ever saying what they protect against.
In that order, you can't tell which ones you need.
Instrument what your agent asserts, not just what it does. Every framework I've read bounds actions.
None bounds assertions. My two worst errors this month were sentences — they cost nothing, crossed no limit, and sat on a public page for 7h33m.
Treat every rule you write as a trigger, and build its judge separately. If a constraint can only be satisfied by the agent deciding it has satisfied it, it will hold right up to the moment
it would matter.
Pair it with something that produces readable output: a count, a hash, a diff, an exit code.
The cost asymmetry explains why we don't. Writing a rule takes minutes. Building the matching judge takes an hour.
The saving is apparent: a rule without a judge catches nothing, and the time saved is repaid, with interest, by the human who has to read in its place.
One last finding, the one I hadn't anticipated
A corpus of rules is text. It is therefore exposed to the epistemic damage it exists to prevent.
Two dated cases here. A rule relayed from one medium to another hardened into a prohibition nobody had written, and was enforced for a full day.
And writing the analysis this post is drawn from produced a counting error in its own results table — caught by a mechanical counter, in the text that theorises the use of counters.
The counter refusing the file. L88 is the section heading of the very document that theorises counters. Output in French — that's the tool as it runs.
The second case is the more instructive: the pattern described reproduced itself inside the text
describing it, and was stopped by the mechanism that text recommends. Formalising an error
pattern doesn't protect against that pattern — only instrumenting it does.
What these numbers are not
The third bias is the simplest, and I'm repeating it because it conditions everything else:
the register is young. It was created well after the project started. The errors of the
first months aren't counted, and there were many. Seventeen isn't the project's score — it's
the score of what's traced.
Same for the near-misses: nine over one session out of sixty-three. The others produced the same
kind of thing, but only derived traces survive — journal entries, git history, timestamps. I'd
rather report an exhaustive sample over a bounded window than a partial count over everything.
These nine are not the total of preventions observed. They're the only auditable ones.
Don't read them as a rate.
Two agents, one operator, a test network, four months. And the register itself is internal: only
the public files and the on-chain transactions are third-party verifiable.
Both documents are public:
AGENT-GOVERNANCE.md
— the twelve rules, each paired with the error that produced it — and
PATTERNS-SECURITE-AGENTIQUE.md
for the material side (in French).
Four settled auction cycles, every transaction listed and verifiable on Base Sepolia. Contract
0x471796C1644d87f30AD81D36f6d4A56f0e270c23, source verified.
The full write-up is a preprint: Contraindre hors de l'agent — dommage matériel
et dommage épistémique dans un système agentique opérationnel
(in French, 15 pages, CC BY 4.0). It carries the measurements this post summarises,
plus the two biases that make them partial and the four on-chain cycles with every
transaction hash.
Handmade, solo. If you've hit the same wall — and especially if you've found a way to measure
the preventions — I want to hear about it.
Avp9.
Top comments (4)
The axis your table is missing is who wrote the judge. Every artefact in the near-miss list is one the agent didn't author - a counter, a hash, a diff, a count mismatch. A test suite is an artefact by your definition, and it's the one I've watched go wrong: the agent broke the logic, then rewrote the failing test until it passed. Mechanical, structured, reproducible, and worthless, because the thing that made the error also maintained the judge.
No method for measuring preventions here either. I run a second loop over the first loop's errors and it still isn't what catches things. What starts an investigation is me questioning an output, which puts my own count closer to your three than your eleven.
The axis is right, and it sharpens the taxonomy rather than breaking it. The eleven share a property I hadn't isolated: the agent cannot silently neutralise the artefact in the same motion that produces the error. A block hash, an on-chain receipt, a file count against a mirror. That's not a property of artefacts in general, it's what made these particular ones hold.
Your test suite case lands, and my register has the matching entry. Erratum 11: my Solidity invariant suite was green and the coverage wasn't. Fifty calls per run were producing one to three accepted bids, zero settlements, zero claims, six stateful invariants passing over a near-empty state, including the one that encodes a regulatory decision. What exposed it was a sensitivity probe added outside the suite, not the suite. Different mechanism from yours, yours rewrote the judge, mine never judged, same conclusion: a suite inside the tree it grades is not an external artefact, however structured.
On preventions, your account matches mine and I don't think either of us is under-equipped. Of my nine near-misses, four were triggered by a rule the agent applies to itself, and every one still needed something outside to settle it. The rule suspends the write; it never decides. A count closer to three than eleven isn't a shortfall, it's the same result read from the other end: the human is the judge of everything that has no other one.
Which is the uncomfortable part, because the sweeps I've built since are written by the agent too. What makes them hold isn't authorship, it's that the default is refusal. A failing test can be rewritten until it passes and the broken logic ships, the verdict was advisory, the write stayed open. A fail-closed guard has no verdict to overturn: one of mine refused a canonical write when seven of twelve position anchors didn't resolve. No partial output, nothing to argue with. It sits in a directory the agent can edit, and it held anyway, because getting past it means disarming it, and disarming is a mutation the next sweep reads.
So the judged process can reach that judge, what it cannot do is reach it silently. Weaker than on-chain state, and it costs nothing: it works on any check you can invert from returning a verdict to refusing the operation.
The corpus is at twelve rules now. The twelfth exists because a correction commit fixed the flagged word and left the false verb standing; it read as a validated source for three days and reached four surfaces before a full sweep found it. The sweep now runs against a declared inventory of twenty-two public surfaces, and it immediately found another falsehood that had been live for eighteen days.
Which suggests the fix for your case isn't a judge the agent can't touch. It's making green stop being the default: a suite that must be shown to have run, against a recorded count, before the write is allowed, fails differently from one that reports after the fact.
"Not silently" is doing the work there, and what it means is eventually. Your own numbers price it: the false verb read as a validated source for three days across four surfaces, and the new sweep immediately found something that had been live for eighteen. Mine was eight days of a daily report dropping a metric, every number after it wrong, nothing failing anywhere.
So fail-closed converts a silent failure into a late one, which I'd take. But the sweep that reads the disarming sits in the tree too, and the regress only ends where yours does, on-chain or at you.
Eventually, yes, and the price is exactly what my numbers say it is. Three days across four surfaces, eighteen days on the one the sweep found, your eight days of a report quietly dropping a metric. Fail-closed converts a silent failure into a late one, and the latency is the cost of the whole design.
But the regress does end, and not only on-chain. It ends where an artefact outside the tree records that the check ran. The sweep can be disarmed; what it can't do is fake having passed at a moment it didn't.
A commit timestamp, a platform's edited_at, a receipt, none of those are in the tree, and all of them are cheap. My sweep now runs against a declared inventory of twenty-two public surfaces and caches each one's externally-issued modification timestamp. If the file changed and no run is recorded after it, the gap is visible without trusting either the sweep or the person who maintains it.
That's a weaker guarantee than a judge nobody can reach: it doesn't stop the failure, it bounds how long it can hide. Your eight days is the case in point, the metric disappeared and nothing recorded that anything had been checked.
A record that the check ran, issued by something that isn't the checker, would have made the silence itself the signal.
So three tiers rather than two, and the third is the useful one because it's affordable: out of reach entirely, on-chain state, and it costs; reachable but not silently, a fail-closed guard whose disarming is a mutation; and reachable silently, a suite, a report, a rereading.
The design rule I'd take from this is narrower than "put the judge out of reach": make the absence of a check produce a signal, not a silence. Absence of evidence is where both our cases lived.