DEV Community

Cover image for Exit Zero: who tests the gates that guard coding agents
Luciano Oliveira
Luciano Oliveira

Posted on Originally published at sozodata.com.br AI-assisted

Exit Zero: who tests the gates that guard coding agents

#ai #testing #softwareengineering #devops

I spent eleven weeks building deterministic guardrails around coding agents, and the rest of the time trying to prove them wrong. Mutation testing turned up a gate that printed FAILED to the screen and then returned exit 0. It was one of two gates I was about to turn on for a client.

Internal SDLC platform · 140 commits · 33 gates · 237 harness cases · Repository names anonymized; every number is one I measured.

The borrowed numbers

I have worked in software for nearly thirty years, long enough to watch the discipline work out its methods, its processes and its testing. In late 2024 I started developing with AI assistance, and the scale of the work changed. On September 11, on one of the products this platform governs, I closed out 59 commits and a diff of 207,859 added lines.

That number is worth breaking down, because breaking numbers down is what this article is about. Of those lines, 184,770 come from 22 ORM migration snapshots generated by a tool. What I actually wrote that day was 16,783 lines of TypeScript (7,762 of production code and 9,021 of test) and about 3,400 of documentation. Artifacts that no human ever reads inflated the headline number twelvefold.

Volume is not a sign of quality, of fitness for the business need, or of software that works. The published data says the same, and more precisely than the slogans built on top of it.

METR ran a randomized controlled trial with 16 experienced open-source developers working on 246 real tasks in their own mature repositories. Going in, they predicted AI would make them 24% faster; afterward, they estimated it had made them 20% faster. Measured, they were 19% slower. That is a gap of roughly 39 points between perception and reality, and it points exactly where it should worry anyone shipping an agentic workflow to production: the people best positioned to notice the problem did not notice it.

GitClear analyzed more than 600 million commits and found that duplicated code blocks rose 81% between 2023 and 2026, to the highest level it has ever recorded (73.0 per million lines changed). Over the same period, copy-paste within a single commit rose 41%, error-masking constructs rose 47%, and two-week churn rose 15%. Meanwhile, the signs of consolidation fell: cross-file function calls were down 35% and refactoring-driven line moves were down 70%.

In two consecutive reports, DORA describes AI as an amplifier: it magnifies the strengths of high-performing organizations and the dysfunctions of the ones that were already struggling. In DORA's reading, the payoff depends on the quality of the internal platform around the tool; the tool alone does not explain it.

Taken together, these numbers say something narrower than "AI coding is bad." They say the model is not the variable that decides the outcome. Research published in July 2026 tested exactly that. Ben Sghaier and colleagues held the model fixed and varied only the harness across 35 consecutive releases of a CLI, evaluated on 50 stratified tasks from SWE-bench Verified, and traced the swings in quality back to harness changes. The paper's title makes the argument: Don't Blame the Large Language Model.

Anyone who builds software spends a lot of time worrying about process, architecture, design, business rules and context, and about keeping all of it coherent from the first commit through deployment and maintenance. So I built an AI SDLC incrementally, trying to follow the classic software engineering practices: spec, design, TDD and review.

Today the platform enforces some of those practices: a delivery missing its required canonical documents does not pass, an exported symbol with no consumer gets flagged, and destructive SQL never reaches the shell. Others it only teaches. TDD is a skill the agent reads, with no gate behind it. Code review is a CODEOWNERS template I hand to every project, and my harness has no way to guarantee it happens. The hook that would restrict pushes is registered and tested, but wired up to nothing.

That gap is the subject of this article: how do I know that the guardrails, controls, rules and hooks I am shipping are themselves following the rules I defined?

Or, more simply: if the harness is the thing being engineered, who tests the harness?

What this article is

An eleven-week engineering log of an internal platform: what I built, what I measured, what the measurement killed, and where the measurement itself was wrong. It describes a method and makes no claims about effectiveness; §07 spells out the limits.

§01 · Where this fits

What Böckeler's taxonomy leaves open

In April 2026, Birgitta Böckeler published Harness engineering for coding agent users, which gave the field its working vocabulary: Agent = Model + Harness, where the harness is everything in an agent besides the model. She splits controls along two axes. By direction, guides steer before the agent acts (feedforward) and sensors observe after it acts, so it can self-correct (feedback). By execution, computational controls are deterministic and fast (linters, type checkers, pre-commit hooks), while inferential ones are model-driven and slower (review agents, custom judges).

That 2x2 is a good map, and I am not going to redraw it. What matters here is the list of open questions Böckeler explicitly leaves at the end of her article. Four of them:

  • there is no equivalent of code coverage for evaluating the harness itself;
  • coherence remains unsolved as the harness grows: how to keep controls from contradicting each other;
  • there is no tooling to configure controls and keep them in sync across the delivery pipeline;
  • harness templates behave badly in legacy codebases carrying a lot of debt.

Reading it, I realized I might already have answered some of those questions. What I have are four partial answers, drawn from a working system, and §07 spells out how far they go.

Measuring the gap on GitHub

Before writing this, I used the GitHub API to survey the most-starred repositories in the space, asking a single question: does the harness test itself? I built the sample on August 19, 2026, from the visible canon of harness engineering, and reran the star counts on September 24, when the same nine repositories added up to roughly 600,000 stars.

Sample and file trees as of 08/19/2026; star counts refreshed on 09/24/2026, both via the GitHub API. None of the nine added harness mutation between the two dates. "Tests alongside hooks" counts files whose path contains both a hook marker and a test marker, a generous heuristic that overcounts. "Harness mutation" means any mechanism that deliberately breaks a control to prove that some test notices.

Repository Stars Files Tests alongside hooks Harness mutation
obra/superpowers 291,202 255 2 none
github/spec-kit 138,776 686 7 none
ruvnet/ruflo 73,218 7,117 34 none
bmad-code-org/BMAD-METHOD 53,425 785 0 none
SuperClaude-Org/SuperClaude 23,906 475 0 none
diet103/…-infrastructure-showcase 10,029 161 0 none
buildermethods/agent-os 5,444 31 0 none
disler/claude-code-hooks-mastery 3,926 153 1 none
karanb192/claude-code-hooks 524 236 21 none

The pattern shows up at both ends of the distribution. The most-starred of the repositories that explicitly bill themselves as infrastructure demos (10,029 stars, 161 files of hooks, agents and skills) has no tests at all for its own hooks. The best-tested repository in the sample is also the smallest, by two orders of magnitude: karanb192/claude-code-hooks ships a unit test per plugin and is the closest relative of everything I describe here. It includes plugins called config-guard ("who guards the guards"), protect-tests (which stops the agent from disabling a test to get to green) and dead-rules-audit (which scores how well the model actually follows each of its written rules and flags the chronically ignored ones for promotion to a hook). These are the right instincts, and the author arrived at them independently.

Not one repository in the sample deliberately breaks a control to see whether anything notices, and I could not find that in the arXiv literature either.

Every one of these repositories asks the agent to obey the controls. None of them checks whether the controls still work.

Related work worth reading

Two 2026 papers arrive at the same thesis. IACDM (Moreira) gates progress on a state machine outside the model ("the agent may request advancement, not grant it") and applies 19 critique lenses to 12 projects; every lens caught at least one defect that no other lens did. Agentic Harness Engineering (Lin et al.) evolves harnesses automatically, pairing each edit with a declared, falsifiable prediction, and raises Terminal-Bench 2 pass@1 from 69.7% to 77.0%. Neither one verifies that any individual control fires.

§02 · Mechanism 1: The registry

A gate nobody calls is just a document

The platform is an internal SDLC layer distributed as a plugin: 22 agents, 54 skills, 19 hooks, 33 gates, 467 markdown files and about 7,600 lines of shell, built over 140 commits between June 29 and September 16, 2026. What matters here are the gates, which intercept tool calls and session boundaries: a destructive-SQL guard, secret scanning, push restrictions, a plan-first check, a lint-and-coverage runner and a canonical-document check, among others.

The first problem I hit had nothing to do with a gate failing. It came from the gates simply existing.

The documentation said which gates were mandatory, a separate catalog said what each one checked, and a third file said how strictly each one was enforced. All three were hand-maintained prose, and all three drifted apart, because nothing in the system connected the sentence "this gate is mandatory" to a process really being invoked.

The fix was a registry: a single machine-readable file that serves as the authority on which gates exist. Every entry has to answer three questions.

// one entry, abridged
{
  "runner":     "hooks/anti-destrutivo.sh",
  "nivel":      "🔴",  // mandatory
  "inviolavel": true,
  "gatilho": {
      "tipo":  "EXEC",        // EXEC | PROSA-CMD | orfao
      "onde": ["hooks.json:PreToolUse(Bash)"]
  },
  "corpus":  "stdin JSON tool_input.command | arg $1",
  "harness_cases": [
      "anti-destrutivo/bloqueia-drop-de-tabela",
      "anti-destrutivo/bloqueia-delete-sem-where",
      … 10 more
  ]
}
Enter fullscreen mode Exit fullscreen mode

What it runs (runner), where it is wired up (gatilho.onde, literally "trigger.where": an actual event in an actual manifest) and what proves it bites (harness_cases: named, executable cases; see §03). On every commit, a checker validates the registry against reality. If a runner is listed but not wired up, wired up but not listed, or cites cases that do not exist, the commit fails.

The third trigger type is the one I would defend hardest. orfao (Portuguese for "orphan") means the control exists, is tested, and is currently wired to nothing. Three of the 33 entries are orphans. They are not bugs, and nobody hid them; they are declared dormancy, with the reason on record. Without that field, what you get is a hook sitting in a folder, looking enforced and never firing, which is exactly the failure mode this file exists to prevent.

  • 33 gates in the registry
  • 27 wired to an event
  • 3 invoked by a documented command
  • 3 declared orphans

This is my direct answer to the "tooling to configure and synchronize controls" that Böckeler says is missing. It is a simple fix: a JSON file plus a shell script that refuses to let the JSON lie. All of the value is in that refusal.

The rule underneath

Skip is not pass: a gate never goes green without checking. A missing tool counts as a failure, never as a skip, and "not applicable" is a separate, declared verdict. A false green is worse than no gate at all, because it is a gate you have stopped watching.

§03 · Mechanism 2: The harness

237 cases, one folder for each

A gate is a shell script that reads something and decides, which makes it testable like anything else. The format that survived real use is deliberately dumb: one folder per case, one file per assertion.

casos/anti-destrutivo/bloqueia-drop-de-tabela/
  runner         → hooks/anti-destrutivo.sh
  arg            → psql -c '<destructive DDL on a table>'
  exit           → 2
  espera_stderr  → destructive DDL
  motivo         → guard-rail #1: dropping a table never passes.
Enter fullscreen mode Exit fullscreen mode

No test framework, no assertion DSL, no mocking library. A runner reads the folder, executes the named script with the given input, and compares exit code and stderr. Cases that need a filesystem get a {SETUP} block; cases that need stdin use stdin.json instead of arg. There are 237 cases across 244 folders (seven of the folders are shared fixtures), and the gate entries in the registry cite 227 of them by name.

Two design choices matter more than the format.

Every case carries a motivo (Portuguese for "reason"). That file states the rule being defended, in the rule's own vocabulary, rather than describing the assertion. When a case goes red two months later, whoever is looking at it needs to know whether the rule changed or the code broke, and the assertion alone cannot tell them.

stderr is asserted along with the exit code. It looks pedantic, and it is the single most valuable constraint in the format, for a simple reason: a script that crashes on an unexpected argument also exits non-zero. If you check only the exit code, "the gate blocked the dangerous command" and "the gate is broken" show up as the same green. I know because I made exactly that mistake while measuring; it is one of the three errors in §06.

Cost, because slow gates get bypassed

The full suite runs in pre-commit. The mutation engine (§04) stays out of that path because it takes minutes; I run it by hand from a clean tree, after touching a runner and before cutting a release. The slowest gate I measured runs in 1.14 s against a declared ceiling of 60 s. That headroom is there for the case where something hangs, not for the normal case.

Why not a test framework

The harness has to run inside client repositories whose stack it does not control: Java, .NET, PHP, Node, Bun, Python. A test format that needs a runtime installed gets skipped exactly where it matters most. A folder and an exit code are the only things universal enough.

Live footnote

The command in the example above is obfuscated for a very literal reason. While I was writing this article, the platform's own destructive-SQL guard blocked the write to the file, because the test-case example quoted the command. The gate bit, and its error message explained the difference between quoting a command and running it, and offered a way around. That is the cheapest kind of false positive: it accuses, explains itself and shows its output, instead of quietly letting something through.

§04 · Mechanism 3: Mutation

Who tests the tests

237 green cases prove the gates behave correctly on 237 inputs, but they do not prove that any case would notice if a gate stopped working. That second claim takes a different mechanism: break the runner on purpose, mechanically, one small change at a time, and require that some case goes red. A surviving mutant is a real way to break that gate that no case detects.

The operator list is short on purpose, because too many operators generate equivalent mutants, and the noise ends up killing the mechanism:

Operator What it simulates
exit 2 → exit 0 the gate stops blocking
exit $FALHA → exit 0 the enforcement mode stops blocking
fail=1 → fail=0 a failure stops counting
if ! … → if … a condition loses its negation
-eq→-ne · -le→-gt · -lt→-ge a comparison flips
grep -q → grep -qv a match inverts
block "…" → : "…" the block call is neutralized

The operator I threw out

The obvious eleventh operator was && → ||, and I measured it and threw it out. It produced 35 of the 51 survivors in the first full run, a 2:1 noise ratio, and nearly all of them landed on optional-dependency guard lines like [ -f prisma/schema.prisma ] && roda …. Flipping the connective on those lines only changes behavior when the optional dependency is in the opposite state, which is exactly the state no fixture covers. A report nobody reads is the same kind of failure as a gate that is permanently yellow.

What the mutants found

They found three kinds of problem, and the first is the reason this article exists.

A false green. The problem was in two lines of the main gate orchestrator: the fail=1 assignments for two specific gates in block mode. Zero those out and the runner prints ⛔ FAILED to the screen and to stderr, then exits 0. A rejection message printed on top of a successful exit, in the orchestrator that everything else calls.

The part that should worry you

Those were exactly the two gates queued to go from warn to block at a client, so the code path about to go live in production had nothing at all proving it worked. Every case covering those gates was green because every case exercised them in warn mode, where fail=1 is never read.

A ledger that could go silent. An inverted comparison in the commit-tracking hook makes a deduplication condition always true once the ledger has more than one entry. From then on, the hook silently stops recording new commits, the session-close gate stops seeing any commits in the session, and the session closes clean. This one only became testable after I added a debug seam, for the same reason as its sibling: quietly exiting 0 is what the hook does when it works, and an exit 0 survives any mutation.

An uncovered branch that was mandatory on paper. The branch that checks database migration heads was marked mandatory in the catalog but had no case and no entry in the accepted-survivor list. It was a silent survivor, living in the gap between two ledgers.

The accepted-survivor ledger, and the bug inside it

Some mutants legitimately survive: equivalent mutants that change a string rather than a comparison, branches you cannot reach without specific tooling, environment-dependent paths. Those go into a file of declared survivors, each with a class, a reason and a date, and the engine prints the full list on every run. Any survivor not on that list fails the run. Debt is allowed as long as it is listed; silent debt is not.

That file had a bug of its own, and it is the most instructive thing in this section. Entries were keyed on runner + line number, and line numbers move. I measured it twice: on August 5, 2 of 14 entries had already come unanchored; on August 15, 3 of 12. One of them pointed at a grep -q in a Java branch that was no longer in the file: a dead entry holding onto a line number that any future mutant could inherit.

The failure mode is asymmetric, which is why it took me two measurements to take it seriously. A displaced entry shows up again as a new survivor, which is visible noise. But a real survivor that lands on an inherited line number gets accepted silently. The key is now runner + operator + code snippet, the line number is just documentation, and the engine warns whenever it re-anchors an entry. Erring toward noise is the only cheap mistake available here.

  • 102 mutants killed
  • 2 undeclared survivors
  • 11 declared, classified, dated
  • 237 green harness cases

For this version of the article I reran the engine on September 24, 2026, against revision 1.27.2, on a copy of the repository. The session hooks that trigger the engine run from the working tree itself, so a mutated hook would have taken effect in my own session mid-run. The result: 102 killed, 11 declared, and two undeclared survivors, both in gates added after the first version of this article. The observability gate, added September 2, stays green when exit 2 becomes exit 0. The PR-merge gate, added August 27, stays green when the comparison that detects a timeout (-eq 124) is inverted.

On August 20, the same run had come back with 98 killed and zero undeclared survivors, and it left the working tree byte-for-byte identical to how it found it. That zero did not maintain itself. The gates written afterward shipped with passing cases, and only mutation showed that those cases would not notice the gate stop blocking. Reaching the August zero took two rounds and four new cases. The first round logged six live mutants as "pre-existing and unrelated" and moved on, which is what a team does when the mechanism is new and the backlog is long. When I went through the six one at a time, none of them was noise: three were false greens and three were dead anchors.

Answering the coverage question

This is the closest thing I have to the metric Böckeler says is missing. It is a kill rate over a fixed set of operators, not coverage, which makes it a weaker and more honest claim. It says that for these ten mechanical ways of breaking a control, something notices. It says nothing about the eleventh.

Prior work

Mutation testing is 1970s technology, and Test Double has already written about using it to keep coding agents honest about product tests. The only new move here is aiming the mutation engine at the harness itself. It is a small step, and that is largely the point.

§05 · Mechanism 4: Measured precision

The rule that had to earn its way in

The house rule is that no rule becomes a gate without a measured signal, and that rejection is a legitimate outcome. The candidate here was export-sem-import (roughly, "export without import"), which flags a symbol exported in the task's diff that nothing consumes. It turns into code a rule that until then was just a sentence in a definition-of-done document: implemented, tested and imported by nobody is dead code, and dead code was not delivered.

Before writing the gate, I wrote a throwaway probe and ran it over the history of three repositories. I wrote down the threshold (precision ≥ 70%) before looking at any results.

Probe run on 08/10/2026. Repo C returned *not applicable, not clean, because it is CommonJS and has no ES exports to evaluate. Recording that as "clean" would have been the easiest lie in the measurement.*

Repo Commits Evaluations Clean Uncertain Accusations Rate (decidable only)
A (analytics workspace) 400 471 325 117 (25%) 27 7.7%
B (production gateway) 69 132 113 0 17 13.1%
C (marketing site) 16 0 — — — n/a

Then I hand-triaged all 17 accusations. Thirteen were real: the symbol was still sitting there with no consumer. Four were premature, because the symbol picked up a consumer later, and in three of those four the consumer arrived the same day, in a later commit in the same session. After a fix I describe below, none were false positives.

The number depends on what you call a unit

Precision per commit is 13/17 = 76%. Per task, which is how the gate actually runs (on the session's accumulated diff rather than commit by commit), it is 13/14 = 93%, because the three same-day accusations never fire. Same probe, same data, two answers, and they land on opposite sides of the line the industry uses.

That line comes from Google. Tricorder, Google's static-analysis platform, sets a ceiling: any analyzer whose results reach developers has to stay under roughly 10% effective false positives, because above that, developers start ignoring it and turning it off. Tricorder's own overall rate sits just under 5%. Against that ceiling, this gate fails as a per-commit check and passes as a per-task check.

False positive rate against the confidence ceiling

The same rule, measured on the same 17 accusations, in two different units of work.

Effective false-positive rate
Tricorder (Google), overall <5%
Tricorder ceiling 10%
This gate, per task 7%
This gate, per commit 24%

Google's number is the effective false-positive rate published for Tricorder's analyzers; the gate's two numbers are the complement of 93% and 76% precision over 17 accusations hand-triaged across two repositories. With an n this small, it can support a decision, not a benchmark.

I want to be careful with the conclusion, because there is a cheap one and a true one. The cheap one is "pick whichever unit flatters you." The true one, as I see it, is that the unit of measurement is a design parameter of the gate, and choosing it is the same decision as choosing the threshold. Here, aggregating per task describes what the gate actually does, so it is not a reporting trick: a symbol that picks up a consumer in the next commit of the same session was never a defect.

Three design decisions the measurement forced

A class of false positive the probe found for me. Version 2 flagged three symbols in a schema file, and all three were false: the consumer does import * as schema from "./schema" and the ORM reads the whole module, so the symbol name never appears anywhere. It is the same blind spot as an export * barrel, just with different syntax. Version 3 handles both the same way: a file consumed through a namespace import becomes uncertain, never an accusation. Measured effect: accusations went from 30 to 27, uncertain from 105 to 117, and false positives from 3 to 0.

Uncertain is a first-class verdict, and it accounts for 25% of the cases in the large repo. It is reported as incomplete, never as passed. It is the skip-is-not-pass rule again, and it is why the gate ships in warn mode rather than block mode.

A new gate always ships in warn. A gate that blocks from day one is red all the time, and within a week someone negotiates it away. It earns promotion only after a real stretch of work passes green.

Four days later, the gate was audited against its own rule

The audit found two blind spots, both of exactly the kind this platform exists to fight: cases the gate could not decide, without it saying that it did not know. The symbol parser recognized function, class and const/let/var, but export type and export interface fell through to a default branch and disappeared, counted as neither a finding nor uncertain. On top of that, the corpus was git diff HEAD, which cannot see untracked files, so before a git add the answer was "nothing to check." Both are fixed: the first case now yields a declared uncertain, and the fix for the second widened the corpus.

What the gate caught

An entire orphaned module in production: an HTTP handler serving /metrics that no file imported, so it never ran. A protocol file whose type guards were used only by its own test file: tested and covered, with no call anywhere in production code. The first measurement found eight of those; a later, sharper one found nine, in three distinct classes. And two cases where the file is imported but the flagged symbol is not, which a file-level check would miss by design.

Cost

3 min 02 s to sweep 400 commits in a 2,377-file repository, about 0.45 s per diff. In normal use the gate runs over a single diff, not four hundred.

§06 · The platform, audited

Two config files no code ever read

In August I ran the platform against a list of 97 findings that someone else had produced while dogfooding a different agentic pipeline. The ground rule: a difference between repositories is not a defect. Every item had to be reproduced by running this repository's own code, with the command and its output, or it did not count. Four items made it through that filter. The audit refuted, reframed or confirmed each one, and turned up six new findings of its own. Two of those six are why I would tell anyone building something similar to start with the registry and get to the gates later.

The enforcement levels were decorative. A config file declared, for each gate, whether it was set to off, warn or block. The file had been calibrated with clients and was documented, but through version 1.9.0 no code read it. The mode was edited directly in the runner, which even admitted as much in a comment. Anyone who set a gate to block in that file believed they had hardened their pipeline, and had changed nothing.

The threshold file lied about itself. A comment at the top of the quality-threshold file claimed the gate read its lint, test and coverage commands from there. It did not: the runner hardcoded the npm-ecosystem equivalents. The measured result was a production repository running 100% on Bun (765 tests passing, 49 skipped, none failing, 95.51% line coverage) that sat permanently red and could not close a session. Lint failed because there was no ESLint config file, and dependency analysis failed because there was no package-lock.json. Neither failure says anything about code quality; the gate was measuring the ecosystem, not the repository.

A config file that no code reads is just a lie with a schema.

Today both files go through a single reader, a function that every hook and runner calls, so the mode is resolved in exactly one place. The reader loads an allowlist and ignores any key not on it, which keeps a client's local config from loosening an inviolable gate by inventing an entry. A related mechanism deals with the legacy problem Böckeler raises. Each repository records the platform version it was onboarded on, and any gate introduced after that version reports dated debt in yellow instead of blocking. That way a repository that has already been delivered does not go red just because the platform kept evolving.

Most of what I measured, I killed

I did not expect this going in, and I think it is the strongest argument for the discipline: most of the candidate rules I measured during this period did not survive the measurement.

I wanted every one of these rules. Four of the five rejections came out of measurements I ran specifically to justify building them.

Candidate Corpus Verdict
Dependency outside the lockfile 2 accusations, both true rejected: the accusations are real, but they point to lock drift, which npm ci already fails on. Catching a truly hallucinated package would take a network call on every gate run.
Tool-repetition guard 1,985 transcripts · 49,712 calls rejected: exact repeats within a session are 1.31% of calls, and signatures repeated ≥3x show up in 3.5% of sessions. Hand-triaging 15 cases killed the rule.
Anti-tampering gate on the harness 123 commits × 6 surfaces rejected: one real loosening in the entire history, zero without accompanying product code, and zero without a written justification.
Eleventh mutation operator 51 survivors in the 1st run rejected: 35 of them came from that single operator.
Context-load pruning plan 53 skills · 6,047 chars rejected: nested sub-skills cost zero tokens, and the candidates for pruning turned out to be the most-referenced ones.
Visual-evidence gate 1,683 commits · 7 repos · 180 days approved: 405 commits (24%) touch UI, and 5 of the 7 repos have UI work, ranging from 22% to 67% of commits. The same table forced a redesign.

The anti-tampering case deserves a closer look, because it is the cleanest refutation I have. The hypothesis was that an agent, or a human on a tight deadline, would quietly loosen the harness: lower a threshold, delete a case, weaken an assertion. Sweeping six surfaces across 123 commits turned up exactly one loosening. It came with a written human decision, including a reason, an accepted cost and a review date, and the same commit added a harness case designed to go red once the rule is tightened again. The signal was zero. The only event related to the hypothesis was precisely the kind of auditable, declared loosening that a well-designed gate would have to let through.

The approved gate teaches the opposite lesson. The same table that justified building it also wrecked the original design: only one of the seven repositories declares how to drive a browser, so the gate as specified would report "could not verify" in every session in four of them. A warning that repeats with no way out becomes noise people learn to skip, which is just turning the gate off through the back door. That measurement is what produced the escape valve: a file of declared exemptions, each with a cause, an owner and a deadline, which fails closed if a declaration is incomplete.

Three times the measurement itself was wrong

On a single day in August, three measurements produced confident, wrong numbers.

  1. A shell loop over a concatenated list broke on a directory name with a space in it and produced 594 hits, along with the conclusion "this rule is unworkable; it will drown the team in noise." The real count was zero, and the rule was fine.
  2. I counted exit != 0 as "blocked," forgetting that a script invoked outside its manifest exits 1 on an unexpected argument. The conclusion was "all three wildcard forms are blocked." In fact, all three went straight through.
  3. A root path that was never resolved to an absolute path made a portability gate flag the platform's own name, as if it did not recognize itself.

The transferable lesson

All three survived because the result was plausible. The 594 hits confirmed that "this rule will generate noise," and "BLOCKED" confirmed that "the hooks work." The measurement that agrees with what you already believe is the one you check least, and that is exactly where all three were hiding.

One practical rule came out of it: before you conclude anything from a number, run a negative control, a case that has to produce the opposite result. And a gate that blocks needs to print why, because a non-zero exit code on its own can just as easily mean the script crashed.

On measuring before building

Every measurement above is written up as a document, with a date, a corpus, a threshold declared before the result, and an explicit statement of what was not measured. That format does most of the work. A measurement with no threshold declared beforehand is an argument looking for a number.

A related finding, at the prompt layer

The platform requires a documented failing baseline before any new skill gets written: you have to watch the agent fail without the skill, or the skill does not get in. This finding comes from the baseline for that very rule. Five fresh, independent samples were asked to write a skill and describe what they did to make sure it worked. Five out of five never ran a baseline or tested whether the skill changed behavior. What they called verification was checking that the cited file paths existed and that the frontmatter matched sibling skills. Since the prompt actively nudged them toward verification, the result is conservative.

The only third-party tool measured

A code-graph indexer, tested against the thesis that "grep finds names; the graph finds edges." Median tool calls fell 70% across 5 of 5 pairs and median tokens fell 57%, though 1 of 4 pairs flipped sign, at n=4. It passed with a declared caveat and was never adopted for anything: the measurement doc did not change a single file in the platform.

§07 · Declared limits

What this does not demonstrate

By the platform's own rule, debt is allowed as long as it is listed, and only silent debt is forbidden. Applying that rule to this article:

I make no effectiveness claim here, and I would have no basis to make one. I have not measured whether teams using this platform ship faster, ship better, or ship at all, and there is no control group. Given that METR found experienced developers misjudged their own speed by 39 points, my subjective sense of the platform's value is worthless, so I will not offer it.

The harness declares its own gaps, and they are real. On every run the suite reports three gates with no cases at all (no proof they bite) and five more with partial coverage, where a branch is declared but not proven. That goes on the scoreboard instead of being swept under the rug, which is the point, but do not read 237 green cases as 33 fully proven gates, especially with two undeclared survivors in the September 24 run.

Every n in this article is small: seventeen hand-triaged accusations, two repositories with ES modules, and five samples in the skill baseline. Twelve projects would already be a small study, and this is smaller. The numbers support the claim "this mechanism produced this signal in this context," which is enough to justify a design decision but not enough to generalize from.

The mutant kill rate does not measure coverage. Ten operators is a deliberately small set, chosen for signal-to-noise. Even a scoreboard with zero undeclared survivors would say nothing about failure modes those ten operators cannot express. The false green in §04 was found by fail=1 → fail=0, which is to say it only surfaced because someone happened to write the operator that expresses it.

This is one person's platform, dogfooded: eleven weeks, 140 commits, and two client repositories at the edges. The gates that matter most, the ones queued to go from warn to block, have not yet run in block mode for real. That is the biggest hole, and the false green in §04 shows what it looks like to catch one before it gets expensive.

Some of this is not new, and I would rather say so up front. Ablating a skill to check whether it changes behavior is already standard practice, with public tooling; policy outside the model is established literature; mutation testing is fifty years old. What I could not find in print was aiming the mutation engine at the harness itself and tying every control to the cases that prove it fires. Coming up empty after a few dozen searches does not prove it does not exist; if someone has already done this, I would love the reference.

What is worth stealing

  1. Write down which of your controls are wired to nothing. Put it in a field of the same file that claims your controls exist, not in a bug list. Three of my 33 are dormant, and knowing that is worth more than the three controls themselves.
  2. Assert stderr, not just the exit code. A crashed script and a blocking gate produce the same non-zero code, and if you only check the code, neither you nor your suite can tell them apart.
  3. Break a control on purpose this week. Change a fail=1 to fail=0, or delete a block call, and see whether anything goes red. If nothing does, you just learned the most useful thing in this article for ten minutes of work.
  4. Declare the threshold before you look at the result. Mine was precision ≥ 70%, written down before running the probe. Without that, every number is post hoc and every rule survives.
  5. Let measurement kill things. Five of my six candidates died. If your measurements only approve what you already planned to build, they have become ceremony, and the three wrong measurements in §06 show what that looks like from the inside.
  6. Ship every new gate in warn mode. A gate that blocks from day one is red all the time, and by Friday someone has negotiated it away.

None of this is really about AI. It is the oldest lesson in the discipline showing up somewhere new: an assertion nobody has ever watched fail is not really an assertion. We spent thirty years learning to distrust untested code, and then built a control plane for autonomous agents out of shell scripts and JSON that nobody tests. For the first six weeks, I did exactly that.

The harness is production code: it runs on every commit, it gates every delivery, and when it fails, it fails silently, and green. Treat it accordingly.

How this article was written

The platform, the measurements, the manual triage and every finding above are mine. I drafted the prose with AI assistance and then edited it myself. An article that argues for declaring what you actually did, rather than what you would have liked to have done, does not get to make an exception for itself.

Reproducibility

The platform is proprietary and will not be open-sourced. Everything needed to rebuild these four mechanisms is described above: the registry's three mandatory fields, the full harness-case format, all ten mutation operators, and the survivor-ledger keying bug along with its fix. None of them takes more than a few hundred lines of shell.

Sources

Engineering log · September 24, 2026 · Repository names anonymized at the author's request; all numbers as measured · Drafted with AI assistance, edited by the author.

Originally published on the Sozo Data blog.

Top comments (0)