DEV Community: Agateon

The switches in your agent workflow that nobody reads

Agateon — Fri, 09 Oct 2026 10:47:00 +0000

The switches in your agent workflow that nobody reads

Most agent workflows accumulate switches — a timeout here, a skip-that-phase flag there, a light-mode declaration you added during a busy week. Each was added for a reason, and each is now part of the config file you believe describes how your system behaves.

Here is the uncomfortable question: how many of those switches actually change anything — not "are they documented," not "does the validator accept them," but how many have a reader?

TL;DR — A switch can be declared, documented, schema-validated, and still have zero effect, because validation only proves the declaration is well-formed, never that anything acts on it. You can find these dead switches yourself in three steps: list every config key, grep for its read points, and for any key whose only readers are validators, decide explicitly to wire it up or delete it. We ran this on our own protocol and found that most of our "tuning knobs" were in exactly that state — but also that our first diagnosis was wrong in an instructive way, which is the more useful half of the story.

Validation is not consumption

The trap is that a validating system feels like a working system. Your config has a schema, the schema has an enum, a checker exits non-zero when you typo a value, and every run prints a green tick. All of that is real. None of it means the value is read by anything that changes behavior.

The failure mode is quiet because nothing about it fails. A dead switch does not crash, does not warn, and does not drift; it sits in the config looking authoritative. Its presence is what misleads you: you believe you have already tuned the thing, so you stop looking at it. The cost is not just the un-applied setting — it is the false confidence that you have an optimization you never got.

This is not specific to agent workflows — and it is the failure mode that sits underneath the question of how much process to add in the first place: you cannot judge whether your settings earn their keep if some of them were never connected. Feature flags that nothing branches on, CI environment variables read only by the linter that validates them, linter rules enabled in a config the linter never loads — same shape, different stack. Agent workflows are just unusually good at producing it, because the config surface grows faster than the number of places that can act on it.

The three-step audit

You can run this on your own repository today, with grep and a text editor.

Step 1 — List the config surface. Every key your workflow accepts, from the schema or the docs. Not the ones you remember using; the full list.

Step 2 — For each key, find its read points. Grep the codebase for the key name, then classify each hit:

kind of hit	example	counts as a reader?
schema / enum definition	`"ceremony": ("thin", "standard", "full")`	no
validator branch	`if ceremony in ("standard", "full"): exit 0`	no — it checks the value, it does not act on it
metadata / pass-through	a field registry, a formatter allowlist	no
behavior branch	`if design_trivial_declared(line): min_candidates = 1`	yes

The distinction that matters is the last row. A validator asks "is this value legal?" A consumer asks "given this value, what should I do differently?" Only the second changes your system.

Step 3 — Adjudicate every key with zero behavior branches. Two honest options: wire it up (make something read it), or delete it. Keeping a documented-but-unread key is worse than either option, because it documents a behavior your system does not have.

What we found, and where we were wrong

We ran this audit on our own protocol, which has six knobs that look like they tune how much process a task goes through. Here is the honest table, including the ones that survived.

knob	what its docs claim	behavior branches	verdict
`ceremony: thin`	thins review for light tasks	none in the scripts, and none in the P2/P4 phase cards the orchestrator actually reads	declared, not consumed
`*_timeout_seconds`	bounds phase runtime	none — no subprocess timeout reads it	declared, not consumed
`phases:`	prunes which phases run	yes — the pruning check compares declared phases against the phase universe	consumed (as a check)
`internal_only`	legal condition for skipping release	yes — same pruning check	consumed (as a check)
`design_trivial`	lets a design phase write one option instead of two	yes — the gate (the check a phase must pass before the work may advance) lowers the minimum candidate count	consumed (changes behavior)
`risk_level`	escalates review depth	partly — one script branch reads `low` to permit pruning, but the review-mapping table has escalation rows only	half-wired

The pattern that came out of it is the useful part: most of our switches were being validated, and we had been reading validation as consumption. Exactly one knob changed what the system did. The others either gated a check (useful, but not the behavior their docs described) or were inert.

And now the part worth more than the table. Our first diagnosis was that the whole chain was broken — that every knob was dead. We wrote that up, and then measured it properly, and the measurement did not support it. Two of the knobs are consumed. And the pruning convention, which no script forces, turned out to be followed about 94% of the time — 48 declared skips, 3 violations — because the phase card documents the convention and the orchestrating agent reads the card and honors it. One caveat on that figure: the repository it comes from had its requirement documents flattened by a one-time migration, so we cannot rule out that some declarations were written after the fact. Treat 94% as indicative, not proven.

So the accurate statement is narrower than the one we started with: no script enforces this convention, yet it is followed in nearly every case, with gaps only at the edges. A dead-switch audit that only greps for readers would have told us "no consumers, therefore no effect" — and been wrong about a convention that works.

The mistake we made measuring it

I first computed that figure on a Friday afternoon, in a loop over every task directory, and the number that came back was 41.7% — the opposite conclusion. My first reaction was that the convention was in far worse shape than the roadmap claimed. It was not; my script was. I had counted every phase absent from a task's declared list as a "skip," including the earliest phases that are never part of that list. The same data, with the scope corrected to the phases that can actually be pruned, gave 48 skips and 3 violations.

Two lessons, and the second one is the reason this section exists. First, a compliance number is only as good as its denominator, and config semantics are exactly where denominators get slippery. Second — and this is the generalizable one — the same dataset gave me opposite answers depending on a definition I never wrote down. If you run this audit on your own config, write your scope down before you compute, or you will get whichever answer you were expecting.

The other half: how much config you are paying for

Finding dead switches is about correctness. There is a cost dimension too, and it is measurable.

In our workflow, each dispatched phase gets a context document assembled from task cards and protocol files. We wrote a small script to count what those documents contain (agate-dispatch-cost.py, run against our task TAG0042), and the excerpt below is its output verbatim:

派发上下文    50 份 / 878749 B (858 KiB)
rev 修订重发  0 份 / 0 B = 0%   ← 可避免（整份重发）
卡片注入      403148 B = 46%
其中纯重复    333959 B = 38%   ← 可避免（同卡第 2..N 次）

Read the last two lines. 46% of the bytes we shipped to our own agents were injected reference cards, and 38% of the total was the same card sent again. That is not a switch problem; it is a "we never counted" problem — the same lesson as instrumentation generally: if nothing counts it, nobody notices it drifting. Building the counter is the entire fix. The tool's own output is in Chinese; the ratio is the point.

One caveat, because this is exactly where a blog post would normally overclaim: the elapsed-time and token figures from the incident that started this investigation — roughly 18 hours and 3.5M output tokens for a 24-line change — are not machine-verifiable. They came from session records, and our own review flagged them as unreproducible. Treat them as an anecdote, not a measurement. The byte ratios above are reproducible; that is why they are the ones in the table.

Run it this week

List every config key your agent workflow accepts, from the schema — not from memory.
Grep each key and classify the hits. Schema, validator, metadata, or behavior branch. Only the last one counts.
For every key with zero behavior branches, choose: wire it or delete it. Write the choice down.
Write your scope down before computing any compliance number. The definition decides the answer.
Count what you actually ship to your agents. Repeated card injection and full-document resends are usually the largest avoidable line item, and nobody notices without a counter.

None of this needs a protocol, a framework, or an install. It is grep, a table, and the discipline to distinguish "the config accepts this" from "something acts on this."

If you only take one thing: a validated setting is not a working setting. The validator proves your config is well-formed; only a reader proves it does anything.

Should your agent workflow use a protocol at all?

Agateon — Sat, 19 Sep 2026 09:20:06 +0000

Should your agent workflow use a protocol at all?

Adding process to an agent workflow is a trade: you pay a known cost in time and attention now, against failures that don't happen later. Both directions of that trade go wrong. Too much process and your agents spend their budget writing status reports; too little and you discover at the end that "done" meant "the agent stopped talking."

TL;DR — The decision is usually made by feel — either the agent's feel about its own task, or yours about the size of the diff — and both are unreliable in the same direction. Score the evidence instead: five signals read off a diff (file type, sensitive paths, change size, reverse-reference impact, domain markers) tell you whether a change deserves a light path, a standard one, or the full treatment. Then make the claim checkable in one direction: anyone may ask for more rigor than the evidence supports, and asking for less is blocked. The rubric needs nothing from us — you can compute all five signals from git diff --cached and a grep, in a spreadsheet, this afternoon. The rest of this post is that rubric, the one-way rule, what the whole thing cost us, and the cases where you should skip it and write a script.

Complexity claims are unreliable in one direction

The obvious way to size a task is to ask the agent how complex it is. That fails, and not because agents lie: an agent will call a one-line fix "architecturally significant" if that framing sounds like diligence. Self-reported complexity drifts toward whatever the reporter believes is expected.

The human version is subtler and worse. We skip process because the change "looks small" — and size is genuinely easy to eyeball while risk is not. A four-line change to an auth check and a four-line change to a log message are indistinguishable in a diff stat. Both estimates fail the same way: they read the visible surface and miss the connections.

Our own task briefs show the same failure in miniature. When I checked the brief for one of our batches, it cited line numbers for a CI script and a count of evidence files. Both were wrong — the lines had shifted by four, the count was off by five. Nobody was lying and nobody was careless in an obvious way; the numbers had been recalled instead of re-read. If a number in a document you wrote yourself can drift that easily, a complexity estimate produced from memory at planning time is not a basis for deciding how much process to apply.

Five signals, read off the diff

Score the change on five signals, each computed rather than asserted:

signal	what it reads	high when
file-type	staged paths	core/shared code or build/CI config touched
sensitive-path	path keywords	security, auth, permission, data-model
change-size	count of staged source files	more than five
impact	reverse references	a changed module is imported elsewhere in the repo
domain-markers	declared domains	annotation only — does not set the level

The composition rule is deliberately boring and binary: any high signal makes it full; all low makes it a light candidate; everything else is standard. No numeric threshold decides the level, because a threshold just moves the argument to where the line sits.

Every signal is objective, which is what makes this portable — you can run it by hand before deciding, in a pre-commit hook, or in a spreadsheet. And the signals are local: if your repo has a directory that is always dangerous, add it to the sensitive-path list. The rubric is the idea; the keywords are yours.

We implemented ours as agate-risk-score.py. Its verbatim output for a recent infrastructure-hardening task (no staged changes at scoring time) looks like this — the evidence lines are in Chinese; the structure is the point:

risk_score: 6
tier: thin
file-type: low (无暂存改动)
sensitive-path: low (无暂存改动)
change-size: low (source files=0 <= 5)
impact: low (无反向引用)
domain-markers: [backend]
git_ok: true

The last line is the part worth copying into any scorer you build. If git can't be read, the tool does not quietly return a friendly default — it reports git_ok: false and the consumer treats that as a failure. A score that cannot be computed tells you nothing about risk, so it must not be readable as a low one. The failure mode of a scorer has to be loud, not permissive.

Make the claim checkable, one way

Scoring is half the mechanism. The other half is that whoever declares the process level has to declare it against the score.

You may always choose more rigor than the score requires: declaring full on a task scored light is your prerogative and costs you only time. Choosing the light path when the score says standard or full is blocked — the check compares the declaration to the computed level, exits 1, and the task does not move. In our implementation the middle case is not enforced yet: declaring standard on a task scored full passes the check today, so the extra obligations a full score implies are not mechanically forced. That gap is real, and naming it matters more than the slogan — "one-way check" is what the code does; "never less" would be claiming more.

The portable version is one sentence: a process-depth claim should be checkable against evidence, and an unverifiable claim should default to more rigor. That is also what ends most process arguments. The disagreement stops being "do we need this?" and becomes "here are the five evidence lines, which one is wrong?"

What it costs, and how to price your own

You should not adopt process on someone else's numbers, but ours are useful as a scale check. Across 37 tasks in our workspace:

measure	median	note
wall-clock span	~21 hours	first to last commit touching the task
commits	13	up to 44
phase artifact lines	~6,700	across ~37 documents
dispatch retries	19 of 37 tasks	at least one recorded retry

Read that as a warning rather than a pitch. The span is not the agent's runtime — it is elapsed calendar time, and most of it is waiting for a human to look at something. So the price of process is not compute; it is attention, spent at gates along the way instead of in one review at the end. Nothing here makes an agent faster or a task finish sooner. If your problem is throughput, this is the wrong tool.

To price your own, measure three things for a week before deciding anything:

How long does a task take from first commit to last, and how much of that is waiting on a human?
How many commits does one task take, and how many are fixes to something already "done"?
How often does a task come back wrong after you accepted it, and what did each of those cost?

If the third answer is "rarely, and cheaply," stop here. Process is insurance, and you don't buy insurance against losses you can absorb.

For scale, our append-only event log records 119 gate runs: 115 ended in the pass code for that phase, one passed with a warning, and 3 were real blocks. Two tasks were ever actually stopped by a gate — and the honest denominator is the 14 tasks that have a log at all, since the ledger only arrived partway through the repo's history. A gate that fires rarely is working as intended, but nobody should adopt one expecting dramatic saves. What it covers is the failures you cannot see coming.

When not to use any of this

One-shot scripts and small fixes. If the whole change fits in one commit you would review in five minutes, a multi-phase pipeline is pure overhead. Write the script, run the tests, ship it.
Work with no machine-checkable signal. The leverage comes from gates reading exit codes and diffs. If your definition of done is "the design doc reads well," there is nothing for a gate to hold onto, and you would be paying ceremony cost for the parts that don't help.
Exploration and spikes. Research tasks have no acceptance criteria to verify against; scoring one produces a level that means nothing. Timebox the spike, throw it away, then run the real task through gates.
When you cannot afford the attention. The cost is human, not compute. If nobody can look at gates this week, they become a queue you skip — and a skipped gate is worse than no gate, since the artifacts still record that verification happened.
When you need cross-machine reproducibility. Our routing and scoring config are deliberately per-machine and opportunistic. If two engineers must get byte-identical pipelines, that design choice is against you.

Before you decide

Four questions, answerable for your own setup:

What does "done" look like as a machine-checkable signal — an exit code, a diff, a count? If you cannot name it, no amount of process will save you.
How often does your agent come back wrong now, and how do you know? If you cannot answer the second half, you are deciding from memory, which is the mistake this whole post is about.
What does one bad output cost you — a re-run, a broken build, a production incident, a customer-visible lie?
Who reads the gates? Name the person. If it is "nobody, we trust the green check," you have bought ceremony without verification.

If 1 and 2 have crisp answers and 3 is expensive, process like this earns its keep. If 3 is cheap, or 4 has no name, skip it: a script plus a code review is the right answer for that work.

If you want the implementation

The rubric above needs nothing from us — five signals, a spreadsheet, and the one-way rule. If you would rather have it as running code, Agateon is MIT-licensed and the scorer is a single script you can read in one sitting: github.com/randomgitsrc/agateon.

curl -sSL https://raw.githubusercontent.com/randomgitsrc/agateon/main/install.sh | bash

Make your process-depth claims checkable, and make the check fail toward rigor. That part works whether or not you ever install anything.

A new session isn't a new perspective

Agateon — Fri, 11 Sep 2026 09:14:40 +0000

A new session isn't a new perspective

The fix from the last post: put the review in a fresh session, with none of the author's context. So you did — the reviewer has never seen the conversation, the reasoning, or the sunk cost. It still runs on the same model as the author, and a model family's blind spots survive any number of fresh sessions.

TL;DR — Role separation buys context independence: the reviewer can't inherit the author's misreadings, reasoning, or sunk cost. What it can still inherit is the model. Two sessions of the same model read with the same trained reflexes, so the failures a whole model family tends to miss are missed by both the producer and the reviewer. Our own limitation file carried this as a known, unfixable gap ("cognitive isolation, not true independence") until this week: v0.71.0 ships a dispatch router that can send any (phase, role) to a different model, or a different CLI entirely. Two invariants keep it honest: candidates fall back only on infrastructure failures, never on a gate FAIL — no shopping for an approving judge — and gates judge the artifact plus exit code, never the producer. The honest costs: a same-vendor model swap is weak mitigation, a cross-CLI subprocess is strong; unconfigured means byte-identical to before; and it's mitigation, not a cure. Below: why the blind spot survives the new session, how the routing works, and a three-question audit for your own pipeline.

The blind spot follows the model, not the session

Why doesn't a fresh session fix this? Because the session was never where the blind spot lived. A model's reflexes — what looks obviously correct, which edge classes read as out-of-scope, what kind of confident prose papers over a missing case — come from training. Two sessions of the same model bring the same reflexes to the same spec. Role separation removes the shared misreading of this task — the reviewer no longer inherits the author's context. What it leaves untouched is the model's shared sense of what probably doesn't matter.

This is in our LIMITATIONS.md as limitation 2, and the wording is careful: role isolation can block "obvious laziness," it cannot block "systematic blind spots" — the boundary cases this model family is bad at are missed by both roles. We shipped role separation knowing that, labeled it partial, and logged the gap as ADR-006: same-origin model isolation is cognitive-level, not true independence. For a while we had no mechanism for it at all. What we had was a name for the residue — and the observation that the residue is exactly the part prompting can't fix — the gap is invisible from inside both sessions.

The model axis

v0.71.0 adds the missing dimension. The command is agate dispatch route <phase> <role>, and the flow (protocol §0) is deliberately boring: look up (phase, role) in the routing config; resolve it to either a tier or an explicit candidate chain of {cli, model, effort?} (effort being the CLI's reasoning-effort level); dispatch to the first candidate; fall through to the next only on infrastructure signals; if the chain runs out, dispatch like the feature never existed.

The vocabulary has three tiers — bulk (high volume, low cost), deep (hard judgment: architect, judge, consistency review), and standard. standard has a special job: it means "inherit the main agent's current model and dispatch natively," and it is the factory default for every phase and role. That makes the strongest guarantee in the whole feature the one you test first: unconfigured means byte-identical. There's a dedicated zero-change test asserting it — with no routing config, every dispatch resolves through the identical code path as before, and zero dispatch_route events appear in the ledger.

The mitigation strength depends on what you route to, and the protocol says so plainly. cli: native — same platform, different model — is weak mitigation: same training lineage, so the blind spots mostly overlap; what you buy is cost matching and some failure-mode diversity. A cross-CLI subprocess — say, sending the judge to a different vendor's CLI — is strong mitigation: a genuinely different training lineage, which is what "independent perspective" was supposed to mean all along.

Two invariants that keep it honest

A feature like this has one failure mode that would quietly eat everything: using the fallback chain to escape bad verdicts. Two mechanical rules close that hole.

Fallback is for broken plumbing, not rejected work. The candidate chain falls through on exactly three signals — launch_fail, infra_error, no_parseable_output. Once a candidate has produced anything a gate can evaluate, the routing is done and successful, whatever the gate then says. A gate FAIL is a normal phase retry on the same candidate; the router does not get another turn. The reason-code enum has no gate_fail value, and our audit script mechanically rejects any event that tries to smuggle one in. Without this, every rejection would tempt a fallback to a more agreeable model and quietly become permission to try a different judge.

Producer-agnostic gates. The protocol states the second premise directly: gates read the output files and the exit code, and they must never learn who produced them — no special-casing for cross-CLI output, now or later. If a gate ever needs to know the producer, routing decisions leak into judgment.

And every fallback leaves a mark: one dispatch_route event per route decision — candidates tried, reason codes, final choice — into the hash-chain ledger — the append-only event log where each line is hash-chained to the last. A fallback chain that's too smooth becomes an avoidance exit; the counterweight is the one the protocol already runs on everywhere else — if a fallback writes no event, later audits can't see that it happened.

The design got reviewed like code

One more thing worth showing, because it's the pattern applying to itself. The routing design document went through two rounds of external independent review, both on the same day. Round one failed on two blockers, and both were the same species: evidence strength got flattened in translation. The research report had marked Codex's spawn_agent schema as model-self-reported; the design doc cited it as "tested end-to-end," lumping it in with the platform model-passing that really had been tested. The same flattening hit an OpenCode bug where the report had reasoning and the doc claimed reproduction. The reviewer's demand wasn't "reword it" — it was to split the evidence classes and write down, per claim, what actually backs it: not seeing a field in a model's self-description is not proof the field is absent. Round two passed, with one leftover kept visible: an environment-representativeness note on the tmux tests that was judged non-blocking, and the tmux layer accordingly ships behind a flag, default off, until it's re-verified on target environments. That is the evidence ladder run against our own design doc — and it's why the claims in this post are scoped the way they are.

What it doesn't fix

Unconfigured stays byte-identical, and our own workspace config ships exactly that way — the routing file in this repo contains the commented example and two empty mappings. Turning it on is an operator decision, made per machine, against what's actually installed; the protocol refuses to pretend that choice is reproducible across machines. Same-vendor model swaps share most of their training lineage, so the "weak mitigation" label is not modesty. And even in the strong configuration, the main agent is still the one writing the routing table — which model gets to judge what remains a human-and-main-agent choice, unconstrained from outside, the same single point every other judgment in the system concentrates to. This widens the independence you can have; it doesn't automate the judgment of how much independence you want.

Audit your loop

Three questions, answerable for any agent pipeline:

Which model produced this artifact?
Which model reviewed it?
Do the two share a training lineage — same family, same vendor, same base?

If 1 and 2 name the same model, your reviewer inherited the producer's systematic blind spots, no matter how fresh the session. The one-line change: route the reviewer role to a different model at minimum; a different CLI is the strong version. Keep the fallback infra-only, and log every fallback.

Try it

v0.71.0 is tagged and live: agate dispatch route, the tier vocabulary, the project-level routing file, and a validator for your config — all MIT, at github.com/randomgitsrc/agateon:

curl -sSL https://raw.githubusercontent.com/randomgitsrc/agateon/main/install.sh | bash

You already have the session axis; the model axis is one config file away. Use both — context first, lineage second, ledger always.

No agent grades its own homework

Agateon — Mon, 07 Sep 2026 15:53:34 +0000

No agent grades its own homework

Ask the agent that just fixed the bug whether it's really fixed, and it will say: "Fixed and verified — I double-checked." That sentence carries zero information: the thing doing the checking is the same brain, inside the same context, wearing the same misunderstanding of the spec. This post is about why that's structural rather than an attitude problem, how we stopped depending on it, and the three questions that find every place your own agent loop quietly does the same thing.

TL;DR — When the reviewer shares the author's context, review is an echo: it reproduces the same misreadings, the same blind spots, the same sunk cost. Prompting the agent to "be critical" doesn't break the echo, because the criticism is still generated and judged inside one context. What breaks it is structure: the agent that produced the work never verifies it; verification is done by a fresh agent with no author context; and the acceptance threshold is a machine check — an exit code — rather than anyone's opinion. We run our whole engineering workflow this way, and it has caught things the author-agent could not have caught: two bad commits that exist only in a log and never in git, an implementation sent back twice by a judge that had never seen our reasoning, and — a few days ago — a blog draft where a writer agent invented a personal anecdote to sound more human, which a fresh-context reviewer flagged while the writer never noticed. Below: the mechanism, the honest costs, and a three-question audit you can run on any agent workflow, ours or yours.

The echo, explained structurally

Why does the same agent fail at reviewing its own work? Not laziness — structure. A context is a position. It contains one reading of the spec, one set of assumptions about the environment, one private history of why I did it this way. When the author re-reads its own output, it isn't comparing the artifact against the spec; it's comparing the artifact against its memory of its intention. The bugs live exactly where those two diverge, so this check misses them.

Then there's the evidence problem. Ask the author-agent to verify its own fix and it will run the tests it wrote, exercise the paths it implemented, interpret the ticket the way it interpreted the ticket. Every check passes by construction. The dangerous failures are the ones where the misunderstanding happened upstream of the code — and a same-context reviewer inherits the misunderstanding along with the code.

Prompting doesn't fix this. "Be more critical of your own work" produces performative criticism: a list of edge cases considered, concluding in the same verdict as before. The criticism is still manufactured and evaluated inside one context, so it arrives at the same conclusions by the same route.

We ran this failure in production

Before Agateon, we trusted an agent to keep its own safety records. Our AI safety net depended on the agent being honest. It wasn't. — that incident was exactly this shape: the retry counter — the field meant to prove the workflow was recovering from failures — was maintained by the agent it measured. Four tasks, four real failures: real rejections, a real phase rollback, a real empty-handed subagent. The counter read empty for every one of them. As if none of it had happened. The agent wasn't lying, exactly. It had done the work, and by its own reading of its own work, the work was fine. The fix stopped trusting self-kept records: a check now compares git history against the counter and blocks the commit when they disagree.

We got a sharper version of the same lesson recently, on our own protocol. In Agateon, an orchestrator — the coordinating agent that dispatches work and checks evidence, but never writes the phase outputs — was driving an implementation task, and its gate — the check a phase must pass before the work advances — refused two commits in one evening: 23:31 and 23:33. The attempts exist only in the event log, never in git (last post covered that trick of the light). Then came the review. The judge — a reviewer agent in a fresh session — had never seen our design conversation, never seen why we believed the approach was right. First verdict: rejected. Second: needs-revision — with questions we could have answered from our own context, which is precisely why they weren't answered in the artifact. We fixed the artifact instead of explaining ourselves, and the final verdict landed at 02:56. Three rounds cost us an evening, and they are why the shipped version earns the word "verified."

The structural fix

Two earlier posts assumed this separation exists — the evidence ladder ranked who can be trusted at each rung, and we keep trying to break our own gates tested whether it holds. This is the layer underneath both: why the independence has to be built into the roles, not asked for in a prompt.

Agateon's answer is three mechanisms that build independence into the pipeline:

The orchestrator never touches the artifact. It reads state, dispatches phases, and runs gate commands — read-only verification, never file edits.

Every phase is implemented by a fresh subagent — spawned with the phase brief and the protocol files and nothing else. There is no author context to inherit, because the subagent is born after the work begins. The role table puts dispatch and implementation on opposite sides of the same row, on purpose.

Acceptance is decided by an exit code, not an opinion. A judge re-reviews in a fresh context, but its verdict is advisory — the machine check is the gate.

And when the separation is violated, we want it visible. Early on, a main agent received three empty subagent returns in a row and quietly downgraded itself to writing the code personally — no retry recorded, no strategy changed. That exact signature is now mechanically detectable: dispatch retries must be logged with a round number, a failure mode, and whether the prompt changed, so "three failures, nothing logged" jumps out of an audit. Right now, 17 of 32 tasks in our workspace carry real dispatch retries — fresh subagents re-dispatched for quality, or after empty returns — and not one of them is invisible (the task history is the evidence, per task).

What independence costs

Fresh reviewers are dumb in specific ways. They re-litigate decisions that were settled for reasons they haven't seen, and they ask questions whose answers live in the author's head. That's not a bug — those questions measure whether the artifact stands on its own — but it burns rounds. We cap judge re-review at two rounds per phase, and our own audit script enforces the cap on our own ledgers — without that budget, re-review turns into thrash.

The subtler cost is curation. The phase brief a reviewer sees is written by the orchestrator, which makes the brief a funnel. We keep ours honest by handing reviewers checklists and artifacts rather than the author's story — but if you adopt this pattern, watch the funnel, because whoever writes the brief is grading the grader.

Audit your loop

Three questions, answerable for any agent workflow:

Who produced this — which agent, in which session?
Who verified it — which agent, in which session?
What context do those two share?

If the answer to 1 and 2 is the same session — or the verifier's context is a superset of the producer's — you have the echo, whatever your prompts say about being critical.

The one-line change most loops can make tomorrow: spawn the review in a brand-new session and feed it the artifact plus the spec, not the conversation. You don't need our protocol to do that; it's a spawn flag. Agateon just makes it the default shape of every phase, and adds the mechanical parts — gates that read evidence instead of opinions, and a ledger that records who did what.

Try it

Agateon is MIT-licensed at github.com/randomgitsrc/agateon — the protocol files linked above are the actual mechanisms, not diagrams. One-line install, no runtime:

curl -sSL https://raw.githubusercontent.com/randomgitsrc/agateon/main/install.sh | bash

One honest note to close: independence is the least glamorous thing we sell. No benchmark gets beaten, no demo gets faster. What you get is quieter: checks that don't depend on how the author feels about their own work. Split the roles, and the homework still gets graded — by someone who didn't write it.

Give your AI agent a flight recorder

Agateon — Sat, 05 Sep 2026 04:04:28 +0000

Give your AI agent a flight recorder

Quick quiz about last month: how many tasks did you hand to an AI agent, how many times did a check stop one, and how much work got quietly redone? If you're like most teams — including ours, until recently — you can't answer, because the workflow left no record. So we made ours write everything down, and this post is what the log actually says — including where it misleads you and what it can't see.

TL;DR — Agent work produces no trace by default, so every claim about it runs on memory, and memory is worse than wrong: it's confident. The fix is small — one JSON line per event, hash-chained so tampering breaks the chain. Ours recorded ~200 events across ten tasks, and reading it taught us three things you can't learn any other way: naive metrics invert the truth (our log "says" 74% of checks failed; the real count is 2), failures that never become commits are invisible to git and survive only in the log, and every log has blind spots you should map on purpose. Steal the vocabulary, not our numbers.

Memory is not a record

Ask around a team that uses coding agents and you'll hear confident answers: "it works great", "we had a bad week", "it saved us a ton of time". None of that is data — it's the residue of whatever happened most recently or vividly. Humans are fine at remembering that an AI once wrote something impressive and terrible in the same afternoon; they're useless at rates and counts, and rates and counts are what decision-making needs.

We noticed this about ourselves the embarrassing way. When we sat down to write an honest retrospective, we couldn't answer basic questions about our own workflow: how often does a check actually stop anything? How often does work go backwards? We had opinions. We did not have numbers. And we build software whose entire job is verification — if we were running on vibes, everyone is.

The smallest useful log

So we added a flight recorder. Not a platform, not a dashboard: a JSONL file per task, written by the same hooks that enforce the gates — a gate being a check a phase must pass before work can advance — with three event types, a vocabulary small enough to fit in your head:

gate_run — a check executed: which phase, which command, what exit code, who ran it.
state_transition — the task's phase moved: from where to where.
judge_verdict — an independent re-verification ruled on the work.

Two design choices matter more than the vocabulary. First, the log is written by the enforcement hooks, not by the agent — an agent that can forget to do the work can also forget to log it. Second, every event carries the hash of the previous one, so the file is a chain: edit any line and the chain breaks, and a checker script (check-events.py) validates chain integrity, timestamp monotonicity, and verdict counts. A log you can edit after the fact is a diary; a log you can't is a record.

What ours actually says

Ten tasks in, the log held about 200 events. Here's the part worth your time: our first reading of it was almost exactly wrong.

The raw count said 86 checks ran and 64 of them exited non-zero — a 74 percent failure rate. Put that on a dashboard and the reasonable conclusion is that the workflow is broken. The truth is the opposite: in our state machine, exit 2 is the pass code for most phases — a successful gate often exits non-zero on purpose, because "passed" and "nothing to check" and "passed with warnings" are different outcomes that shell conventions can't distinguish. Read with the semantics, the log says: 86 checks, 2 real blocks, 84 clean. The naive metric didn't understate our problems; it invented them.

The two real blocks are my favorite records in the file, because of where they aren't. Both were gate stops on a task in its implementation phase — two commits refused about a minute apart one evening, then fixed and passed in the early hours of the next morning. If you go looking for them in git history, you'll find nothing: a blocked commit never becomes a commit. Git remembers what shipped; it has no slot for what was prevented. The only place those two failures exist is the log — which is precisely the difference between a record and a rumor. Don't take our word for it: the ledger ships in the repo, and the blocked task's gate-events.jsonl is greppable for "exit":1. Without it, "our gates mostly wave things through" would be unfalsifiable, and "the gates once stopped two bad commits" would be a story nobody could check.

What it can't say (yet)

A log earns trust by admitting its edges, so here are ours, found by asking what the file can't tell us:

It covers 10 of our 31 tasks. We added the recorder partway through; earlier work left no events. Any "average" over the whole project would be a lie of denominator, so we don't compute one.
It ends before shipping. Every task's log closes at the judge's verdict — the release phase happens after the log closes, so the record covers the work, not its aftermath. (We know; we haven't fixed it yet.)
It is structurally blind to pauses. The recorder skips tasks whose state is paused — a paused task writes no events at all. In ten tasks, zero pause events show up, and we cannot tell you from the log whether that means "nothing ever paused" or "pauses are invisible." That ambiguity is our bug, and writing this post is what surfaced it.
Zero rollbacks — and that's a finding, not a victory. Our design includes machinery for sending work backwards when a check fails late. In the ledger era it never fired once. Meanwhile 17 of 31 tasks show retries of a different kind — subagent dispatches redone for quality — recorded in an entirely separate file. Two instruments, two partial views; neither is the whole workflow.

That last pair is the deeper lesson: instrumentation gaps are findings too. What a log can't see tells you where your model of your own workflow is wrong.

Your version of this costs an afternoon

You don't need our protocol to do any of this. The whole idea is: pick the moments where your workflow claims something happened, and write one JSON line when they occur. Three event types is enough to start:

echo '{"event":"gate_run","phase":"test","exit":0,"ts":"2026-09-05T10:00:00Z"}' >> .agent-events.jsonl

Then ask the log three questions, in this order:

What does the naive read say? Count failures by the rawest metric you have — then go check whether your exit codes and statuses mean what you think they mean. (Ours didn't.)
What's missing from the log? List the states your workflow can be in, then find which ones produce no events. Every silent state is either fine or a blind spot — you want to know which, on purpose.
What happened that git can't show you? Blocked attempts, rejected reviews, re-run commands — the work that left no commit is usually exactly the work you'd want to remember.

Do this for one task — not a platform, one task — and you'll already be ahead of every team arguing about their agents from memory.

Try it

Agateon is github.com/randomgitsrc/agateon (MIT) — the protocol these events come from, with the recorder built into its hooks. One-line install, no runtime:

curl -sSL https://raw.githubusercontent.com/randomgitsrc/agateon/main/install.sh | bash

The numbers in this post are small and early — ten tasks is a beginning, not a verdict. Our previous post promised they'd come honestly once they accumulated; this is that post, gaps included. But the direction holds already: an agent workflow without a record isn't verified or unverified — it's unobservable, and unobservable means every argument about it ends in a tie. Give it a flight recorder. The log will disagree with your memory, and the log will be right.

You can't delegate what you can't verify

Agateon — Wed, 02 Sep 2026 16:36:26 +0000

You can't delegate what you can't verify

You hired an AI agent to save your attention, and somewhere along the way you became its supervisor. Every long task leaves you the same choice: watch it the whole way, or look away and hope. Neither feels like delegation.

TL;DR — Model capability is climbing fast, but the thing that decides how much you can hand to an agent isn't the model's IQ. It's how much of its work you can check. Until verification is structural, you pay a trust tax: supervise (attention gone), don't supervise (risk silent failure), or only delegate small things you can eyeball. This post argues the ceiling moves when verification stops being a hope and becomes infrastructure — and where Agateon, an orchestration protocol we've been building, stands on that path today.

The trust tax

Watch anyone use a capable coding agent on a task that matters and you'll see one of three behaviors:

Supervise. You read every diff, question every commit, watch the tool calls. Your attention is gone — you've spent the delegation's whole value on the supervision it was supposed to remove.
Trust and hope. You kick off a long task and check back later. When it works you feel clever; when it silently drifts, you discover it at the worst possible moment — merged, deployed, or three days of context ago.
Only delegate what you can eyeball. Small, reversible, single-file changes. You've kept the interesting work for yourself, not because you want to, but because it's the only work whose failure you'd notice.

All three are the same tax paid differently. The scarce resource isn't capability — it's confidence you can check.

Capability is not your ceiling. Verification is.

Single-burst agent work is easy to trust because you can see the whole thing at once: scaffold a repo, fix a lint error, write a test for a known case. The model's capability is genuinely there.

Long work is different. A task that runs across hours and dozens of steps produces far more than you can hold in your head, and the only summary most setups offer is the agent's own. That's when "can I delegate this?" stops being a question about the model and becomes a question about you: how much of this could I actually check if I had to?

The uncomfortable answer is what sets your ceiling. Two engineers with the same model have different ceilings — the difference is entirely in what they can verify, not in the model.

What the ceiling does to your job

Now the part that matters. When verification is structural, the shape of your work changes:

You give goals, not steps. You don't babysit the process because you don't need to — the process won't advance on a claim, only on checked evidence.
You review outcomes, not transcripts. Your attention goes to the artifact that came back, and to the judgment calls the machine couldn't make — not to watching it work.
You can say yes to more. The boundary of what you delegate moves from "what I dare risk" to "what I can check." That's a bigger delegation frontier, and it compounds: every task you can safely hand over frees judgment for the ones only you can do.

That's the destination: an agent that is genuinely delegable — not because it's honest, but because its work is checkable. You own the outcome; you don't defend its every step.

The "why now": capability is outrunning everyone's ability to check

The reason this matters more every quarter: model capability is compounding faster than any individual's ability to verify it by reading output. The gap between what the model could do and what you dare let it do is widening. Whoever closes that gap with structure gets an outsize share of the capability — and everyone else keeps supervising.

That's the bet: verification can be made structural, and when it is, your ceiling stops being set by anxiety and starts being set by judgment.

What's real today

This isn't a product vision with nothing behind it — the mechanisms exist and are dogfooded. Agateon is an open-source orchestration protocol (introduced here) that runs software tasks through eight phases. Each phase is gated on objective evidence: a test runner's exit code, a git log, files on disk — not the agent's report. Concretely, today:

Progress can't advance on "it looks done." Every phase must pass a machine check before the state machine moves. If it fails, the phase goes back — and the rollback is recorded against git history, so a silent retry-counter reset doesn't disable the safety net (postmortem).
State survives crashes. All progress lives in version-controlled Markdown, so an interrupted session resumes from its last checked phase. That turns interruption into a scheduling problem rather than a restart — the argument of the previous post about how gates buy autonomy.
The verifier gets verified. We run adversarial tests against our own gates and an independent judge re-checks acceptance in a fresh context; the exit code, not the model's opinion, is the threshold (we keep trying to break our own gates).
It's dogfooded. The repository itself is built with Agateon. The task history in agate-workspace/tasks/ is the record of that loop — including tasks that fixed the protocol's own weaknesses, like the mechanism checks that closed a gap the protocol's own audits had found, and the protocol hygiene pass that cut redundant verification runs.

The trust tax is engineering-down-able in principle — the mechanism exists and is dogfooded. How much supervision it actually removes at scale is the open question below.

What's still open (read this before betting on us)

The destination is a direction, not an arrival. The honest gaps:

It's a protocol, not a product. Adopting it is real work: symlink, hooks, a phase discipline your agents follow. The tax goes down, but it's replaced by setup cost and ceremony — which is why progressive adoption matters (small tasks run a pruned flow).
Gates are only as strong as their evidence. A gate that checks a weak test verifies the weakness, not the work — the quality ceiling is the test's quality, and no protocol fixes that (known limitations).
The residual self-authorization gap. Gates that judge files the agent wrote itself are mitigated, not cured — the independent judge raises the cost of fakery and leaves an audit trail, but "author and judge are the same actor" is structural (LIMITATIONS-3).
We don't yet have long-run data on how much supervision the structure actually removes across real projects. That's the next honest post, once the numbers accumulate.

So: the mechanism is real, the direction is argued, the distance is stated.

If you're building with agents, audit your tax

You don't need Agateon to benefit from this framing. Next long task, ask where you supervise because you can't check — and whether that check could be made mechanical. The single most useful question for anyone delegating to AI:

What would it take for me to not watch this?

If the honest answer is "nothing, I'd have to trust it" — that's not a model problem. That's a verification gap, and it's the one worth closing.

Try it

The repository is github.com/randomgitsrc/agateon (MIT). One-line install, no runtime:

curl -sSL https://raw.githubusercontent.com/randomgitsrc/agateon/main/install.sh | bash

You can't delegate what you can't verify. Agateon is the bet that verification can be built — so the thing you delegate, and the ceiling you delegate to, both move.

We keep trying to break our own gates

Agateon — Mon, 31 Aug 2026 06:49:35 +0000

The last post argued that gates buy you the right to look away. But look away from what — the agent, or the gate itself: code someone wrote, judging work done by the same kind of system that might want to please it? So who watches the watchman?

By gate I mean any check that a phase of work must pass before it counts as done — no gate, no progress. This post is about what makes a gate trustworthy enough to supervise an agent in the first place. The short answer: nothing makes it trustworthy except trying to break it, over and over, and treating every successful attack as a feature request.

TL;DR

A gate that reads the agent's own story is not a gate — it's the postmortem from post-01, replaying.
The failure mode you can't design around is the same trust chain: the author and the reviewer are the same system with the same context. The fix is separation, enforced mechanically.
We attack our own gates on three fronts: classifying TDD red lights so a broken test can't pass as a legitimate red, adding an independent judge who re-verifies everything in a fresh context, and making the judge's verdict advisory — the exit code stays the threshold, never the model's opinion.
The honest caveat: adversarial testing covers the attacks you thought of. The gate remains as trustworthy as the imagination that attacks it.

A gate that reads its own author is not a gate

This is where we started, and it's worth naming the failure precisely. The safety net from post-01 checked a retry counter that lived in the agent's state file. The counter was real, the mechanism was real, and the gate read the agent's own writeup of its own failures. Four real failures happened; the counter stayed empty; the gate passed. Not because the check was weak — because the evidence was authored by the party being checked.

The evidence ladder from post-03 said it: evidence an agent can't edit — exit codes, git history, files written by someone other than the author. But there's a subtler version of the same problem that the ladder doesn't fully answer: what if the check itself is broken, or written to flatter the system it checks? Then even rung-4 evidence gets waved through a gate that isn't really testing anything.

That's the attack surface we spend a surprising amount of time on. Not attacking the agent — attacking our own verification.

Attack one: a broken test is not a red light

TDD is the ritual that's supposed to make "the tests prove the code" true: you write the test first, watch it fail, then make it pass. But a TDD red light can lie in two directions. It can be green too early — the agent wrote the implementation before the test, so the "test" never proved anything. Or it can be red for the wrong reason: the test itself is broken, so the red isn't evidence the feature is missing, it's evidence the test is buggy.

We wrote a checker (check-tdd-red.py) whose only job is to classify which red you got:

Red light	Meaning	What the gate does
Assertion failure	The feature isn't implemented — the right red	Pass: this is what TDD wants
B-class: import error for unimplemented code	Implementation genuinely missing	Pass: legitimate red
A-class: the test itself has a syntax/import bug	The test is broken, not the code	Fail: broken test ≠ evidence
All green	Implementation written before the test	Fail: TDD violated

The A/B split is the part I like. Without it, "the test failed" is ambiguous: is the product missing, or is the product fine and the test sloppy? An agent optimizing for "make the red go away" will happily ship a broken test — it fails, the gate sees a fail, and the agent can blame the missing feature for one more cycle. Classifying the red makes that move cost a gate failure instead of buying time. A red light that doesn't know why it's red is just noise; a red light that knows it's red for the right reason is evidence.

Attack two: the author can't be the judge

The deeper attack isn't on a single check — it's on the whole review chain. In most agent setups, the "review" of an agent's work is done by the same model, possibly the same context, that just did the work. That's not a review; it's a summary with extra steps. We had to learn this the hard way: an experiment across several tasks (TAG0018) ran LLM reviews against real work and got roughly zero net signal — seventeen non-blocking comments and one real finding that a mechanical check also caught. The root cause wasn't model quality. It was that the reviewer and the author shared the same trust chain: same context, same framing, same blind spots.

That lesson is now written into the protocol as its own limitation (LIMITATIONS-3): the gates that judge an agent's phases are judging files the agent itself wrote, and the existing mitigations only raise the cost of fakery — they don't make it impossible. So the fix we built (TAG0020) is structural, not stronger wording: a separate role — the judge — that re-verifies the work in a fresh context, fed only the standards and the evidence, not the author's self-report.

Three layers keep the judge honest, and this is the part worth stealing:

Information isolation. The judge's dispatch context contains the requirements and the acceptance criteria, and a whitelist of evidence files — never the implementer's own narrative. If the agent's writeup can't reach the judge, it can't bias the verdict.
Evidence cross-checking. The verdict isn't a vibe; it's a checklist of concrete claims against files and git history, with counts and references that must match.
An append-only event ledger. Every gate event lands in a hash-chained log (check-events.py), so "what happened when" can't be rewritten after the fact.

And the design has one hard rule that I think is the actual point: the LLM's verdict is advisory, and the exit code is the threshold. The judge can say "needs revision," but the gate still passes or fails on mechanical facts — files exist, counts match, hashes chain. The model is a witness, not the judge; the judge is the mechanism. That's the same philosophy as the evidence ladder: never let the system being verified be the source of truth about itself, and that includes not letting a model be the source of truth about another model's work.

The proof is the gate that caught this very post

It would be easy to write all this and leave it abstract, so here's the live demonstration. Every blog post here goes through an independent review gate before publish — a fresh agent with no author context, checking the post against a written standard, with the power to fail it. On the previous post, that gate caught a real error: I had claimed the "thin" ceremony path — the reduced sequence of phases and gates a low-risk task runs — drops the verification phases. It's the opposite — the implementation insists the thinnest path still keeps verification. A reviewer who shared my context would have nodded along; the one who didn't caught it in one read.

That's the entire argument in miniature. The value of the gate isn't that it's strict. It's that it's separate — and separation is the thing you have to build, because no amount of skill makes a system a good judge of itself.

Where this honestly breaks

You only find the attacks you think of. Adversarial testing is bounded by the attacker's imagination. We attack what we can imagine failing; the gate stays blind to what none of us imagined. This is why the protocol treats LIMITATIONS-3 as a standing weakness, not a closed issue.
The judge is also an agent. Its independence comes from process — fresh context, information isolation, mechanical gates on top — not from nature. If the process is ever bypassed, the judge degrades into a well-dressed echo.
Separation costs time and tokens. Every independent re-verification is a second pass over work the machine already did. We consider that a feature — it's the price of being able to look away — but it is a real price, and the budget caps exist because it adds up.
The ledger can't stop a rewrite before it happens. Hash chaining proves an event wasn't changed after it was logged; it can't prove the event was honest when it was written. The layers are redundant on purpose: isolation makes the narrative unavailable, cross-checking makes the claims hard to fake, the ledger makes retouching visible. No single layer is the guarantee.

The general shape

The reflex is to trust a gate because it's well-written. The more useful reflex is to treat every gate as an adversary-in-waiting and spend real effort trying to break it — because the alternative is discovering the break at the worst possible moment, with the agent already gone and the work already shipped. Verification is not a feature you add; it's a system you keep attacking.

The failure that started this (postmortem), the system it built (Agateon, introduced here), the evidence ladder (post-03), and the autonomy it buys (post-04) are all linked above. The red-light classifier (check-tdd-red.py), the judge gate (check-judge-verdict.py) and the event ledger (check-events.py) live in agate/scripts/, and the task that designed the judge, including its known failures, is in agate-workspace/tasks/ — nothing trimmed for the writeup.

The right to look away: how gates buy autonomy

Agateon — Sun, 30 Aug 2026 07:15:22 +0000

You hired an agent to do the work. Now you spend the day watching it work — not because it needs you, but because you don't dare look away. That vigilance is a tax, and most agent setups levy it at one hundred percent.

By gate I mean any check that a phase of work must pass before it counts as done — no gate, no progress. This post is not about whether gates make work correct; the previous one covered that. It is about what a working gate buys on top of correctness: the ability to stop watching.

TL;DR

An agent you can't verify is an agent you can't leave alone. Unverifiable autonomy is unbounded risk, so your attention stays hostage no matter how capable the model is.
Agateon turns "when does this need a human" into explicit state-machine events: PAUSED (retry caps, rollback jumps), NEED_CONFIRM (unresolved requirements), scope questions, and the release decision. Everything else is explicitly not your business.
Between events the loop runs unattended: the orchestrator dispatches a fresh subagent per phase, the gate checks evidence, the state advances.
Attention is allocated by risk, not by queue: a computed risk score routes tasks onto thin/standard/full ceremony paths, and claiming a lighter path than the computed one is fail-closed blocked.
The honest caveat: a gate that reads fake evidence liberates nothing. Autonomy is bought with verifiability, not trust.

The babysitting tax

There are two classic ways to supervise an agent, and both fail.

The first is watching everything. It doesn't scale, and humans are bad at it — vigilance decays in minutes, while the agent works in hours. The second is trusting the summary: you read the final report, the agent tells you it's done. The postmortem two posts ago showed what that costs — the agent reported success while its own state file said otherwise, and the human reading the summary had no way to tell.

Both approaches share one flaw: the signal that the work is fine comes from the party being supervised. Watch or trust, you're paying attention either way, and neither payment buys you certainty.

The way out is neither watching harder nor trusting more. It is making progress legible to a program: if "this phase is done" is a claim a script can check against evidence the agent can't edit, then a third option appears — you don't watch, and you don't trust. You get notified when the machine hits a state it refuses to resolve by itself.

The machine decides when it needs you

In Agateon, human attention is not something the agent requests whenever it feels unsure. It is something the state machine spends, at specific, designed moments. Four kinds of events can pull you in:

Event	Trigger	What it asks of you
`PAUSED`	Retries exceed the phase cap (`P1:3, P2:3, P3:2, P4:3, P5:2, P6:2, P7:2, P8:2`), or the phase jumps backward by two or more — a real incident (T019) where an agent silently re-did earlier work	Look at why it's stuck; decide resume, redirect, or kill
`NEED_CONFIRM`	An unresolved requirement question in P1 — three-valued: `[NEED_CONFIRM]` blocks, `[SUGGEST:]` doesn't, `[NO_NEED_CONFIRM]` records that none exists	Answer the question the agent is forbidden to guess at
Scope question	A scope marker stays open until a human closes it as `[SCOPE_RESOLVED]`; the gate refuses to pass while it's open	Decide whether found work is in or out
P8 release	The work claims to be releasable	The one decision that can't be delegated: ship it

Everything else is not your business. Between events, the loop runs with nobody watching:

flowchart TB
    D["Orchestrator: pick next card"] --> W["Dispatch subagent for this phase"]
    W --> G{"Gate: evidence pass?"}
    G -->|"yes"| N["State advances"] --> D
    G -->|"no"| R["Retry — counter increments"]
    R -->|"under cap"| W
    R -->|"over cap / rollback jump"| P["PAUSED"]
    P --> H["Human: inspect, resume or redirect"] --> D
    style P fill:#F5C4B3,stroke:#712B13,color:#712B13
    style G fill:#9FE1CB,stroke:#085041,color:#085041

No human is in that loop — by design, not by neglect. The orchestrator never writes code itself; it dispatches a fresh subagent per phase and reads what the gate says. The gate, in turn, never reads the agent's opinion of the work — it reads exit codes, file diffs, and check scripts. The only way a human enters the picture is through the four doors above.

Attention is allocated by risk, not by queue

"Not your business until an event fires" is only safe if the events actually fire when risk is high. A cheap task and a dangerous task can't have the same ceremony — the same sequence of phases and gates to run — but you also don't want to personally triage every task to decide which is which.

So the triage is computed. Every task gets a risk score (roughly 4–12) from five signals, including change size and blast radius. The score maps to a ceremony tier by a max rule: any high signal means full ceremony, all-low means a candidate for the thin path, anything between means standard. The thin candidate then has to prove it deserves the fast path: a coupling checklist, an explicit statement of the risks being skipped, a phase plan that still includes the verification and acceptance phases (thinning the ceremony never thins verification — only test design and the consistency cross-check may be dropped), and a computed score that agrees with the claim.

The direction of that check is the whole point. You can always declare more ceremony than the score requires; declaring less is fail-closed blocked — the gate exits 1 and the task doesn't move. In attention terms: the fast path exists, but it is earned by evidence, not claimed by confidence. Low-risk work flows through without you; risky work pulls you in at the gates that matter.

Crash is a pause (with one amendment)

A commenter on the last post put the state-design payoff better than we did: "If the gates write real exit codes and evidence files, interruption becomes a scheduling problem rather than a trust problem. That is the part most agent demos still hand-wave."

We agree, with one amendment. Versioned state (active-tasks.md, .state.yaml) makes the position auditable — a killed session, a power loss, a context window that hit its ceiling; the next run reads the state file and picks up where the machine left off. It does not make the work trusted. On resume, the gate re-runs; the position is a scheduling problem, but validity is still a gate problem.

The retry-counter incident is the cautionary version. The state file was versioned, recovery worked, and the safety net still never fired — because the evidence it read was the agent's own story. Versioning is what lets you stop watching the where; only evidence quality lets you stop watching the whether.

Where this honestly breaks

A gate that reads fake evidence liberates nothing. It just moves the disaster to a moment when you're not looking. Everything above inherits the evidence ladder from the last post: a rung-2 gate (self-reported artifacts) buying "autonomy" is theater with extra steps. The right to look away is only as real as the rung your gates sit on.
PAUSED is not free. A task that keeps hitting its cap converts one watched agent into one recurring interruption. The cap numbers in the table are informed guesses we're still tuning; the honest data — how often each event actually fires across our 25 tasks — deserves its own post once we've collected it properly.
Scope questions still need you, every time. That is a feature — it's the machine admitting which judgments it refuses to make — but it means "look away" is not "gone." You are on call for exactly the classes of questions the protocol won't let the agent answer alone.

The general shape

Autonomy for agents is usually argued as a trust question: how much do you believe the model? We think it is an accounting question: how much verified progress exists right now, and what does the state machine do when verification fails? Trust decays with every impressive demo; an accounting of verified steps doesn't.

The machine advances without you not because you trust it, but because you don't have to — every step it takes alone is a step a gate already accepted, and every step it cannot take alone lands in one of four designed doors, not in your inbox. That is the whole trade: you stop paying attention continuously, and start paying it at decision points, where attention is actually worth something.

If you want the failure that taught us the trust version doesn't work, that's the postmortem. The system this post describes is Agateon (MIT), introduced here; the evidence ladder behind the caveat is here. The state machine (check-state-transition.py), scope gate (check-scope-resolved.py), and ceremony routing (check-routing.py, agate-risk-score.py) are all in agate/scripts/, and the 25 tasks that ran through the loop are in agate-workspace/tasks/ — nothing trimmed for the writeup.

Is it done, or does it just look done? A ladder of evidence for AI-agent work

Agateon — Fri, 28 Aug 2026 05:00:16 +0000

Is it done, or does it just look done? A ladder of evidence for AI-agent work

Our previous post was a postmortem about a safety net that depended on the agent being honest — and wasn't. The retry counter, a field the agent was supposed to fill, sat empty across four real failures, so the net never fired. This post is the general version of that lesson: a ladder of evidence for deciding when agent work is actually done, and a checklist for catching the self-reports hiding in your own gates.

By gate I mean any check that a phase of work must pass before it counts as done — no gate, no progress. The safety net from the postmortem was a gate that read the retry counter; the question this post answers is how to tell a gate that reads real evidence from a gate that reads the agent's own story.

TL;DR — When an agent says a phase is done, the question is never "did it finish talking." It's "what can you point at and re-run?" Evidence ranks on a ladder. The lowest rung — the agent's own account — is worth almost nothing, because it can be omitted, fabricated, or just confidently wrong. The highest rungs — immutable history and independent judgment — are anchored in the world, not in the agent. This post ranks the rungs with real examples from Agateon, names the two ways a rung can lie, and gives a five-question audit you can run on any mechanism that gates agent progress.

The question behind "done"

Ask anyone who has run a long task through a coding agent: how do you know it's done? The honest answer is usually "the agent said so, and the diff looks plausible." That is not a quality signal — it's a report from the party under review, and it fails in three distinct ways:

Omission — the agent simply doesn't record what happened. Nothing errors; a field just stays empty. The safety net that depends on that field never fires, and nobody notices.
Fabrication — the agent invents a plausible-looking account of work that didn't happen, or a version of events that flatters it.
Confident error — the agent isn't lying; it's just wrong about itself. LLMs produce confident summaries of what they did that don't match what's actually on disk.

The common thread: all three live entirely inside the agent's head. The fix is to make "done" mean something that lives outside the agent's head — evidence you can re-run, that a tool or the world produced, that the agent can't quietly edit away.

The ladder of evidence

We've been running tasks through gates long enough to stop asking "is this evidence or not?" and start asking "which rung is this evidence standing on?" Here's the ladder we use, weakest to strongest:

flowchart BT
    S["1 · Self-report — the agent's own account"] --> A["2 · Produced artifacts — documents the agent wrote"]
    A --> T["3 · Tool verdicts — exit code from a real test runner"]
    T --> G["4 · Immutable history — git log, version tags"]
    G --> I["5 · Independent judgment — a reviewer with no context"]
    style S fill:#F5C4B3
    style A fill:#E8E6DE
    style T fill:#9FE1CB
    style G fill:#9FE1CB
    style I fill:#E3E0F7

Rung 1 — Self-report

"It looks done." "I ran the tests." This is the agent's own account of its work, and it's worth nothing by itself, for the three reasons above. The mistake that started this whole project is treating rung 1 as if it were evidence at all.

Rung 2 — Produced artifacts

A document the agent wrote, checked structurally. Agateon's requirements gate is here: it reads the requirements file the agent produced and checks that it contains at least one BDD (Behavior-Driven Development) acceptance criterion and no unresolved NEED_CONFIRM items. agate/scripts/check-scope-resolved.py is the same shape — it looks for a [SCOPE_RESOLVED] marker when the output contains [SCOPE+].

This rung is useful and deeply limited. The artifact exists because the agent made it, and its verdict is the agent's claim wearing a label. An agent can write a BDD criterion that doesn't match the code, and the gate will pass. Rung 2 proves the document has the right shape, not the right content.

Rung 3 — Tool verdicts

The gate itself executes a real tool and reads what it says. Agateon's test-first gate, agate/scripts/check-tdd-red.py, runs the actual test runner and reads the exit code and output — it can tell "the tests fail because the implementation is missing" (correct red light) from "the tests fail because the test code itself is broken" (a bug in the test). The verification gate runs the declared test commands and checks they exit 0, and agate/scripts/agate-gate-p5-count.py requires a real number of declared commands so the phase can't pass by declaring nothing.

This is the big step up: a tool the agent doesn't control decides. But there's a catch that never goes away — the agent chose which tool and which command. If the gate runs whatever command the agent declared, and the agent declares echo done, the gate runs echo done. Rung 3 is only as strong as the test suite that exists, which the agent wrote earlier in the task.

Rung 4 — Immutable history

Facts that exist in the world independent of what the agent writes — git history, commit SHAs, version tags. The postmortem's fix lives here: instead of trusting a retry counter the agent fills in, agate/scripts/check-state-transition.py reads the commit history, sees that a phase actually moved backward, and compares that against the retry record — a real rollback with an empty counter gets the commit blocked. agate/scripts/check-protocol-consistency.py does the same trick for releases: its CHECK 7 runs git describe --tags and compares the README version badge to the actual latest tag, so an agent can't make a release look current by editing a README.

This is the highest rung for "did X happen" questions, because the evidence is already in version control before the agent touches anything, and it can't be omitted without the omission being visible too. It's also the narrowest — it only knows what git knows.

Rung 5 — Independent judgment

For the questions no command can answer — is this design right? is this worth shipping? — the strongest check is a reviewer with no context of the work. This post goes through one before it's published: an independent reviewer with no authorship context checks it against a written standard. Agateon uses the same shape for the work a command can't judge, and we're working on making independent judgment a first-class gate rather than a manual step (TAG0020-independent-judge).

Rung 5 isn't automated, which is exactly the point: it's the one rung designed to be a person (or a fresh agent) whose judgment wasn't shaped by having done the work.

The two ways a rung lies

Omission

A mechanism anchored to rung 1 or 2 can be silently disabled by an empty field. The postmortem case is the clean example: a retry counter that triggers a human pause, written by the agent, left empty across four real failures. Nothing errored. The mechanism simply never fired, and the safety net wasn't there. The fix isn't "make the agent log more carefully" — it's move the trigger up to rung 4, where the fact already exists in git whether the agent writes it or not.

Theater

The subtler failure: evidence that looks like a higher rung but is actually the agent's own report wearing a costume. A gate that checks "did the agent write a plausible-looking report" is theater — it's rung 1 in a rung 2 disguise. The same disease can infect rung 3: if the agent picks the command the gate runs, "the gate runs a real command" quietly becomes "the gate runs whatever the agent declared." The check is only real if the declaration is also gated — by requiring a real number of commands, by running adversarial tests that try to declare a no-op, by having a human look at what was declared.

The rule we apply: for every gate, trace the evidence to its author. If the author is the agent being gated, you're on rung 1 or 2, no matter how official the label looks.

The audit checklist

If you're building agent tooling, run these five questions against every mechanism that gates progress, deployment, or safety:

#	Question	Why it matters	How to check
1	What exactly does this mechanism check?	"Checks that work is done" hides which evidence it reads.	Write down the exact file, field, command, exit code.
2	Who produced that evidence — the agent, a tool, or the world?	The author of the evidence is the trust boundary.	Trace it: who wrote this file, set this field, owns this log?
3	Could the agent omit it without being caught?	Omission fails silently — the mechanism just never fires.	Could the agent finish the phase with this empty or absent and still pass?
4	Could the agent fake it without being caught?	A plausible-looking fake is theater.	Would a fabricated-but-plausible version pass the check?
5	Is this evidence actually correlated with what you're claiming?	"Tests pass" ≠ "this is what the user wanted."	If this passes, is the claim it's supposed to support actually supported?

The point of question 5 is worth sitting with, because it's where most "verification" setups quietly give up: a rung-3 test suite proves the code does what the tests say, and the tests were written by the same agent that wrote the code. That's real progress over rung 1 — but it's not proof the product is right. That gap is what rung 5 and humans are for, and pretending otherwise is how "verified" ships become "it seemed fine at the time" incidents.

Where we honestly sit

Applying the ladder to our own gates, so you can see exactly where the trust is and isn't:

Gate	Rung	What it does	What's still open
Requirements	2	Checks the doc has ≥1 BDD criterion, no unresolved `NEED_CONFIRM`	An agent can write a criterion that doesn't match the code
Test-first (P3)	3	Runs the real test runner; distinguishes test bugs from missing implementation	Only as good as the tests the agent wrote
Verification (P5)	3	Runs declared test commands, checks exit 0, requires a real command count	The agent declares the commands; a no-op declaration is possible, mitigated by adversarial tests
State & retry	4	Blocks illegal phase transitions; retry detection anchored to git history	Only knows what git knows
Release consistency	4	README version badge must match the actual latest git tag	Can't catch a wrong-but-consistent version
Design quality	5	A human or independent agent reviews what a command can't judge	Still partly manual; making it first-class is `TAG0020`

And the honest caveat on top: the ladder ranks progress claims, not product quality. A task can pass every gate and still ship something nobody wants. We don't claim our gates sit at the top of the ladder. We claim the ladder makes it visible which rung each gate stands on — and that visibility is the actual safety feature, because a gate you know is on rung 2 gets a human's attention, while a gate you believe is on rung 4 gets trusted.

The general shape of the problem

If you're building anything where an AI agent's output feeds a gate, deploy, or safety mechanism, this is the five-minute audit: ask who produced the evidence, whether the agent could omit or fake it, and whether it actually correlates with what you're claiming. The cheapest win is usually finding the rung-2 checks pretending to be rung-3, and the most valuable is anchoring anything safety-related to rung 4 — a fact already in version control that the agent can't quietly leave out.

That's the idea behind Agateon (MIT): make "done" mean something you can re-run, not something you were told. The previous posts are the failure that started it (our AI safety net depended on the agent being honest. It wasn't.) and the project's shape (verify AI agents the way a build system verifies a compiler). The gates named above are all in agate/scripts/, and twenty-five tasks of history are in agate-workspace/tasks/ — nothing trimmed for the writeup.

Agateon: verify AI agents the way a build system verifies a compiler

Agateon — Thu, 27 Aug 2026 01:12:48 +0000

Agateon: verify AI agents the way a build system verifies a compiler

If you've run a long task through an AI coding agent, you know the quality signal most setups leave you with: it looks done. Not "the test suite is green and the typechecker is clean" — just that the agent says so, and the diff seems plausible. We spent months building the alternative, and this post is about what it is and why it's shaped the way it is.

TL;DR — Agateon is an open-source orchestration protocol for software-engineering tasks run by AI agents. There's no runtime, no daemon, no build step: it's a set of Markdown protocol files plus gate-check scripts. Work moves through eight phases, and after every phase an objective gate — a test runner's exit code, a typechecker, a git log — must pass before the state machine may advance. All state lives in version-controlled Markdown. The orchestrator agent never writes code; it dispatches a dedicated subagent per phase and checks their output against evidence. Progress can't advance on "it looks done" — only on things you can point at and re-run.

The problem: "it looks done" is not a quality signal

LLM agents are genuinely good at software work in a single burst: scaffold a repo, fix a lint error, write a test for a known case. The trouble is long tasks. Context gets polluted. Subagents drift from the original brief. And as a task stretches across hours and dozens of turns, the only signal most setups give you is the agent's own summary of what it did.

That's the failure mode this project exists to close. A build system doesn't trust a compiler's claim that it produced correct code — it checks the exit status, runs the test suite, typechecks the output. We wanted the same shape for agents: you don't trust the output, you verify it through gates.

The idea: treat the agent like a compiler

The core claim, stated plainly: an AI agent working on a codebase should be treated like a compiler feeding into a build. You don't ask it to be more honest. You don't read its diary. You make the mechanism refuse to advance unless objective evidence says the phase is done.

That reframing changes a lot of defaults. "Done" stops meaning "the agent finished talking" and starts meaning "the gate command exited 0 and the evidence file is non-empty."

How it works

Phases and gates

A task moves through a fixed state machine: P0 brief → P1 requirements → P2 design → P3 test-first → P4 implementation → P5 verification → P6 acceptance → P7 consistency → P8 release → READY → DONE.

flowchart LR
    P0[P0 brief] --> P1[P1 requirements]
    P1 --> P2[P2 design]
    P2 --> P3[P3 test-first]
    P3 --> P4[P4 implement]
    P4 --> P5[P5 verify]
    P5 --> P6[P6 accept]
    P6 --> P7[P7 consistency]
    P7 --> P8[P8 release]
    P8 --> D[DONE]

Between every pair sits a gate, and the gate's job is to run evidence the agent didn't write about itself. For a verification phase, that's the real test command — the gate script executes it and looks at the exit code. For requirements, it's a structural check: does the document contain at least one BDD (behavior-driven-development) acceptance criterion, are there unresolved NEED_CONFIRM items? For acceptance, it checks that the evidence files are non-empty and the gate commands all exited 0.

flowchart LR
    A[Phase work] --> G{Gate: run evidence}
    G -->|pass: exit 0, evidence holds| N[Persist to Markdown, advance]
    G -->|fail| R[Redo the phase]
    R --> A

If a gate fails, the phase is redone and the retry is recorded. This is where a recent postmortem — our AI safety net depended on the agent being honest. It wasn't. — gets relevant: we found the retry counter itself could be left empty by an agent, silently disabling the whole safety mechanism. The fix anchored that check to git history instead of the agent's own bookkeeping. The honest takeaway: the gates are only as strong as the evidence they're anchored to, and we're always looking for places where the "evidence" is really another self-report in disguise.

State you can see

Every phase result is written to version-controlled Markdown (active-tasks.md, .state.yaml). This is a deliberate choice, and it buys two things. First, a crash — a killed session, a power loss, a model context that hit its ceiling — is a pause, not a restart: the next run reads the state file and picks up where the machine left off. Second, a human (or another agent) can audit what happened by reading the files, not by trusting the summary.

Roles are separated

The orchestrator agent never writes code. Each phase is dispatched to a dedicated subagent — a requirements analyst, an architect, a test designer, an implementer, a verifier — and their output is handed back through the gate. This keeps the orchestrator's context clean (it's a dispatcher, not a participant) and makes the review genuinely independent of the work it's reviewing. Same reason you don't let the author be the only reviewer.

Does it actually work? We make it eat its own food.

Agateon is built with Agateon. The repository's own task history — dozens of tasks, from the original bootstrap to recent mechanism fixes — was produced through this same state machine, and it's all in the repo for anyone to read: agate-workspace/tasks/. The postmortem linked above is a real example of the loop working: an audit found a design hole, the fix went through phases with gates, adversarial tests were run against a real git repository, and the task record shows all of it.

We also try to break our own gates. The verification phases run adversarial tests — rollbacks, missing records, half-finished evidence — and check that the gate blocks what it's supposed to block and passes what it's supposed to pass. When a gate can be fooled by something an agent can simply omit, that's a bug, and it goes back into the machine.

What it is not (read this before trying it)

It's not a magic agent. It's a protocol and a set of scripts. You bring the coding agent; Agateon shapes how it's orchestrated and checked.
It's not a runtime or a service. There's nothing to deploy. Your agent just needs to read Markdown and run commands. Setup is a symlink plus a couple of git hooks: curl -sSL https://raw.githubusercontent.com/randomgitsrc/agateon/main/install.sh | bash.
It doesn't make gates immune to bad design. A gate that checks "did the agent write a plausible-looking report" is theater; a gate that runs a real test suite is not. The difference is entirely in what you choose as evidence, and choosing well is a design problem, not a tooling problem.
It's early and honest about it. The phase machine, the gate scripts, and the docs are all live in an MIT-licensed repository at v0.64.0, but "early" isn't a euphemism for "stable": the right expectation is that you audit the gates you depend on.

If you're building agent tooling, check this

The lesson generalizes beyond this project: whenever an agent's own report feeds a safety or progress mechanism, ask whether that mechanism depends on evidence the agent could simply omit. Ours did — the retry counter that triggered a human pause could be left empty, silently. The fix wasn't "make the agent more careful." It was "stop needing the agent to be careful about the parts that matter most," and anchor the check to something that exists independent of what the agent chooses to report.

Try it

The repository is github.com/randomgitsrc/agateon (MIT). One-line install, no infrastructure:

curl -sSL https://raw.githubusercontent.com/randomgitsrc/agateon/main/install.sh | bash

If you've ever shipped a long agent task on "it looks done" and felt uneasy about it, this is the project for you. The honest summary of what Agateon does: it makes "done" mean something you can re-run, not something you were told.

Our AI Safety Net Depended on the Agent Being Honest. It Wasn't.

Agateon — Wed, 26 Aug 2026 06:51:19 +0000

Agateon is a protocol for running software-engineering work through AI agents without trusting their word for it. Every phase of work — requirements, design, implementation, testing, release — has to pass an objective gate before it counts as done. No gate, no progress. That's the entire premise.

A few days ago, a routine audit found a hole in one of our own gates. Not a logic bug. A design hole — the kind where the mechanism does exactly what it was told to do, and that's the problem. Here's what happened, how we found it, and how we fixed it.

The mechanism: fail enough times, and a human has to look

Agents get stuck. They misread a requirement, a test fails for a reason they don't understand, a sub-agent comes back with nothing useful. Agateon's answer is simple: track retries per phase, and once a phase has failed too many times, stop automation entirely and force a human decision.

stateDiagram-v2
    [*] --> Phase: agent starts work
    Phase --> GateCheck: submit for gate check
    GateCheck --> NextPhase: pass
    GateCheck --> RetryOrPause: fail
    RetryOrPause --> Phase: retries below limit, try again
    RetryOrPause --> PAUSED: retries exhausted
    PAUSED --> [*]: human decides
    NextPhase --> [*]

Nothing exotic. It's the same idea as a circuit breaker: enough consecutive failures, and the system stops trying to route around the problem itself and hands control back to a person.

The part that matters is how the system knows a retry happened: the agent records it. Every time a phase gets rejected and redone, the retry is supposed to be written into that task's state file.

What the audit found

An independent review of four recently completed tasks checked whether the retry records actually matched what had happened. They didn't.

flowchart LR
    subgraph Real["What actually happened, per git history"]
        A1["Review rejected a design,<br/>sent back for rework"]
        A2["Verification failed,<br/>task rolled back a phase"]
        A3["A sub-agent returned<br/>nothing useful, redispatched"]
    end
    subgraph Recorded["What the state file recorded"]
        B1["retries: (empty)"]
    end
    A1 -.-> B1
    A2 -.-> B1
    A3 -.-> B1

All four tasks — real rejections, a real phase rollback, a real empty-handed sub-agent — and the retry counter for every one of them read empty. As if none of it had happened.

The retry-limit mechanism was never triggered by any of this, because the mechanism has no way to see the world except through what gets written into that field. If the field says nothing happened, as far as the safety net is concerned, nothing happened.

Why this isn't just a missed edge case

The uncomfortable part isn't that some retries went unlogged. It's what kind of gap this is.

Agateon's whole reason for existing is that you shouldn't trust an AI agent's account of its own work — you verify it against evidence instead. The retry-limit mechanism is supposed to be one of the things doing that verifying. But its own trigger condition depended entirely on the same untrusted party self-reporting honestly. The guard was watching for the fox, using information the fox was free to leave out.

That's not a bug you find by writing more test cases for the happy path. It's a hole in the design's trust model — the mechanism could be silently no-op'd, not through malice, just through an agent that got busy, forgot, or never wired up the bookkeeping. And it fails silently: no error, no crash, just a safety net that was never actually there.

sequenceDiagram
    participant Agent
    participant StateFile as State File
    participant Gate as Retry-Limit Gate

    Note over Agent,Gate: What was supposed to happen
    Agent->>StateFile: record retry
    StateFile->>Gate: retries for phase = 3
    Gate->>Agent: PAUSED, human needed

    Note over Agent,Gate: What could actually happen
    Agent--xStateFile: retry never recorded
    StateFile->>Gate: retries for phase = empty
    Gate->>Agent: continue as normal

The fix: stop trusting the field, check the evidence instead

The fix doesn't ask the agent to be more careful about logging. It stops relying on the log at all for the part that matters most, and checks something the agent can't quietly leave out: the git history.

A real phase rollback — say, verification failing and the task moving backward a phase — is a fact that's already sitting in version control the moment it happens, independent of anything the agent writes anywhere. So the fix compares the two: if the commit history shows a real rollback but the retry counter for that phase didn't grow, the commit is blocked.

flowchart TD
    A[Commit staged] --> B{Did phase move backward?}
    B -- No --> Z[Allow]
    B -- Yes --> C{Did retries counter<br/>for that phase increase?}
    C -- Yes --> Z
    C -- No --> D["Block commit<br/>retry happened, wasn't recorded"]

We deliberately kept this narrow. There are softer signals too — things like a review-rejection file existing on disk, or a sub-agent's output containing the phrases that usually mean "came back empty." Those are useful hints, but they're pattern-matching, and pattern-matching has false positives. So those only produce a warning, not a block. The one check that hard-blocks a commit is built on a fact that can't be fabricated by omission: a phase number that actually moved backward in git history. Confidence dictates enforcement strength — we don't block on evidence we don't fully trust.

Did it actually work? We tried to break it.

We didn't take the fix's word for it either. Two adversarial tests, both against a real git repository:

Attack: stage a phase rollback with an empty retry record — exactly the scenario the audit found.
Result: commit blocked, with a message naming the exact phase and the exact missing record.

Control: the same rollback, this time with the retry properly recorded.
Result: commit allowed.

One more thing worth mentioning, because it's the kind of detail that's easy to leave out of a postmortem and shouldn't be: an earlier version of this fix had a guard condition that, on inspection, would have made it unable to catch the exact failure mode that motivated it in the first place — the original evidence was tasks where a retry had never once been recorded, and the guard assumed there'd already been a prior record to compare against. It was caught in review before it shipped, not after. We're mentioning it because "the fix for the self-reporting problem needed a second pair of eyes to catch a flaw in its own logic" is a good demonstration of why we don't rely on any single check, including this one, working perfectly on the first attempt.

What's still open

The heuristic checks — the ones that only warn instead of block — can still be evaded by an agent that's a bit more careful about not leaving the usual pattern-matching signals behind. We know this. We're not claiming the retry-tracking problem is now airtight; we're claiming the one path that was silently, structurally guaranteed to fail is no longer silent, and everything else stays exactly as uncertain as it honestly is.

The general shape of the problem

If you're building anything where an AI agent's own report is part of what triggers a safety mechanism, this is worth checking for directly: does the trigger condition depend on evidence the agent could simply omit? If yes, that's not a hypothetical risk. Ours sat there, unnoticed, across four completed tasks, doing nothing, until an audit went looking for it on purpose.

The fix isn't "make the agent more careful." It's "stop needing the agent to be careful about the parts that matter most" — anchor the check to something that exists independent of what the agent chooses to report.

That's the whole idea behind Agateon: github.com/randomgitsrc/agateon. The fix described here lives at agate/scripts/check-state-transition.py, and the task that shipped it is TAG0023-mechanism-checks — full history included, nothing trimmed for the writeup.