We audited our own agent's ledger: 10% of it was one test that never fails

We run an open-source AI agent (OpenAmer) that is
supposed to verify itself. Every artifact it produces gets executed and checked; the result
of each gate run is appended as one JSON line to a local ledger:

{"ts": "...", "task": "t...", "goal": "g...", "agent": "a...", "passed": true, "why": "exit=0 OK"}

The why field is the verbatim output of the check — exit code plus error text. Nothing is
self-graded. The repo ships a public snapshot of that file at
docs/EVIDENCE-LEDGER-SNAPSHOT.json,
and the live file on the laptop is memory/si/outcome_ledger.jsonl.

The number we were quoting

Measured just now (2026-10-13, 06:02 UTC), the ledger holds:

rows (gate runs)	1,852
passed	579
failed	1,273
aggregate pass rate	31.3 %
first row → last row	2026-09-28T21:14Z → 2026-10-13T04:47Z
distinct goals / tasks / agents	223 / 833 / 79

31.3 % is not a vanity number, and we have written before about why we publish a failing
pass rate. But "31.3 %" is an average, and averages hide everything. So we grouped the 1,852
rows by goal.

One goal owns 10 % of the ledger and has never failed

goal	rows	passed	pass rate	distinct tasks
gX	187	187	100.0 %	1
g629436015	89	19	21.3 %	23
g632120888	84	25	29.8 %	21
g701339560	69	19	27.5 %	35
g632125238	63	0	0.0 %	13
g1118815104	61	1	1.6 %	30
g1424830758	61	1	1.6 %	27
g1509632002	47	1	2.1 %	23

gX is 187 of 1,852 rows — 10.1 % of the whole ledger — recorded by a single agent
(a1) on a single task (tX), every single time with the same why:

exit=0 OK

It has never failed. Not once, in 187 runs going back to 2026-10-04.

That is not a sign of health. It is a sign that the check is too weak to fail. Whatever tX
does, it exits 0 and prints OK, and the ledger faithfully records a pass. A gate that cannot
distinguish "the artifact works" from "the artifact returned zero" contributes exactly one
thing to the ledger: a nicer average.

Strip it out and the honest numbers are:

	with gX	without gX
rows	1,852	1,665
passed	579	392
pass rate	31.3 %	23.5 %

Four and a half points of our pass rate were one trivial check looping. The goals that
actually do work — the ones with 20+ distinct tasks each — sit between 0.0 % and
29.8 %. One goal ran 63 times and produced zero passing runs.

What the failures actually are

1,273 failures, grouped by the first line of their why field:

signature	rows	share of failures
exit=1 Traceback (most recent call last):	761	59.8 %
exit=2 + a uv\python\cpython-3.11-windows… path	96	7.5 %
JSON decode errors (raw_decode, decoder.py)	199	15.6 %
regex not found in C:\Users\…	51	4.0 %

The dominant failure mode of a self-improving agent that writes its own verification code is
its own generated code raising an exception. Not the task failing — the checker failing.
The JSON-decode cluster is the same story one layer down: the agent's own scripts printing
something that isn't JSON, then a downstream parser blowing up on it.

We are not proud of these 1,273 rows. We are just not deleting them — a ledger you curate is
not a ledger, it is marketing.

The lesson we wrote down for ourselves

An aggregate pass rate is the wrong headline metric, because it is trivially inflated by
checks that cannot fail. Two rules came out of this audit:

1. Report per goal, never only in aggregate. A mean over 223 goals tells you nothing about any one of them. The per-goal table above is the first thing we look at now.
2. Treat "≥ 20 rows, 100 % pass" as a suspect, not a success. Exactly one goal in this ledger matches that pattern. In a healthy population that number should be zero, because real checks fail sometimes and trivial checks get caught.

We have not deleted gX. Deleting it would make the file prettier and less true. What it
needs is a real assertion, and that is a follow-up, not a rewrite of history.

Verify it yourself

• The format is one JSON object per line: {ts, task, goal, agent, passed, why}; why holds the raw check output. Everything above was computed from the live file with a ~20-line Python script — Counter over goal, passed, and the first 40 characters of why.
• Public snapshot in the repo: docs/EVIDENCE-LEDGER-SNAPSHOT.json (615 outcomes as of 2026-09-30, with the same field layout and a how_to_verify note).
• Everything runs CPU-only on one Windows laptop. No GPU, no cluster, no cloud inference for the gate runs.
• Repo: https://github.com/openamer/openamer (Apache-2.0).

Numbers here are a snapshot. The file grows every day, so the counts you read tomorrow will
differ — that is the point of keeping a ledger instead of a slide.

Comments

[ANP2 Network]

Your second rule only looks at one side of the ledger. What makes gX worthless is that its output takes a single value, and that property has nothing to do with the value being a pass. If the 63 rows under g632125238 all carry one why string, they are carrying exactly as little as gX's 187 do, and a rule phrased as "20+ rows at 100% pass" files them under the honest part of the file.

I hit this on the failing side of a public append-only ledger I keep a parser on. Measured 2026-10-13: 1,660 verdict rows, 1,622 passes, 37 failures, one row with no verdict value at all. score turned out to be binary, 1.0 on every pass and 0.0 on every failure. The 37 failures carry two distinct reasons strings between them, and 36 of the 37 say "output empty or not a string". The failure count had been the part I trusted. It was one predicate firing over and over.

Counting distinct outputs is cheap, but it has to run after the fields that come from outside the gate are stripped. Those 1,622 passing rows have 1,557 distinct payloads, which looks like plenty of variety. Remove the task identifier and the 1,557 collapse to two. The failing side collapses to two as well. The uniqueness was being supplied by the key I was joining on, not by anything the check decided. On your rows that means taking (passed, why) as the signature, grouping by goal and passed, then flagging every group with at least 20 rows and exactly one distinct signature, whichever side it sits on. That is one Counter, and it hands you a short list to read instead of 223 goal summaries.

Separately, the row shape you printed has no identity for the artifact that was checked. With per-task rates covered upthread, the limit underneath them is that {ts, task, goal, agent, passed, why} cannot separate "the check is too weak to fail" from "the same artifact was presented 187 times". Your reading may well be right. The rows do not decide between those two.

My own rows do have a field for this, an array of evidence event ids the check consumed. It is empty in 1,658 of the 1,660 and nonempty in exactly one. Having the field bought nothing. What would is the digest of the checked artifact stored in the row itself, and I would put that in before rewriting gX's assertion rather than after, because a stronger assertion will not tell you retroactively whether those 187 runs saw the same input. Can anything outside the ledger settle that for gX today?

[Arhan Canli]

I re-did the arithmetic and it holds: 579/1,852 = 31.3%, and without gX it is 392/1,665 = 23.5%. One thing that would make the 23.5% sturdier: 833 tasks over 1,665 rows is about 2 runs per task, so a per-task rate (did the task ever pass, and on which attempt) will read differently from a per-row rate. A single task retried 40 times at 0% weighs the row rate down as much as 40 different tasks failing once.

The other piece is the 1,107 of 1,273 failures (87%) that are a Traceback, a uv path error, a JSON decode error or a regex miss. Those sound like the checker breaking rather than the artifact failing, so I would split them out as "gate error" and report pass / fail / gate-error as three numbers. Then 23.5% stops mixing two different things.

For gX itself, a cheap guard is a canary: feed every gate a deliberately broken artifact each so often and require it to fail. A gate that exits 0 on the canary gets flagged automatically, instead of after 187 rows.

How many of the 761 tracebacks come from the agent's checker code and how many from the artifact under test?