Your eval harness has a column of green. Some of those greens are measurements. Others come from a script that ended on echo and returned 0 whatever it printed. The dashboard can't tell them apart, and for seven weeks mine couldn't either.
On August 26 our memory smoke test printed 0/10 queries with >=1 memory, avg latency 17ms. The nightly board recorded PASS. The next night was a clean run that scored 89% overall. The same test printed 4/10 and passed again. The seven nights before had all read 10/10. The script's last line was an echo, so its exit code was always 0, and the runner graded exit codes.
Queries that found a memory (all recorded PASS)
Four verdicts, and CONTAMINATED is not the same as fail
Between August 27 and 30 I rebuilt our nightly QA around one rule: a grader reports what it saw, and that includes seeing nothing. Seven commits, ten files. Every step now ends one of three ways: ok, fail, or unknown. Unknown exits 2 and means there was no evidence, for example because a dependency was unreachable or the turn being graded hadn't finished. On top of that, the whole run can be stamped CONTAMINATED when the system under test changed while it was being measured.
scripts/qa/qa.sh takes a fingerprint at the start and the end of a run: the daemon's process id, the engine binary's mtime, and git HEAD. If any of them moved, the scorecard JSON, its markdown and the history row all say CONTAMINATED and name what moved. Then I went one level up and graded the plan documents that describe the instruments, with scripts/plan-status-guard.py.
What a nightly run is allowed to conclude
61% on August 26, and four of the seven reds were one daemon restart
That same nightly scored 61% with seven reds. The retrieval bench said Connection refused. The inject bench measured 0.29 recall. Runtime status said the daemon was not reachable, and a console end-to-end test failed its warm check. The scorecard filed these as four product defects, and a session spent the next morning chasing them.
They were one event. Someone rebuilt the engine binary at 22:58. The run started at 23:10, and the daemon restarted at 23:19, in the middle of it. Nothing in the harness knew the thing it was measuring had been swapped out mid-sweep. I verified the fingerprint by touching the binary during a fast run, and the flag showed up in all three places. It has fired for real since then: on September 10 the daemon pid, the binary mtime (+545 seconds) and HEAD all changed during one run. Static checks like extension unit tests and skill validation stay actionable in a contaminated run. Only the tiers that need live services need a clean rerun.
A healthy daemon answers ping with ok: false
With the contamination explained away, three findings were left. Each was a grader that couldn't report what it had seen.
The smoke test was the first. It now exits 1 below an 80% floor and exits 2 when the daemon is unreachable. It doesn't run its own reachability probe. It asks the step that already owns that answer, because a second probe could disagree with the row above it. I provoked all three verdicts and watched each one happen.
The second was library-e2e.test.ts. Its skip gate asked whether the gateway was up, but every assertion in it reads the daemon's index. So a daemon outage came out as 2 failed / 1055 passed with no filename, three rows below a status check that already said the daemon was down. I changed the gate to ask about its real dependency, and that turned up a trap. I probed a healthy daemon, and it answered ping with {ok:false, kind:'broken', reason:"unknown command: ping"}. A gate on ok would have skipped the suite forever, and a skipped suite looks exactly like a passing one. The gate reads kind now.
The third was account-gate.test.ts. It pinned a default value while the module under test loaded dotenv on import, and this machine's .env sets that variable. The test had been red locally for weeks and green in QA, so nobody trusted it. After the fix the console suite ran 1153/1153.
21 false alarms in one day from the receipt grader
The receipt grader compares what a turn's receipt says it used against what the log shows it used. It reported "6 of 21 receipts show FEWER lanes than the log holds". All six were false, and it raised 21 of these in a single day. A turn the gateway declares silent sends no receipt at all. But a row still gets inserted for every turn, and the daemon adds its own lanes to it. The grader treated that row as a short receipt, when there was no receipt to be short. A row that carries only daemon-written lanes is now skipped, and the six went down to two.
Then I ran the full tier and the grader manufactured a failure of its own. A receipt row is created when a turn starts and filled in when it ends. I sampled one receipt twice, minutes apart: NULL lanes the first time, eight the second. I had measured a turn while it was still running. Unfinished turns are unknown now. Of the eight failures left after that fix, all eight are real: two whole surfaces under-report, with 8 to 10 lanes in the log and one on the receipt. That's filed as its own defect.
Two more steps from the same week had the same shape. Our build-inventory command exited 0 unconditionally, so the drift step printed its findings and recorded a pass. In July a daemon serving a stale debug build cost us 38 hours of timeouts, and this step would have printed that and passed too. It now exits red on a build disagreement or a provably dead process, and warns on untracked ones. The other step was the runner's own step count. I added two steps without bumping the expected count (13 run against 12 expected). That turned the guard against skipped steps into noise.
The status guard flagged about 20 rows on its first run, and every one was noise
A fresh-eyes pass checked about 320 claims across our plan documents. The premises held up, but the status markers didn't. P4 was marked COMPLETE with three of its five numbered changes unbuilt. One step cited a commit as having fixed a config file, and that commit never touched the file. Four overclaims had been written the same day by the session that built the work. Every one was an intended end state written down as a delivered one.
The guard checks three rules. Every cited SHA resolves. Every backticked repo path exists. Every row marked done cites evidence, or says why it has none. On its first run it flagged about 20 rows, and all of them were noise: matrix cells full of checkmarks, a content_hash and an md5 read as SHAs, and a runbook whose evidence is an observation rather than a commit. Then it missed the exact case it was built for. ✅ **COMPLETE** is two done markers in one cell, and my "this is a matrix" heuristic counted markers instead of cells, so it skipped that row. Once that was fixed, the clean tree passed and 43 cited commits resolved.
Every grader needs an observed red, and every red needs an unchanged system
The failure class, stated as something you can go and check: for every step your runner grades, there is a recorded execution where it exited non-zero, and the condition that made it fail is written down. Next to it sits a second property: a verdict counts as a product reading only if the system's identity (process, binary, commit) was the same at the start and end of the run. If you have never seen a check fail, you don't know it can.
Provoke each check, fingerprint the run, resolve every SHA in your docs
First, find the checks that have never gone red. Any CI results table works:
SELECT step_name,
COUNT(*) AS runs,
SUM(CASE WHEN exit_code <> 0 THEN 1 ELSE 0 END) AS reds
FROM ci_step_results
WHERE started_at > datetime('now', '-60 days')
GROUP BY step_name
HAVING reds = 0
ORDER BY runs DESC;
If a step shows 60 runs and 0 reds, go and provoke it. Take away its dependency and read the exit code:
docker stop my-vector-db
./checks/retrieval-smoke.sh; echo "exit=$?"
docker start my-vector-db
exit=0 fails this check. A passing check exits non-zero, ideally with a distinct code for "unreachable" so an outage doesn't get blamed on retrieval. Then fingerprint the run itself:
fp() { printf '%s %s %s' "$(pgrep -o -f my-daemon)" \
"$(stat -c %Y ./bin/my-daemon 2>/dev/null || stat -f %m ./bin/my-daemon)" \
"$(git rev-parse --short HEAD)"; }
START=$(fp); ./run-evals.sh; END=$(fp)
[ "$START" = "$END" ] || echo "CONTAMINATED: [$START] -> [$END]"
Last, check that the commits your status docs cite actually exist:
grep -rhoE '`[0-9a-f]{7,40}`' docs/ | tr -d '`' | sort -u |
while read -r sha; do git cat-file -e "${sha}^{commit}" 2>/dev/null || echo "unresolved: $sha"; done
No output means every citation resolves. Expect a few hashes that aren't commits on the first run, and exempt them by what the line says they are.
Agent-verification writing checks the agent, not the checker
The current advice is good about agent output. self.md says agents "declare success the same way they declare failure: confidently," and gives a seven-check loop. GBQA scores QA agents with a critic against human-annotated ground truth. neural pruning recommends a monthly audit for stale context files, which is the closest anyone gets to grading the status layer. QAgent requires every result to carry "a source, observation time, and availability state", and I agree with that completely. And this piece on agentic evaluation makes the key point that "the environment itself is now part of the test object."
That last piece is about a model gaming a checker. What I hit was simpler: the environment changed under the checker and nobody gamed anything. None of these sources ask whether the verifier has ever been seen failing, or whether the document that says "done" can be audited. A critic compared against ground truth is only as good as the critic's exit code.
Still open: prose gates can't be checked, and a miscounted run still gets a percentage
The status guard checks whether a claim is auditable and whether its evidence has rotted. It can't check whether a gate written in prose, like "the seam is one file," was actually met. A step-count mismatch now fails the run, but the runner still prints a percentage for it. On August 30 that meant 85% for a run whose shape was unknown. Changing what the score means is a separate decision, and I haven't made it yet.
Source: A QA step that cannot go red is worse than no step at all by Chad Priest, from Building Vodou in Public.
Top comments (0)