I spent a day building a tool whose entire premise is that you should not trust a system's own report of itself. Over that same day I believed my own tooling seven times when it was reporting nothing, and corrected my own claims eight times. Here are the four that were worth the price.
The check that compared two files that were not there
A gate compared a generated artifact against its committed copy and passed when the bytes matched. Both paths had moved. It read two missing files, got two empty strings, found them equal, and reported green. It had been green for a while.
Byte-equality on absent inputs is vacuously true. The fix is not a better comparison, it is refusing to compare until both sides exist. A checker that cannot distinguish "identical" from "neither one is here" is not a checker, and it will pick the flattering reading every time.
The zero that came from my own projection
I asked the GitHub API for failed workflow runs in a window, piped it through a --jq expression that reduced the response to a length, and got 0. I read that as no failures.
There were 2,450. The per_page cap had truncated the response before my filter ever saw it, and the field that would have told me — total_count, sitting right next to workflow_runs in the same payload — was thrown away by my own projection.
Someone made this sharper than I had it, in a comment on the last piece I wrote about this: never project away the field that tells you what you were given. If a count arrives with no denominator beside it, that is the unable-to-tell case regardless of what the count says. That is a narrower and more checkable rule than the one I started with, and it is mechanisable — you can lint for a reducer that collapses to a scalar without carrying its bound.
The exit code that belonged to tail
npm ci ... | tail returned 0. I read the install as successful and moved on. The 0 was tail's. The install had died on a native module and node_modules was missing the thing I was about to run.
Same shape, third instance that day: a shell pipeline reports the exit status of the last stage, and the last stage is almost always something that cannot fail. I now know two spellings of this bug and I still walked into the third.
The gate that asked a different question than the one that mattered
A monthly maintenance task runs wsl --shutdown, which kills every process in the guest. Its safety gate counted running cargo processes and skipped the window if any were found. Sensible: don't kill a build.
Measured on the day: cargo was 0. Two long-running servers belonging to a colleague's session were live. The gate would have passed and the task would have killed them mid-session.
The gate was not miscounting. It was asking "is a build running" when the operation it guards forces the question "is anything running that must not be killed". Those are different predicates and only one of them is safe. I widened it, and the widening surfaced a second bug: pgrep -c cargo matches process names, and one of the things I needed to catch is a bash script, which has no matching process name. pgrep -f and the full command line. The original gate could never have found it either.
What I actually changed
Not resolve-to-do-better. Four things that execute:
Count with two spellings before reporting a number. Four of my eight corrections that day were single-spelling scans. A test count that missed every #[tokio::test(...)] and undercounted one crate fourfold. A call-site scan whose single-line regex skipped every line the formatter had wrapped, 37 read as 30. A dependency check where .split("[dependencies]") grabbed a comment containing that literal string. A line count where the tool silently dropped 509 blank lines. Every one of them exited 0.
A negative control beside every gate, and a rule for reading a green one. A deliberate break that comes back green has two causes, not one: the checker is asleep, or the break did not land in anything the checker reads. I hit the second and spent an hour on the first diagnosis. Name the assertion, check whether what you mutated is inside what that assertion reads, before you conclude the test is weak.
Stop treating "running" as "progressing". Nine background agents ran between six and fifteen hours with zero reports. I read the running state as work happening. Four of them had produced no artifact at all when I finally checked the filesystem instead of the process list.
Make the discipline refuse rather than remind. I wrote "record the ruling in the same turn you make it" into my own template, and then broke it three times the same day, twice within minutes of writing it. What fixed it was a pre-tool hook that denies every write and shell call once the ledger is more than a few turns stale. It has stopped me four times, and each time I had believed I was already compliant. A prohibition survives being forgotten because not doing something is the default. A prescription does not: forget it and you silently revert to the behaviour that caused the incident.
The one I did not find
Two defects in a plugin example I had published were reported by a reader who read the code. Neither was found by us.
The first: on approval the hook returned the parameters unchanged and let the host's own write run afterward. The re-application was unconditional, so anything landing between our commit and that write was overwritten with no record. A single caller was enough; no concurrency required.
That code had a comment on it. The comment said the redundant write was a real property of the design and not an omission. It was right that the write was intentional and wrong about everything that mattered, and it had been sitting there telling every reader — including me, several times — that this part had been considered.
A comment that says "this is fine" is the cheapest possible way to stop anyone from checking whether it is.
Where this actually stands
Alpha. 2,926 of 2,988 probes pass, 59 fail, 44 of those need database or probe services the checkout does not start. Four adapters public, three on crates.io. Zero external users, zero revenue. Continuous integration has not started a job since 20 August because the account's billing is blocked, which means every green check on a recent commit is a check that never ran — and I only found that out by opening a job and seeing an empty steps array.
The tool is about not trusting self-report. The day was an extended demonstration of why that is hard, performed by the person building it.
TraceFold, Rust, Apache-2.0. The limits page is longer than the feature list.
Top comments (0)