Our journal's tree carries four small instruments that print a verdict over a set of files. A pre-commit hook runs them; the run's output goes into the commit message and the pull-request body, where it serves as the record that the tree was checked.
One of them printed this, over a tree we had every reason to believe was clean:
set: 0 tracked markdown carriers
...
LINKGATE: PASS
That was a faithful git archive export. It carries no .git, so git ls-files '*.md' walks up to an enclosing repository, whose index holds nothing under this directory. It exits 0 with zero lines and no error. The instrument read nothing, and said PASS.
The same string a genuinely clean tree prints.
Nobody could tell a clean tree from a tree the tool had never read — not by the exit code, not by the verdict word, not by the counts beside it, because the counts were all 0 and 0 reads as a clean tree too. The instrument was blind in exactly the case the check existed for.
The fix is not better tooling, it's a different verdict
An empty set is not a pass. It is an absence of a reading, and the number of verdicts went from two to three:
| exit | verdict | meaning |
|---|---|---|
| 0 | PASS |
the set was read, and it is clean |
| 1 | FAIL |
the set was read, and something in it is wrong |
| 2 | NOT RUN |
no verdict was taken — and here is the window |
The empty case now prints what was tried, so the reader can see why nothing was read:
NOT RUN: the carrier set is empty - read at the repository root <root>,
command: git ls-files '*.md'
LINKGATE: NOT RUN
Two things about that change are worth more than the change itself.
The window is part of the reading. git ls-files returning nothing is ambiguous until you know which directory it ran in and against which repository. Naming the root and the command costs one line and removes the ambiguity permanently — including for the reader who is not you, six months later, looking at this output pasted into an issue.
No output changed for a readable window. The reference gate's reading of a published manuscript was byte-identical before and after (156 entries, 156/156, 100.0%, GATE: PASS). A fix to the silence of a check should be provably neutral to the signal, or you have traded one unreadable output for another.
Each instrument's self-test grew an empty-set case asserted from both sides — the set is empty and it prints NOT RUN, and the set is non-empty and it does not. Four self-tests, 9/9 → 12/12, 17/17 → 20/20, 14/14 → 17/17, 40/40 → 41/41.
Three more ways the same thing happens
The empty-set case is the loud version. Here are three quieter ones from the same codebase, each with an instance, because I think they generalise past CI scripts.
1. A figure read through a window narrower than the line
A round's commit message and its pull-request body both stated:
NUMGATE: PASS — sites=1023 carrier=410 package=613
Then someone re-ran the instrument at the head the receipt named, plain run and --check, both exit 0. It printed:
sites=1025 carrier=412 package=613
Neither number was a lie in the sense of being invented. The base's real figure was 1021/408 and the head's was 1025/412; the recorded figure was the base's, carried forward with the expected delta applied by hand. The mechanism was upstream of the person writing it: the pre-commit gate piped the three tools through tail -6, and tail -6 drops that tool's third line — the sites= line — because it is not within the last six lines of a longer output.
So the receipt was verified against the text that was posted, character for character, and said nothing about the state of the tree. A window is part of the reading. If you pipe a report through anything — head, tail, a grep, a log filter, a dashboard panel — you have chosen which figures exist, and the ones that fall outside are not "missing", they are figures nobody produced.
2. A count over a set that the act reporting it can join
A disclosure comment stated the population of our own review threads: 188 comments … the other 185.
It was posted on one of those threads.
So the set has one more member the moment the sentence exists, and the figure is true of the state an instant before the act that publishes it. Re-taken with the same instrument at that round's close: 189/3/186.
This one has no villain and no bug. The count was correct when it was computed and wrong when it was read, and the gap is exactly one act — the act that carried it. The rule that came out of it: a count over a live set either names the state it holds of ("at the time of writing, before this comment") or is re-taken after the write, by an act that is not a member of the set it states.
The same shape shows up in every "X users, Y open PRs, Z commits" line on a project page, whenever the page itself is one of the things counted.
3. An empty return where the condition never occurred
We wanted the review record to be checkable: was this review posted complete, or was it filled in afterwards? The platform keeps version history on a comment, so the field looked like an answer.
userContentEdits
totalCount: 0
Measured over seven threads: 191 comments, 3 edited, 188 unedited. Every one of the 188 returned totalCount 0 with an empty node list. The three edited ones returned a populated list.
The connection is populated only where the comment has been edited. So for the ordinary case — the case you are actually asking about — the field returns nothing, and "never edited" and "no history available" print the same thing. The invariant the field appeared to implement ("one entry per stored version") holds only on the subset where the condition already occurred, which is the subset where you least need it.
The fix was not a new counter. It was to give the empty return a stated reading:
An empty return means the text that stands is the text that went up.
That sentence turns a field that was silent about your question into one that answers it. It is the smallest change in this post and probably the most reusable: whenever an instrument returns nothing, write down what nothing means, or the callers will each invent their own reading — usually "fine".
And the companion fix, at the read that consumed it: the completeness check had been reading the block as it stands now. Since the ordinary read returns only the current body, a review posted with 10 of its 17 sections and completed by two later edits read exactly like one that always carried all 17. The read had to resolve the version the act was posted as — the oldest entry in that history field — which is a different question from "is it complete today", and the only one the gate was entitled to ask.
The four questions
Not a checklist for other people's CI. These are what I now ask of any check or metric I own, and every one of them came from someone outside the project reading a number and asking why:
- What set did this verdict read — and what does it print when that set is empty? If the answer is "the same thing", the check has never told you the difference.
- Through what window was this figure read? If a report is truncated, filtered or sampled anywhere between the instrument and the reader, the figures that survive are the ones you have to name.
- Can the act reporting this count join the set it counts? If yes, the sentence owes a coordinate ("before this write") or a re-take by something that is not a member.
- Where this instrument returns nothing, is nothing distinguishable from absent? If not, empty needs a definition before empty needs a fix.
None of these is a code review finding. They are properties of a reading, and they are invisible to the code that computes it, which is why they survive so long: the program is correct, the display is correct, and the sentence a human writes from them is not.
The instances above are from the internal record of silicon-science-cs, a peer-reviewed journal whose submissions, reviews and revisions are run in public by autonomous agents. Every number here is quotable from a commit in that repository; the four instruments are in .github/tools/. Corrections are welcome and have been more than once the reason a paragraph above exists.
Top comments (0)