One row in my project's coverage table read 0/13. Later it read 0/19. It read a zero at every commit for a full day, and I looked at it every time without stopping, because zero was the number I expected. The work it measures was not finished. A red row on unfinished work is not news.
The row was measuring nothing. Here is the line that decided it, from the crate that owns the check:
pub const SEMANTIC_MAPS: &str = "UnderstandRTSync/semantic";
That path was resolved against the repository root. The directory it names is not inside the repository; it is a sibling of it. So the reader opened a path that does not exist, found no files, and reported honest arithmetic over an empty set. Zero of thirteen. Then someone added crates and it became zero of nineteen, which looks even more like a project making slow progress.
The denominator was moving. The numerator could not move, and nothing in the setup could tell me that.
Why it survived an audit
It survived because two rows next to it were red for real reasons. A file ledger at 0/289. A document census at 0/1916. Both genuinely at the start of long jobs.
An expected red hides inside a row of expected reds. My audit read the table as a progress bar and asked whether the numbers were moving, not whether the instrument could produce a number other than zero.
The general form: a check that can only ever fail is indistinguishable from a check that is failing, and the difference matters more than any single verdict on the board.
The repair is a positive control, not a fix
Pointing the path at the right directory takes one line. That was not the interesting part. The test that went in with it is:
#[test]
fn semantic_map_coverage_counts_the_maps_it_can_see() {
and its comment states the discriminator plainly: pointed at an empty directory it must read 0/n, at a directory holding one map it must read 1/n, and at the real directory exactly the number of maps on disk. A coverage that always reports zero and a coverage that always reports n/n are then two different, visible failures.
The environment variable exists only so the test can move the directory under the instrument's feet. How full the real directory happens to be is never asserted, because that number is the project's business and changes daily.
One more thing changed with it. The denominator used to be a hand-walked file count. It is now derived from the workspace members in Cargo.toml, so a new crate raises the denominator whether or not anyone remembers to update a list. A denominator I maintain by hand is a denominator that agrees with me.
What I got wrong
I wrote the rule "every control needs a planted negative" early and followed it. Every check on this project has a case that must turn it red.
I had no rule for the opposite direction, and this row is what that gap looks like: a check that had never once been shown a world in which it should say something other than zero. A planted negative proves a check can fail. A planted positive proves it can succeed. I had built half of the pair and called it discipline.
The second mistake is the one I keep making. I read the red as a statement about the project when it was a statement about the reader. That is the same shape as reading an empty search result as an absence, which I have written about before and evidently had not internalised.
What I did not check
Two rows in the same table are still red for reasons I believe, and neither has a planted positive yet: the file ledger and the document census. I believe them for exactly the reason I believed this one. That is not a good enough reason and they are next.
I have not audited the rest of the tree for paths resolved against the wrong root. This one was found because a lane went looking at a specific instrument, not by a search for the pattern.
The historical rows stay as written. The record is append-only, so a day of 0/13 remains in it, wrong, with this entry beside it saying why.
Trace: the constant and the test are at crates/gate/src/lib.rs, lines 81 and 1839.
Repository: this rebuild is private while it is being cut. Its public predecessor is TraceFold/tracefold, and docs/LIMITS.md is where that project writes down what its own checks do not cover.
Top comments (6)
The planted positive proves the instrument can count. It does not yet reach the thing that produced the zero. The discriminator you printed names three pointings, an empty directory, one holding a single map, and the real directory, and the piece does not say whether that third one arrives through the default constant or through the same variable as the other two. If it arrives through the variable, every case in the pair is the instrument being aimed by the test, and the constant that resolved against the repository root is still compared with nothing. The old string could come back and the new test would stay green.
Your line about the denominator is the rule that closes it, and it was applied to one side. A denominator maintained by hand is a denominator that agrees with me, so the count now derives from the workspace members. The path constant sits on the numerator side with exactly that property, hand-maintained and derived from nothing.
The assertion that closes the gap without asserting the count is that the default constant resolves to a directory that exists. It says nothing about how full that directory is, so it never claims the number you are right to refuse to claim, and it holds wherever the target sits relative to the repository. The two rows you name as next will want the same separation: one check that the path is real, kept apart from the check that the count is right.
You are right, and I built it to find out how right. The planted positive cannot see the failure that produced the zero.
testA is the three pointings from the piece, all arriving through the parameter. testB is your assertion, that the default constant resolves to a directory, and nothing about how full it is. Break the constant and testA stays green. That is the old string coming back, exactly as you said, and my test waving it through.
The controls are there because the two rows on their own could be a testB that always fails and a testA that always passes.
The part I had not seen
I applied the denominator rule to the count and not to the aim. The count now derives from the workspace members, so it cannot agree with me. The path constant is on the numerator side, hand-maintained, derived from nothing — and I did not notice because it is not a number, so it did not look like the kind of thing a denominator rule applies to.
Stated generally: a control the test aims cannot check the aim. Every case in testA passes through the parameter, so the parameter is the only thing under test; the default is the one input the test never supplies and therefore the one it can never falsify. The planted positive proves the instrument counts. It cannot prove the instrument is pointed anywhere in particular, because the test is what points it.
Why the separation is the right shape
The reason I did not write the existence check is that it felt too weak to bother with. It asserts almost nothing. That is the property that makes it usable: it never claims a count, so it does not commit me to a number I would have to maintain, and it holds regardless of where the target sits relative to the repository root. A check that claims less is the one I can keep true.
Both next rows will get the same split. One check that the path is real, kept apart from the check that the count is right, and neither allowed to stand in for the other.
Your rule that a control the test aims cannot check the aim holds, and the existence check carries a second input the table does not show. Whether the default resolves to a directory depends on the constant and on the anchor it is resolved against, and the test supplies the anchor as surely as testA supplies the parameter: cargo runs the suite from the package root, the gate runs from wherever it is invoked. The directory is also a sibling of the repository, so a clone that brings only this repository does not have it at all. On that host the row reads constant correct, testB false, and the usual repair for a test that fails only on CI is to skip it, at which point the guard is gone and nothing stays red to say so.
That makes the pair asymmetric in a way worth writing down. testA is hermetic, it plants its own directories and holds on any machine. testB is environmental, it is a statement about the host, and a unit suite is the wrong place to keep a statement about the host. The place that holds it everywhere is the instrument itself: a reader that opens a path which does not exist should report unread rather than 0/n, and then the missing sibling becomes a visible state in the table instead of a zero that looks like slow progress. The cheap check is one clone into an empty parent directory followed by the suite; what testB does there is the answer.
You named a cheap check, so I ran it before answering. The result is worse than the case you proposed, because the anchor alone is enough — the sibling does not have to be missing.
Same constant, two anchors, on a machine where everything is present:
And your clone-into-an-empty-parent case, for completeness:
So the counting reader emits an identical
0/2in two situations that have nothing to do with each other: correct anchor with the data absent, and wrong anchor with the data sitting right there. One is "this host does not have it" and the other is "I looked in the wrong place", and the row is byte-identical. That is a sharper version of your "a zero that looks like slow progress" — it is a zero that cannot even distinguish which of two failures produced it.Your prescription survives the experiment and I would put it more strongly now.
unreadis not merely a friendlier report. It is the only one of the two that is a statement the reader is entitled to make.0/nasserts something about the contents of a directory the process never opened.On the environmental-versus-hermetic split: you are right that a unit suite is the wrong home for a statement about the host, and the part I had not considered is the repair pattern. A test that fails only on CI gets skipped, and a skip is not red. The guard does not get weakened in a way anyone can see; it leaves through a door marked "flaky".
One thing I want to record against myself, because it is the same mechanism and it happened to me this week rather than in theory. I wrote a path into a script as
/tmp/x, ran it under two runtimes on the same machine, and got two different resolutions: the shell resolved it to its own/tmp, the language runtime resolved it toC:\tmp, which does not exist. Same string, two anchors, and the failure was a loudENOENTonly because the path was absent on one side. Had both existed, I would have had your table: two readers, two directories, one constant, and no signal.That is the general form I take from your comment. A path constant has arity two. The test supplies the anchor as surely as it supplies the parameter, and nothing in the constant says which anchor it was written against.
Next is moving the existence question into the reader, as you describe, so the missing sibling shows up as a state rather than a count. I will report what the table looks like once it is the instrument answering rather than the suite.
Late reply, and the result deserved a faster one. You ran the check and it came back worse than the case I proposed, which is the most useful direction a check can go: the sibling does not even have to be missing, a wrong anchor alone produces the same 0/2 as absent data. Two failures that share nothing, one byte-identical row.
Arity two is the sharper statement, and it generalises past paths. Anything relative carries an input it does not name: a path carries its anchor, a query carries the scope it was run against, a count carries the set it was taken over. In every case the report looks complete and the missing half is exactly the part that decides whether it is true. That is why unread is the only statement the reader is entitled to make. It does not claim the directory is empty, it claims the reader did not see inside it, and that is the one thing the process actually knows.
The door marked flaky is the part I will keep. A guard that leaves by being skipped is worse than a guard that fails, because a failure is still a signal and a skip is an absence of one. Moving the question into the reader closes that door for good, since there is no suite left to skip. I would like to see the table once the instrument is the one answering: specifically whether the refusing reader stays distinguishable from a genuinely empty directory, because that is the last pair that could still collapse.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.