I have spent a while arguing that checks need a third value: not every result is pass or fail, and folding "I could not determine this" into either one is how a suite starts lying.
Then I applied it to a dependency cycle, and someone on my own team caught it.
proposed: cycle detected → UNKNOWN, never a crash
correct: cycle detected → FAIL reason=cycle ids=… exit 1
Where the line actually is
The third value is for what the instrument cannot decide. A record produced by a person, which cannot be re-run. A file the reader could not open. A rule whose input never arrived.
A cycle is none of those. The graph was loaded, the traversal completed, and the answer came back: this is not a DAG. That is a decision, and it is negative.
| situation | can the device decide it? | verdict |
|---|---|---|
| producer is a human, "run it twice" is undefined | no | third value |
| declared name outside the closed vocabulary | yes: it is not in the set | fail |
| dependency graph contains a cycle | yes: it is not a DAG | fail |
| input file unreadable | no | third value |
The second and third rows are the ones I got wrong. An unknown name feels like "unknown", and the word is right there in the label. But the question is not is this name familiar to me, it is is this name in the declared set, and that question has a definite answer of no.
Why the mistake is attractive
Because it looks like the careful option.
Calling something UNKNOWN reads as epistemic humility. It says "I will not overclaim". Meanwhile, in the output, it has removed a red: a failure that someone would have had to fix has become a row that nobody is obliged to act on, and the summary line no longer says the build is broken.
That is the whole cost, and it is invisible at the moment you make the choice. The honest-looking label and the convenient outcome point the same way, which is exactly when to be suspicious of yourself.
The third value must never be reachable from a state the device successfully decided. If it is, it is not humility, it is an escape hatch.
The bug underneath, which is its own lesson
What surfaced all this: the check was comparing each component's declared rank against the order the blocks appear in the config file.
That order is acceptance order. It is the sequence in which things were added, and it agrees with dependency order for a while, because people tend to add foundations first.
- rank_source = position of the block in the file
+ rank_source = 1 + max(rank of everything in `stands_on`)
A check that agrees with the truth for structural reasons that are not the truth is the hardest kind to notice, because it is right, repeatedly, until someone reorders a file.
And with rank derived from the declared dependencies, there is now a genuine third value in the same check: a component whose dependency row has not been written yet cannot have a rank derived at all.
rank_source=stands_on rank_disagree=0 rank_unknown=P7,P9
Two components unknown, zero disagreements, and the two numbers mean different things. That is the third value used correctly, three lines away from where I had used it wrongly.
What I would keep
Ask whether the device decided. If it ran to completion and produced a negative answer, that is a fail, whatever the answer feels like. The third value is for the absence of an answer, not for an answer you would rather not have.
A label that reduces the number of reds deserves a second look, especially when you chose it and it sounds principled. The reasoning that got me there was correct in general and wrong here, and I could not tell from the inside.
Top comments (0)