Originally published at ai.bedvibe.studio.
Most validation tooling has two states: it passed, or it failed. Everything that was not actually evaluated has to be forced into one of them — and it is wrong in both directions.
I did not work that out from first principles. I worked it out by shipping the same defect four times.
One: a run that learned nothing, reported as healthy
In trainproof, a training run whose loss was exactly 0.0 on every step returned PASS.
Every loss-shape check in that tool is guarded against dividing by zero. A curve that is identically zero trips every guard, so all of them skipped — silently. And then the report listed those same skipped checks as having run. A run that learned nothing passed, accompanied by a list of the checks that had cleared it.
That is the whole problem in one artifact. The tool had no way to say I could not judge this, so the absence of a judgement rendered as a favourable one.
Two: the same tool, one loop earlier
The fix for that was a report field listing which checks ran and which did not. Good. Then a nastier version turned up somewhere else.
trainproof doctor walks a directory twice — once to discover candidate logs, once to judge them. The judging pass reported anything it could not read. The discovery pass had except Exception: pass.
So a file that raised while being found never became a candidate, and never appeared in the report at all. Plainly visible on disk. Absent from the output. Indistinguishable from a file that passed.
Same failure as the first, one loop earlier than I had been looking. That is when I stopped treating it as a bug.
Three: not my system
I wrote the first two up publicly. Someone working in infrastructure compliance replied with the identical shape from a domain I know nothing about.
A compliance framework document is mostly prose. Most of it describes things no generated artifact can satisfy or violate — staff training, review procedures, who signs what. Only a fraction maps to something a machine can check.
The trap is reporting against the framework's name. Do that and everything unevaluated looks identical to everything that passed, and the ninety per cent that was never in scope disappears from the output entirely.
That exchange is why this became a library rather than another trainproof feature. One person hitting a shape twice is a habit. Two people hitting it in unrelated domains is a primitive.
Four: found while I was busy being pleased with myself
Then I ran an evaluation harness of my own over a retrieval experiment and found three instances in a single afternoon.
It recorded model refusals under a failure type asserting an ordering the model had never given. It scored ten refusals as correct, because the expected phrase appeared inside the sentence explaining what could not be determined. And it missed eight correct answers because its negative pattern required a comma.
One absent value. Wrong in both directions. Inside the instrument I was using to judge my own hypothesis.
The states
Three instead of two — checked, could-not-check, never-in-scope — and eight once you ask two more questions of each gap: who can fix it, and can it ever change.
| State | Owner | What it means |
|---|---|---|
CHECKED |
— | a determination was made; the verdict is separate |
NOT_CHECKED / DATA_DEGENERATE |
the data | the signal is present and unusable |
NOT_CHECKED / CHECKER_FAILED |
your tooling | the checker raised, timed out, or could not observe |
NOT_CHECKED / WAIVED |
a named person | in scope, deliberately not evaluated, accepted |
NOT_CHECKED / PREREQUISITE_FAILED |
another target | something upstream failed first |
OUT_OF_SCOPE / CALLER |
the caller | not requested |
OUT_OF_SCOPE / DATA_TRANSIENT |
the deployment | does not apply yet |
OUT_OF_SCOPE / DATA_PERMANENT |
nobody | no artifact of this kind can ever evidence it |
Coverage is not verdict. CHECKED is not a result — it says a determination was made, not what it was. The verdict vocabulary belongs to your domain (pass/warn/fail, compliant/non-compliant) and hangs off CHECKED rather than sitting beside the not-checked states. Collapsing those two axes is failure one above. The constructor enforces the separation: a gap cannot carry a verdict, and a checked record cannot carry a skip reason.
What the review changed
Boris Teplitsky, from the compliance side, put three corrections into the schema that I would not have found:
Permanence is relative to a target, never absolute. "Nobody, never" is not a property of a control — it is a property of pairing that control with a kind of artifact. A Kubernetes control is permanently out of scope only while the target has no Kubernetes; change the target and it becomes a row. So OUT_OF_SCOPE / DATA_PERMANENT now requires a permanent_wrt field naming the reference. Unqualified, two reports on the same framework disagree and both are correct.
Rows for the checkable subset, one count for the rest. A framework document is hundreds of pages of which a few paragraphs concern anything an artifact can evidence. One row each makes the report noise. So the excluded corpus can be a single counted rule — and the count cannot be stated without the rule that produced it, and it enters the denominator, so excluding 412 of 415 reports 99% rather than a flattering silence.
The taxonomy starts after the unit exists. In a linter, a row is a check somebody wrote. In compliance, turning prose into checkable requirements is the hard part and it is a judgment call. Nothing in the library governs that, and it should not read as if the rows arrive by themselves.
That last one is scope, not code, and it was the correction I was most wrong about.
And one the change found in my own reporter
Running the README example through the new bulk path printed 413 out of scope (100% of all targets) while one target had in fact been checked — .0% rounded 99.5 up. A reader takes "100% out of scope" as "nothing was measured."
That is this library's own failure mode, committed by its own renderer, one layer above the schema it protects. Fixed both directions: 100% only when everything is excluded, 0% only when nothing is.
MIT, no dependencies, 74 tests: github.com/Mormolykos/notchecked
If you have hit this shape in a fifth domain, I would like to hear it — the case list is a test file now, so a new case goes in with credit.
Top comments (0)