Most checkers answer one of two ways: pass or fail. I want to make the case that a QC tool with only
two answers is structurally dishonest, because it has to guess whenever it can't actually tell — and a
guess dressed as a green check is the most dangerous thing a validator can emit.
Here's the failure mode that shaped the engine I work on. Upload a scanned PDF with no text layer to an
accessibility checker. It looks for tagged headings, finds none it can read, and prints pass — not
because the document is accessible, but because there was nothing it could parse, and its vocabulary only
contained "pass" and "fail." Absence of a detected problem got reported as absence of a problem. Those
are different statements, and the gap between them is where compliance tools quietly lie.
Three states, not two
So every check returns one of three things: pass, fail, or not evaluable. The third is a
first-class citizen, not an error path. It means: the tool looked, and it cannot honestly determine the
answer from what it was given. No text layer to assess. No template declared for the thing being checked.
A 2D barcode that would need image decoding that isn't shipped yet. A resource the parser reached but the
module doesn't cover.
The rule is absolute: a check that cannot run is never allowed to read as a pass. If we can't evaluate
it, we say so, in those words, and we say why.
Why it's harder than one function
Adding a third return value is easy in isolation. Keeping it honest across an entire pipeline is not,
because "not evaluable" is the state most likely to get flattened at a boundary. A module throws on a
malformed stream; if the surfacing layer coerces that into an empty findings array, the public API
cheerfully reports complete, zero findings — a false green built from a swallowed exception.
I have found and fixed that exact class of bug more than once: a not-evaluable or errored signal dropped
at the CLI, the webhook, the scorecard, or the JSON boundary — each one turning "we don't know" back into
"looks fine." The lesson: the verdict has to be carried, not recomputed. Coverage travels alongside
findings all the way out. The CLI renders it and exits non-zero. The webhook marks the result partial.
The in-app view labels how many checks actually ran vs were not evaluable vs errored. The counts have to
add up, or a test fails.
The cost, and why it's worth paying
A tool that says "not evaluable" a lot looks less impressive than one that prints green everywhere. That's
a real cost — it's a worse demo than a wall of check marks. But the user here is someone whose statements,
checks, or EOB mail is already going out the door. For them a false pass isn't a missing feature; it's the
incident. An honest unknown lets them go look. A dishonest pass tells them not to.
A validator that never fails anything is just a mirror. A validator that never admits it doesn't know is
worse: a mirror that lies about what it reflects.
I write about this and the rest of the engine here — this piece in full, with the pipeline detail:
https://preflightdocs.com/blog/the-not-evaluable-verdict?utm_source=devto&utm_medium=article&utm_campaign=noteval
And you can watch the third verdict happen on a real file in the free analyzer:
https://preflightdocs.com/?utm_source=devto&utm_medium=article&utm_campaign=noteval
Top comments (0)