Three different things come back looking identical. A check ran and found nothing. A check started and fell over. A check never happened. In a structured tool surface all three arrive as [], or null, or a key that is simply absent, and the caller has to decide what that means with nothing to go on but the shape.
It decides in favor of the good news. That is not a failure of temperament. The success path already owns the empty value, so every failure path that produces the same value inherits the meaning the success path established. Quiet reads as healthy because healthy is quiet.
Reading it wrong once is a bad inference. Writing it down is the expensive part. An ambiguous value that lands in a dedup key, a cache entry, an idempotency guard, or an append-only log is a bad inference with tenure. Something reads that row back weeks later and sees a clean verdict. No clean verdict was ever produced. A harness had a bad afternoon, and the only surviving record of that afternoon now certifies the code.
The empty array is doing two jobs
Take a bounded tool surface that exposes language-server diagnostics as structured results. The agent calls diagnostics(path) and receives []. Two worlds produce that response. In one, the server indexed the repository, analyzed the file, and found nothing wrong. In the other, the server came up and never finished indexing anything at all. The bytes are equal. No amount of careful downstream reasoning recovers the difference, because the difference was never transmitted.
The usual repair is a sibling call. server_status() sits next to the diagnostics tool and reports readiness, and the caller is expected to consult it before trusting an empty array. This fails twice. It requires remembering, and removing the need to remember is most of the reason a bounded tool surface exists instead of a shell, where a nonzero exit code and a line on stderr would have carried the failure on a channel the payload does not control. It also races. Readiness observed at one moment says nothing about whether the request issued afterward ran against the index you believe it ran against, or against any index.
Availability has to live inside the response that can be ambiguous, in the same object, produced by the same execution. A liveness bit is too weak, because a process can be alive and useless. An index-available bit is too weak on its own if the index covers a revision other than the one you asked about. The response needs to carry enough of its own execution context to say what its empty array is empty of.
An honest answer with nowhere good to go
The same shape shows up, more durably, wherever a scanner is allowed to abstain. Letting a stage say "I cannot prove this yet" is a real improvement over letting it assert an ungrounded critical. The abstention still has to be written somewhere, and that is where it goes wrong.
Say it gets stored flat, as not_reproduced. The finding row carries that string, and the dedup key is built from the candidate identifier plus that outcome. Next run, the key matches. The investigation is skipped. Nothing ever disproved the candidate, and yet the stored value now governs whether evidence about it can be gathered again. Honesty hardens into permanence by way of a storage policy.
The reason is what determines the claim, and the reasons are not the same kind of statement. No reachable entry point found is a claim about the code, bounded by the search scope. The sandbox lacked a dependency is a claim about the harness. The stage ran out of budget is a claim about the scheduler. Only the first is a statement about the artifact under review, and only the first has any business surviving to suppress future work. The other two describe weather. Persist them as reusable negative verdicts and the weather gets read back, later, as climate.
The metric that cannot see it
Escaped-defect rates will not catch this, and it is worth being precise about why. Those rates count against what shipped and got evaluated. A candidate suppressed by a cached abstention never entered the denominator as a missed investigation. It left no row anywhere that says an investigation was owed. A measurement whose denominator excludes the failure mode is structurally incapable of detecting it.
The nastier version is that the dashboard improves. Fewer hard candidates receive an investigation capable of returning an unfavorable answer, so the visible rate goes up while coverage goes down. Catching that needs its own accounting rather than a better threshold: count suppressions as first-class events, keep unresolved candidates in a column separate from clean ones, and re-run a sample of suppressed candidates once whatever blocked them has cleared.
Absent, unset, and default are three things
The small version of this lives in wire formats. A consumer reads a missing blocked field as false. A producer that never evaluated the condition now appears to have granted permission, and the permission is attributed to whoever failed to write the field. Silence acquired an author.
It gets worse at the decoding layer. If the deserializer materializes defaults on the way in, application code never sees the distinction between a field the producer left out and a field the producer deliberately set to false. The evidence was destroyed one layer below the code that needed it, which is why adding a check in the handler does not help.
Chains are blind to the end
Append-only hash chains are the same problem in log form. Each record commits to its predecessor, so any edit to a record that is still present breaks a link and gets caught. Delete a suffix and the surviving chain verifies perfectly. The verifier walks to the shortened tail and stops, because nothing among the remaining records asserts that later records ever existed.
Tail integrity needs a commitment held outside the boundary that can truncate. A retained record count is enough to notice that entries went missing. An expected terminal digest is stronger, since it names the tail the verifier should have reached. Storing either one beside the log, under the same delete permission as the log, buys nothing at all, because the operation that removes the tail removes the witness too. An anchor also only covers the history it committed to, so a verification result should report where its coverage stops. Otherwise a valid prefix quietly becomes a claim about the whole history.
What survives
Start at the observation boundary, with a return type that can express all three states:
{
"state": "observed",
"scope": { "revision": "<content digest>", "target": "<path>" },
"findings": []
}
observed means the check completed over the declared scope, and only then does an empty findings support a negative conclusion. unavailable carries a typed reason for why the tool could not complete. not_run records that no attempt was made, which happens more often than anyone expects once a stage upstream starts making budget decisions.
Reject state: "observed" in the schema when the coverage information is missing. And resist the urge to include findings: [] in every variant for the convenience of callers, because that single act rebuilds the entire trap for anyone who reads one field and ignores the rest.
Abstention records need the same treatment. Store a reason class such as harness_dependency_missing along with the specific dependency and the environment fingerprint of the attempt, and give the record a finite reuse lifetime. scheduler_budget_exhausted needs the stage and the allocation that fell short, and it should probably produce a scheduling decision rather than a verdict. Keep in mind that retention and reuse are separate questions. An old failure is useful for diagnosis long after it has lost the right to authorize a skip.
Then make the dedup key carry the reason class, the code revision, and, for operational failures, the environment fingerprint. The key alone will not save you. A matching record still has to pass a reuse policy before it is allowed to suppress execution, or you have only made the permanent skip more precisely addressed.
All of this costs something real. More fields, more validation paths, more stored bytes as reason histories accumulate across repeated attempts. Expiry means paying for re-attempts that a permanent dedup entry would have saved, and an external anchor is another durable write that needs its own policy for the times it fails. What you get back is that every skipped check has an inspectable justification with an end date, and every empty result can be traced to an execution that actually happened.
So: which stored value in your pipeline can be produced both by a check that passed and by a check that never ran?
Top comments (0)