A delivery report counts the times a new build became live while a worker was still pointing at the old build. That number matters: a rising count is the trigger to roll the delivery mechanism back. Two separate bugs made the number lie in opposite directions. One hid failures that had happened. The other invented a zero that had not been measured.
Counting the outcome instead of the cause
When a promotion fails because a worker is on the wrong build, the recovery tries a rollback and then probes whether it worked. The probe has two outcomes. It succeeds, and the record is labelled "rolled back". It fails, and the record stays UNKNOWN — the rollback may not have happened, and the report is not allowed to guess.
The old report counted only the first outcome. That is a natural thing to do: "rolled back" looked like the canonical form of a wrong-pin failure, and the probe failure looked like a different kind of event. But the revert rule cares about causes. If wrong-pin rollbacks rise, revert. A rise in the second kind — probe failed, state unknown — was invisible to that count, because those records were filed under a different outcome and never entered the total.
The fix counts by cause. Both phases of the same cause are now visible in the same line: a recent 72-hour window showed two wrong-pin failures, one rolled back cleanly and one with an UNKNOWN phase. A single number of one would have hidden half the signal, and it would have been the more worrying half.
The record that could not be read
The report builds its counts from promotion records stored as files. It reads each one and skips what it cannot parse. That is harmless for a stray file. It is not harmless for a record that belongs in the window: skipping it is the same as asserting that nothing happened there.
A record can be unreadable in more than one way. It can be truncated JSON. It can parse as JSON but be an empty list or an empty object, with no usable timestamp. All three are missing evidence, not an empty series. The report now counts them separately and, when any is present, prints the window total as UNKNOWN with the number of unreadable records beside it.
There is a second half to the fix. Making the total UNKNOWN must not erase what was readable. If a confirmed wrong-pin failure sits next to a torn record, the report prints both: the total stays UNKNOWN, and below it a line reads "observed wrong-pin consumer failures: 1 (rolled back 1, UNKNOWN phase 0)". The word "observed" carries the boundary. One line says what the evidence cannot establish; the next says what it did establish. A reader does not have to choose between them.
The tests pin both halves, including three malformed record shapes and the mixed case. They also assert that a torn record is still torn afterwards — the reporter reads evidence, it does not silently patch it to make the table look complete.
Why the label is part of the measurement
These are small changes to a reporting view, but they encode a rule that travels: a count is a claim about the evidence behind it, not just about the world.
- Count by cause when the decision is about a cause. Grouping by outcome can scatter one phenomenon across several buckets, and the bucket you ignore may be the one that should trigger action.
- A missing input makes the total UNKNOWN, never zero. "We could not read one record" and "nothing happened" are different statements, and only one of them is supported.
- Keep the failures you can still see. Honesty about an incomplete total is not the same as discarding the complete part of it.
The check lives with the code, so the next reader can verify the wording and the cases instead of trusting this account. When a dashboard says zero, the useful question is no longer whether the number is small; it is whether every input was readable.
Top comments (0)