A measurement run of mine finished on 12 September, reported 48 out of 48 complete, wrote its day flag, and measured nothing at all. Every stage in the chain said it had worked. It took me two days to notice, and only because a number downstream looked lonely.
Here is the setup, because the shape matters more than my particular pipeline. I ask a fixed set of 16 questions against several answer engines on a fixed schedule, and score whether a given entity shows up in the answer. One engine per run, one row per question and model pair, 48 rows for that engine. The run writes rows, a scorer reads them, a day flag marks the date as done so nothing re-runs it.
What the rows actually contained
All 48 rows came back carrying a throttling marker instead of an answer. Not one row held a response. Not one had triggered a web search. Response times were one to three seconds, which for a search-grounded answer is physically too fast to be real work.
So the evidence that the run had failed was sitting inside the data, in three independent places, and nothing looked.
Three stages reported success
The day log said 48/48 ... DONE, flag set. The run log said scored 48 rows. The day flag was written.
Every one of those statements is true. They are all counting rows. None of them asks whether a row carries a value. A throttled response is a row. An error string is a row. scored 48 rows means the scorer visited 48 rows and assigned each one a score, and the score it assigns to a row with no answer in it is zero, which is also a perfectly valid score for an entity that genuinely was not mentioned.
That is the whole trap in one sentence. A failed measurement and a real zero are the same shape once you are only counting.
Why this class of bug is expensive
A crash is cheap. You see it, you fix it, you re-run. This one is expensive because it is silent and it is durable:
The day was marked done, so the retry logic will never touch it again. The zero entered the aggregate as a real observation, so the headline number for that day rests on two engines instead of three, and nothing in the published figure says so. Anyone reading the trend sees a dip and looks for a cause in the world. The cause was in the collection.
I have now spent time twice looking for an explanation on the wrong side of the instrument. The first time it was a different engine reading zero for several days and I went looking for what had changed in the engine. Nothing had. My own fetch had.
What I changed
Three things, in order of how much they were worth.
First, a run is only complete if the rows carry values. The completion check now asks how many rows hold an actual answer, not how many rows exist. Zero answers means the run failed, regardless of what any log says, and the day flag is not written.
Second, a gap is a value in its own right. There is now a gap register with an entry per missed measurement, with the reason and a snapshot as evidence. The aggregate reads it, so a day built on two engines instead of three is marked as such rather than quietly averaged.
Third, an empty result from a failed fetch never becomes an empty list. That pattern is where these bugs live: catch the HTTP error, return nothing, and the caller computes over nothing and reports serenity. Empty and failed have to be different return values, because downstream they mean opposite things.
The general rule
If your pipeline can produce a zero, ask it which kind of zero it is. Not measured, measured and absent, and failed while measuring are three different states, and most instrumentation flattens them into one number because one number is easier to chart.
The check that would have caught mine is two lines long: did any row in this run carry a value, and if not, fail loudly. I did not write it earlier because the run had never failed that way before. That is not a reason, that is just the date on which you learn it.
Top comments (0)