On Monday I published a piece admitting that my deploy verification tolerates sixty seconds of "not there yet" for a reason I couldn't defend. Three retries, twenty seconds apart. I picked twenty because it was the first interval where my false alarms stopped, my sample was about three deploys, and I had never once recorded how long propagation actually takes.
I made three commitments in that piece. A birth certificate for the constant. A rule fixed before the run it judges. And the one that mattered most: emit the value, not just the verdict — a check that prints only pass or fail hides the exact signal that would tell me it's miscalibrated.
I did the third one that afternoon. Every deploy since writes down how long it took to go green.
Three samples in:
Aug 03 ( 1.7, 21.7 ]s
Aug 05 ( 0, 6.7 ]s
Aug 05 ( 40, 70 ]s
They're intervals rather than points because my poll spacing is twenty seconds. All I can honestly say is that green happened somewhere between the last failed check and the first successful one — a number I can't resolve finer than my own instrument.
The third one failed
Not the deploy. The check.
I shipped a post, ran verification, and got a clean red: page 404, hero missing, sitemap entry absent. Three attempts, twenty seconds apart, exactly as designed. By its own rules the deploy had failed.
Nothing was wrong. A longer script came back 200 on everything. Total elapsed: somewhere between forty and seventy seconds, against a tolerance of sixty.
So the false alarm I widened the interval to eliminate returned on the third recorded sample, four days after I published the sentence "my sample was about three deploys." I'd like to say I predicted this. I predicted the category, not the timing, and the timing is the part that stings.
The part I hadn't considered at all
Here's what the red actually said, in order:
attempt 1 article 404 · hero missing · list page MISSING · sitemap missing
attempt 3 article 404 · hero missing · list page OK · sitemap missing
The blog index had updated. The article page it links to had not.
For about a minute the site was in a state where a reader could see the new post announced on one page and get a 404 by clicking it. Both facts were true at once, and my check has no way to say so, because it was built on the assumption that a deploy is a single transition: not there, then there.
It isn't. It's a set of files arriving independently, and "partly arrived" is a real state my instrument cannot name. It has two words for three situations. Seeing an updated list page beside a missing article, it filed that under "not there yet" — the same bucket as a deploy that hasn't started propagating at all. Those are not the same reading. One is waiting. The other is a live inconsistency a visitor can walk into.
That's the more interesting defect, and no adjustment to the retry count touches it.
What I'm not going to do
Raise twenty to thirty.
That's the move, and I can feel it pulling. It would make this week's red go away, it would take four seconds, and it would be indistinguishable from what I did the first time — widen until the alarm stops, then call the result engineering.
The numbers don't support a bump anyway. Three samples spanning 6.7 to 70 seconds is a tenfold range. No single value in that spread avoids being wrong regularly: pick 20 and you get false alarms; pick 90 and your check is now ninety seconds slow at catching a deploy that genuinely failed. When the spread is that wide relative to what you're measuring, the problem isn't which point you choose. It's that you're choosing a point.
What actually needs to change
Two things, neither of them a threshold.
Report elapsed time even on a pass. The check currently prints one character for a deploy that resolved in six seconds and one that resolved in sixty-five. Different events, indistinguishable logs, no way to see the trend that would warn me before the next false alarm. This was Monday's commitment and I only half-kept it — I started recording the interval by hand, in a separate script, which is precisely the kind of thing that stops happening in about two weeks.
And it needs a third verdict. Not pass and fail but INCONSISTENT — some resources updated, others not. That state means something different to a reader than either neighbour, and it's the only one where waiting is genuinely the right response, which is worth knowing at the moment you're deciding whether to panic.
Three states instead of two. I've now arrived at that same shape from four unrelated directions in one week:
resolved / genuinely absent / unreachable
pass / fail / never ran
visible / removed / never existed
arrived / arriving / half arrived
Every time, the missing third state was one I'd collapsed into a neighbour because I hadn't imagined it could happen.
The honest ending
I still cannot tell you the right number.
Three measurements with a tenfold spread is not a distribution. It's three measurements. What I have is a series that might become one, and that's more than I had a week ago — but only because I wrote the value down at the moment of the check instead of walking away with the impression that it felt fine.
That's the whole difference, and it's smaller than it sounds. Monday's version of me had exactly the same experience of deploying. What he didn't have was a file.
This is part of Stolen from the Feed — what a non-developer carries home from dev.to and actually builds. Sister series to From Zero to Ship.
Top comments (0)