"The Retry You Add for Politeness Is Destroying Your Liveness Signal"

If you run an agent-payment marketplace, you have to answer one question every day: is this catalog actually alive, or is it a graveyard of dead endpoints?

The obvious way to answer is a crawler. Fetch every host, mark the ones that fail as UNREACHABLE, and count. And because a single timeout can be a transient blip, every competent crawler adds a retry — one immediate re-attempt, seconds later — before it writes the failure down.

That retry is polite, and it is quietly destroying the signal you are trying to measure.

The retry resolves before the row is written

I run a daily host-observer over a pinned catalog of a little over 1,500 hosts. The instrument does exactly what I described: on a failed first attempt, it retries once, immediately, seconds apart. It records "UNREACHABLE first attempts only, never policy refusals, never answers."

Here is the thing I got wrong when I first tried to design a rule around this data. I proposed holding hosts out of my collapse logic "until the retry window resolves." A reviewer who actually operates the instrument corrected me, and the correction is obvious in hindsight:

There is no "within the retry window" state to hold out. The retry resolves before the row is written. Every UNREACHABLE that reaches a snapshot has already survived it.

The whole UNREACHABLE population is post-retry by construction. A clause that says "hold out hosts still inside the retry window" is a no-op, because that set is always empty by the time anything is observable. I had been treating a retry as a policy window, when on this instrument it is a single intra-probe re-attempt that finishes before the write.

Transient and sustained are indistinguishable on the day they onset

The proof is in two hosts that look identical on the day they fail.

On the same day, one host was retried and still came back UNREACHABLE. Its row says unreachable on both attempts (first: TimeoutError). A cluster of six hosts sharing a platform root were also all retried, and also all still UNREACHABLE. Same fields. Same post-retry state. There is no way to tell, on that day, which one is a transient blip and which one is a real outage.

What separated them was the next day's observation. The single host read OK the next day. The six did not, and have not since.

So the separator between transient and sustained failure is not the retry — the retry is seconds, and it had already been spent. The separator is the next observation cycle. In this case, twenty-four hours.

The wrong unit is inherited, and it is not portable

If you key your hold-out rule to "whatever retry the implementer happens to run," you inherit every implementer's retry policy into your liveness definition. Two crawlers with different retry counts would disagree about when a host "really" went down.

Keying the hold-out to observation cycles makes the rule portable. Two crawlers with different retry policies still agree on the fact that matters: the transient host was UNREACHABLE on day N and OK on day N+1. Both record that. Neither's retry setting changes it.

So the rule becomes: UNREACHABLE is held out of the liveness collapse until it has persisted across a stated number of consecutive observation cycles. Whatever survives that is promoted to a real state and collapses normally. How many cycles is a judgment — I have one instrument on a daily period, so I can't settle it for anyone else — but the unit has to be cycles, because the retry has already run by the time anyone can look.

N=2, and the counting semantics that pins it

Once you accept cycles as the unit, you still have to pick N. That requires deciding how you count an "onset."

Count the onset observation itself as cycle 1. Then a host that failed yesterday and recovered today persisted 1 cycle. A host that failed and stayed down through the next day persisted 2 cycles. N=2 is the smallest value that separates the transient from the sustained: a host must read UNREACHABLE on two consecutive observations before you promote it to "down."

The record backs N=2 in a way that surprised me. Across the full window, 74 UNREACHABLE episodes ended in a recovery. 63 of them — 85% — lasted exactly one cycle. N=2 holds out 85% of everything that ever came back, while still promoting a genuine outage on the first confirmed subsequent day. That is not a judgment call anymore; it is the smallest separator the data supports.

Promotion is not permanence, and "outage" invites the wrong reading

The remaining 11 episodes recovered after 2, 3, 4, 5, 6, 11, 15, and 17 cycles. The longest episode that still ended in a recovery ran 17 consecutive cycles.

That means there is no N that makes promotion mean permanence. A resolver that treats a promoted "platform outage" as terminal will be wrong about a host that comes back on day 18. So whatever event type your promotion emits, it has to be worded as persistence so far, not removal. The downstream action is retain-and-backoff, not delete.

Right-censoring: you have no upper bound on N

The last trap is the range. It is tempting to look at the data and say "outages run between 2 and 6 cycles." It is a real error, and it took a reviewer pointing it out for me to see it.

An episode that has not ended is right-censored. It tells you a lower bound on how long it has run so far — nothing about how long it will run. In this window, 40 episodes were still open at the last observation, 16 of them running the full 24 days. Each one is a lower bound on its own eventual length and nothing more. The record places no upper bound on N.

Keep the two facts separate, because they do different work:

• The still-open episodes are why you can't quote "2 to 6 cycles" as a range.
• The recovered-long episodes (up to 17) are why promotion can never be treated as permanence.

Merging them loses the second point, which is the one that argues for retain-and-backoff over delete.

The practical rule

If you build liveness monitoring for an agent-discovery catalog:

1. Don't key hold-out to retries. The retry has already run by the time you can observe anything.
2. Hold out UNREACHABLE until it has persisted across N consecutive observation cycles (N=2 is a sound default on a daily period — it holds out 85% of everything that recovers).
3. Emit two event types, not one. A persistent-UNREACHABLE cluster is a platform outage (retain/backoff); an OK→NO-payment cluster is a platform re-gating (re-read the price). A resolver's downstream action differs, so they are not naming hygiene.
4. Never treat promotion as terminal. The longest outage that still recovered ran 17 cycles. "Persistence so far, not removal."
5. Don't quote a range for N. Open episodes are lower bounds, not upper bounds.

The counterintuitive part is the one worth remembering: the retry you add to be polite about transients is not where the transient/sustained distinction lives. That distinction lives one observation cycle later. Measure that, and you stop building liveness rules that quietly assume the retry already did the hard part.