A backend has to fail twice, and one failure is not an outage

The first version of my router failed over on the first error and flapped constantly. The reason turned out to be more interesting than the bug.

What produces exactly one failure

A laptop waking from sleep produces exactly one timeout. So does a local model being swapped in, a container restarting, a wifi handover between access points, and a GPU busy with something else for a few seconds.

None of those are outages, and all of them look identical to one if your only evidence is a single failed request.

So the router would mark the local backend down, send the next request to a paid API, watch the health check succeed, and bring it back. Repeatedly. The visible symptom was a bill. The bug was treating one sample as a measurement.

The fix

const FAILURES_BEFORE_DOWN = 2;

entry.failures += 1;
if (entry.failures >= FAILURES_BEFORE_DOWN) {
  entry.healthy = false;
}

Two consecutive failures before a backend counts as down. A success resets the counter to zero instead of decrementing it, because the thing being detected is a sustained problem, and one success in the middle means it was not sustained.

That was the whole change, and the flapping stopped.

Unknown is not unhealthy

The subtler half, and the part worth stealing.

When the router starts, no health check has run. What is the state of each backend?

The tempting answer is unhealthy, since you have no evidence it works. That produces a specific and expensive failure: every request during the first few seconds of process life goes to the paid API, because nothing has been proven healthy yet. On a short-lived CLI invocation that can be every request you ever make.

So there are three states:

// Unknown rather than healthy. Routing treats unknown as usable, so the
// first request does not skip a backend that has simply never been probed.
this.state[backend.name] = {
  healthy: null, latencyMs: null, failures: 0, checkedAt: null, error: null,
};

null means not yet known, and routing treats it as usable. A backend is only skipped once it has actually failed twice.

The distinction shows up in the output too. nearcall doctor reports unknown separately from down, because "I have not checked" and "I checked and it is broken" call for different responses.

Where the same mistake lives elsewhere

The shape shows up in several places once you have seen it.

Circuit breakers with a threshold of one open on transient errors, which is what the threshold exists to prevent.

Kubernetes liveness probes with failureThreshold: 1 restart pods that were briefly busy, producing the unavailability the probe was meant to detect.

Retry logic that gives up after one attempt does not retry.

Alerting on a single scraped sample pages someone for a garbage collection pause.

Each one treats a single observation as a measurement. One data point has no variance, so it cannot tell you whether what you saw was signal.

Picking the number

Two is not magic. It is just the smallest number above one, and one is what was broken.

The real question is how long you are willing to be wrong in each direction. Too low and you fail over on noise, paying for the cloud and losing your local advantage. Too high and you keep sending requests into something genuinely broken while the user waits for each one to time out.

Two with a short check interval keeps both windows small for this use case. A service where failover is expensive, or where a false failover is worse than a slow request, should sit higher. Picking by feel and never revisiting is the thing to avoid: watch how often single failures occur in normal operation, and set the threshold above that.

And say which one it is

Whatever number you pick, report the state honestly. My health entry keeps failures, latencyMs, checkedAt and the last error, because "down" alone does not tell you whether it just started failing or has been dead for an hour.

npx nearcall doctor

41 tests, zero dependencies. nearcall.