Every uptime monitor I've used has woken me up for an outage that never happened.
The mechanism is always the same, and it isn't a bug. A monitor checks your endpoint from one location. Somewhere between that prober and your server, a packet drops, or a route flaps, or a TLS handshake takes 400ms longer than the timeout allows. From the prober's point of view, that is indistinguishable from your service being down.
This is worth stating precisely, because it explains why the usual fixes don't work: a single observer cannot separate a fault in the thing being observed from a fault in the path to it. It's not a tuning problem. It's an epistemic limit of having one witness.
So we built the check layer around having several, and requiring them to agree. This post is how that works, including the four cases that took far longer to get right than the happy path.
Why the usual fixes don't fix it
Before the interesting part, the two things everyone tries first.
Retries. Check again after N seconds; only alert if it fails twice. This genuinely helps with single dropped packets, and you should turn it on whatever tool you use. But it doesn't help at all with anything that persists for more than a few seconds — a route flap lasting 90 seconds fails every retry from that one vantage point, and you get paged for a network condition that never touched your users.
Multiple probes in one region. Better, and it's what several vendors mean by "multi-node." But probes in the same datacenter share upstream transit, share a BGP view, and often share an ASN. Their failures are correlated, and correlated witnesses don't give you independent confirmation — they give you one opinion with extra steps.
What you actually need is witnesses that fail independently. That means genuine geographic and network separation, which historically meant operating servers in seven countries. That's why consensus verification only ever existed in enterprise tooling: not because the idea was hard, but because the infrastructure bill was.
Why Durable Objects change the economics
Cloudflare Durable Objects have one property that matters enormously here: you can pin an instance to a geographic region with locationHint, and it stays there, holding state, with no servers to operate.
const REGIONS = [
'wnam', // western north america
'enam', // eastern north america
'weur', // western europe
'eeur', // eastern europe
'apac', // asia-pacific
'apac-ne', // japan / korea
'apac-se', // southeast asia
] as const;
type Region = typeof REGIONS[number];
function probeFor(env: Env, region: Region): DurableObjectStub {
const id = env.PROBE.idFromName(probe:${region});
return env.PROBE.get(id, { locationHint: region });
}
That's the whole trick. Seven pinned instances, each egressing from a different part of Cloudflare's network, each capable of independently checking an endpoint and reporting what it saw. What used to be a procurement exercise is now a scheduling problem.
The cost collapse is the point. It means an architecture that was previously enterprise-only can run on a free tier, which is the reason we could build this as a small team at all.
The check cycle
Each cycle, a coordinator fans out to all regions in parallel and collects results. The important design decision is in the result type:
type ErrorClass =
| 'dns_failure'
| 'tcp_refused'
| 'tcp_timeout'
| 'tls_error'
| 'http_5xx'
| 'http_4xx'
| 'body_mismatch';
type ProbeResult =
| { region: Region; status: 'ok'; ms: number }
| { region: Region; status: 'fail'; errorClass: ErrorClass; ms: number }
| { region: Region; status: 'inconclusive'; reason: 'timeout' | 'flapping' | 'unreachable' };
Note that there are three states, not two.
Almost every monitoring system I've looked at models a check as binary — up or down. That collapse is where false positives are born. A region that didn't respond in time has not told you the target is down; it has told you nothing. Those are completely different pieces of information and they must not be added together.
The fan-out itself is unremarkable:
async function runCycle(env: Env, monitor: Monitor): Promise {
const settled = await Promise.allSettled(
REGIONS.map(async (region) => {
const stub = probeFor(env, region);
const res = await stub.fetch('https://probe/check', {
method: 'POST',
body: JSON.stringify({ url: monitor.url, timeoutMs: monitor.timeoutMs }),
});
return (await res.json()) as ProbeResult;
})
);
return settled.map((s, i) =>
s.status === 'fulfilled'
? s.value
: { region: REGIONS[i], status: 'inconclusive' as const, reason: 'unreachable' as const }
);
}
If we can't reach our own probe, that's inconclusive. Not a failure of the customer's endpoint. It sounds obvious written down; it is very easy to get wrong when you're mapping exceptions to booleans at 2am.
The local re-check
Before a region reports a failure at all, it re-checks immediately:
async function check(url: string, timeoutMs: number): Promise {
const first = await attempt(url, timeoutMs);
if (first.status === 'ok') return first;
// A failure is a hypothesis, not a conclusion. Test it again now,
// rather than waiting for the next cycle.
await sleep(250);
const second = await attempt(url, timeoutMs);
if (second.status === 'ok') {
// Transient. Record it for latency analytics, report healthy.
recordTransient(url, first);
return second;
}
return second;
}
This is cheap and it kills a surprising amount of noise — single dropped packets and one-off TLS timeouts die here, silently, without ever entering the consensus layer. Do this even if you never build quorum.
The quorum decision
Here's the core. Given seven results, decide what happened:
type Verdict =
| { kind: 'healthy' }
| { kind: 'regional_degradation'; failing: Region[]; dominant: ErrorClass }
| { kind: 'global_outage'; failing: Region[]; dominant: ErrorClass }
| { kind: 'insufficient_witnesses'; counted: number; required: number };
function decide(results: ProbeResult[], quorum = 4): Verdict {
// Inconclusive results are excluded from the denominator entirely.
const counted = results.filter((r) => r.status !== 'inconclusive');
const failing = counted.filter((r) => r.status === 'fail');
// If too few regions gave us a real answer, we cannot make a call.
// Saying "I don't know" is a valid and important output.
if (counted.length < quorum) {
return { kind: 'insufficient_witnesses', counted: counted.length, required: quorum };
}
if (failing.length >= quorum) {
return {
kind: 'global_outage',
failing: failing.map((f) => f.region),
dominant: dominantErrorClass(failing),
};
}
if (failing.length > 0) {
return {
kind: 'regional_degradation',
failing: failing.map((f) => f.region),
dominant: dominantErrorClass(failing),
};
}
return { kind: 'healthy' };
}
Two things in there are worth more than the rest of the function.
Inconclusive results are excluded from the denominator, not counted as failures. If a region times out and you count it as a failure, one congested route can push you over quorum and page someone for nothing. If you count it as a success, one congested route can suppress a real outage. Both are unacceptable, and the only correct answer is to remove it from the vote.
insufficient_witnesses is a real verdict. If five of seven regions come back inconclusive, you do not have enough information to say anything about the target. Most systems would either alert or stay silent; both are wrong, because both assert something you don't know. We surface it as its own state and it goes to our own operational alerting, not the customer's.
And the classification matters as much as the threshold. Three regions failing while four succeed is not an outage — it's almost always a CDN edge or a geo-routing rule, affecting some users and not others. That deserves a different notification, at a different urgency, than "your service is gone." Flattening both into the word DOWN is how you train people to ignore the word DOWN.
The four things that took longest
The happy path above took a couple of weeks. These took months.
- Flapping probes A probe with a bad network path reports failures that aren't real. If one of your own regions is unhealthy, it poisons every decision it participates in. So probes are continuously assessed and ejected when they misbehave:
const FLAP_WINDOW_MS = 2 * 60 * 60 * 1000;
const FLAP_THRESHOLD = 3;
function isFlapping(transitions: number[], now: number): boolean {
const cutoff = now - FLAP_WINDOW_MS;
return transitions.filter((t) => t >= cutoff).length >= FLAP_THRESHOLD;
}
Three or more state transitions in two hours and the probe is marked flapping and removed from quorum until it stabilises. Its results become inconclusive rather than being discarded silently — the distinction matters when you're debugging your own fleet later.
We publish probe health, including flapping status, on a public page. That was uncomfortable to commit to and has been unambiguously worth it.
- Correlated failure of the witnesses This is the subtle one, and if you build something like this, it's the bug you'll ship.
All seven regions run on Cloudflare. They are geographically independent. They are not independent of Cloudflare. If Cloudflare has a bad day, all seven probes can fail simultaneously — and a naive quorum implementation reads seven-of-seven unanimous failure as maximum confidence and pages every customer at once with a completely false alarm.
Your consensus system, at its moment of greatest apparent certainty, is maximally wrong.
The fix is a witness that doesn't share the failure mode: an out-of-band sentinel on a completely different ASN, on a cheap VPS, deliberately not on the same network as anything else we run.
function reconcile(edge: Verdict, sentinel: ProbeResult | null): Verdict {
// Unanimous edge failure + a healthy independent sentinel means
// the problem is far more likely to be us than the customer.
if (
edge.kind === 'global_outage' &&
edge.failing.length === REGIONS.length &&
sentinel?.status === 'ok'
) {
return { kind: 'insufficient_witnesses', counted: 1, required: 4 };
}
return edge;
}
When every edge probe disagrees with the one witness on a different network, suspect the observer, not the target. This is the single highest-value component of the whole design, and it runs on a VPS costing a few euros a month.
- Distinguishing a blocked probe from a dead one If a probe stops reporting, you need to know whether it crashed or whether something is preventing it from reaching the target. Those imply completely different responses.
So every probe reports liveness on a channel entirely separate from its measurement path. A probe that is alive but measuring nothing is a routing or firewall problem. A probe that is silent on both is a crashed probe. Same symptom in the results table, different root cause, and you cannot tell them apart without the second channel.
- Being un-allowlistable Worth mentioning because it's a real operational consequence of this architecture. Because probes run as Durable Objects egressing through Cloudflare's shared edge IP pool, there is no stable IP list customers can allowlist. We can't give them one, and any list we published would be wrong within a week.
Instead customers match on a header:
Cloudflare WAF custom rule
(http.request.headers["cf-worker"][0] eq "steadystack.dev")
Action: skip WAF / rate limiting
If you build synthetic checks on edge compute, plan for this early. It surprised us, and "just allowlist our IPs" is the first thing a security-conscious customer asks for.
What it costs
The honest accounting: seven regions checking every cycle is 7× the check volume of a single-probe monitor, per monitor, forever.
On classic infrastructure that would be prohibitive at the low end of the market. On edge compute, a check is a few milliseconds of CPU and one outbound request, and the per-check cost is small enough that we ship multi-region confirmation on our free tier rather than gating it behind a plan. That was a deliberate decision — the whole thesis is that consensus shouldn't be a premium feature, because a monitor you can't trust isn't a cheaper monitor, it's a broken one.
What I'd do differently
Model the three-state result on day one. We started with a boolean and retrofitted inconclusive later. That refactor touched everything, and every subtle bug we shipped in the first months traced back to somewhere the two-state assumption was still hiding.
Build the out-of-band sentinel before the seventh region. We added regions for coverage before we added a witness for independence. Coverage is worth less than independence, and it took a scare to notice.
Store the error class from the beginning. Knowing that four regions failed is useful. Knowing they all failed at TCP connect rather than with 5xx is the difference between "routing" and "your origin," and it's the first question anyone asks during an incident.
Honest limits
We check from North America, Europe, and Asia-Pacific. We do not have probes in Africa or the Middle East. If your users are concentrated there, a global outage will still page you, but a problem affecting only those users may not surface. That's on our locations page rather than buried here, because a padded coverage map is worse than a small one.
And consensus is not free of tradeoffs. Requiring four regions to agree means a genuine outage affecting only three regions won't page you as an outage — it'll be classified as regional degradation. We think that's correct. If you disagree for your workload, that threshold should be yours to set.
I build SteadyStack, which is where all of this runs. The free tier is 50 monitors with multi-region quorum included, if you want to look at the mechanism rather than read about it.
If you're building something similar, I'd genuinely like to compare notes on the correlated-witness problem — it's the part I've found least written about, and I suspect a lot of consensus systems have it without knowing.
Top comments (0)