upordown.online

Posted on Nov 5 Edited on Nov 5

30 seconds, 1 minute or 5 minutes? Comparing uptime check intervals with a simulation you can run

#sre #python #monitoring #devops

"How often should we check?" is one of the first questions anyone setting up uptime monitoring asks. The usual answers are opinions: "every minute is standard", "30 seconds if you're serious", "5 minutes is fine for a blog".

Instead of another opinion, let's compare the options with a simulation you can run yourself. It won't give you a universal answer, but it makes the trade-offs concrete: detection delay, missed short outages and the number of requests you'll send.

Note: all numbers below come from the simulation with the parameters shown. They're illustrative, not measurements of real sites. Change the inputs to match your situation.

The question, made precise

For a given check interval, we care about three things:

Detection delay: how long between the outage starting and the first failed check?
Missed outages: how many short outages happen entirely between two checks?
Cost: how many requests per day does this interval generate?

There's a fourth one that matters in practice: confirmation. Most monitors don't alert on one failed check, because a single failure is often a network blip. They wait for 2 or 3 failures in a row, which multiplies the delay.

The simulation

We generate a month of random outages with a mix of lengths (lots of short ones, a few long ones), then "check" the site at different intervals and see what each one catches.

import random, statistics

random.seed(42)
MONTH = 30 * 24 * 3600

def make_outages(n=40):
    outages = []
    for _ in range(n):
        start = random.uniform(0, MONTH)
        # mostly short blips, some real incidents
        length = random.choice([20, 45, 90, 180, 600, 1800])
        outages.append((start, start + length))
    return outages

def is_down(t, outages):
    return any(s <= t < e for s, e in outages)

def simulate(interval, confirm, outages):
    offset = random.uniform(0, interval)
    detected, delays = 0, []
    for s, e in outages:
        # first check at or after the outage start
        k = int((s - offset) // interval) + 1
        first = offset + k * interval
        needed = [first + i * interval for i in range(confirm)]
        if all(is_down(t, outages) for t in needed):
            detected += 1
            delays.append(needed[-1] - s)
    return detected, delays

outages = make_outages()
print(f"{len(outages)} outages, lengths: {sorted(round(e - s) for s, e in outages)}\n")
print(f"{'interval':>8} {'confirm':>7} {'caught':>7} {'median delay':>13} {'checks/day':>10}")
for interval in (30, 60, 300):
    for confirm in (1, 2, 3):
        caught, delays = simulate(interval, confirm, outages)
        med = statistics.median(delays) if delays else float("nan")
        print(f"{interval:>7}s {confirm:>7} {caught:>4}/{len(outages)} {med:>11.0f}s {86400 // interval:>10}")

Run it with python3 sim.py. It uses only the standard library.

What the output shows

Your exact numbers depend on the random seed and outage mix, but the shape of the results is consistent across runs, and that's what matters.

Detection delay grows with interval and confirmation

With a 30-second interval and a single confirmation, the median delay is roughly half the interval plus a bit. With 5-minute checks and 3 confirmations, you're waiting for three checks, so the delay is often over ten minutes before anyone is alerted.

The rule of thumb that falls out:

worst-case detection ≈ interval × confirmations

So "every minute, alert after 3 failures" means you may hear about an outage about three minutes after it starts. That's often fine. Just know that it's the real number, not "one minute".

Short outages disappear at long intervals

A 20- or 45-second blip is rarely caught by 5-minute checks. With 30-second checks and one confirmation, most are caught. Add a confirmation of 2 or 3 and far fewer survive, at any interval. In one run with seed 42, 30-second checks caught 38 of 40 outages with one confirmation and 25 of 40 with three; 5-minute checks with three confirmations caught 12.

That last point is important: confirmation intentionally filters out short blips. That's usually what you want, because waking someone up for a 20-second hiccup that fixed itself isn't useful. But if your users are sensitive to short outages (payments, real-time APIs), you need short intervals and a separate way to track blips, like recording every failed check even when you don't alert.

Cost scales linearly, and from several locations

Interval	Checks/day (one location)	With 3 locations
30s	2,880	8,640
1 min	1,440	4,320
5 min	288	864

For one homepage this is nothing. For 50 endpoints checked from 3 regions every 30 seconds, it's over 400,000 requests a day. Make sure those requests are cheap for your server, and excluded from analytics.

The comparison, summarised

	30 seconds	1 minute	5 minutes
Worst-case detection (3 confirmations)	~1.5 min	~3 min	~15 min
Catches sub-minute blips	Sometimes (if logged)	Rarely	Almost never
Request volume	High	Medium	Low
False alarm risk (1 confirmation)	Higher	Medium	Lower
Good fit	Payments, APIs, revenue pages	Most production sites	Blogs, internal tools, low-traffic sites

The detection figures in this table follow from the interval × confirmations rule above; they're not benchmarks.

How I'd choose

Start from "how long can this be down before it costs us?" If the answer is "a few minutes", you need roughly interval × confirmations below that.
Use confirmation from multiple locations rather than more retries from one place. "Down from 2 of 3 regions" is both faster and more reliable than "down 3 times in a row from one region".
Mix intervals. Critical endpoints every 30 seconds, the rest every minute or five. Not every URL needs the same treatment.
Keep the failed-check history. Even blips you don't alert on are useful when you're trying to explain a customer complaint.

Most uptime tools, including upordown.online, let you set the interval and the confirmation rule per check. The simulation above is a quick way to see what a setting actually means before you pick one.

Try it with your own numbers

Change make_outages to match your history. If you have past incident data, use the real durations. Change the confirmation count. See how many of your real outages a given setting would have caught, and how late.

DEV Community