Kill switch for noisy uptime checks: a feature flag to disable a polling client

KenjiTanaka6849 — Thu, 06 Aug 2026 00:16:37 +0000

Use a kill switch inside the checker when your uptime probes start amplifying an incident — one feature flag, read on every tick, that can disable the noisy checks and stop the retries at the source. Reach for tuned backoff and jitter instead when the retry storm stays inside a single process and never fans out onto a dependency somebody else is paging for.

Both are cheap to build. Only one of them lets you quiet a polling client while its target is already on fire.

I run cron and queue infrastructure, so most of my pages arrive as either "the job didn't run" or "the job ran four times." Health checking sits in the same family of problems: a small, frequent, automated request that multiplies badly when something upstream changes shape. What follows is the runbook I settled on after a fleet of pollers turned a non-incident into a real one — the failure mode, where the switch belongs, the implementation, and how to verify the flip before you walk away from the terminal.

What actually turns a polling client's uptime checks into a retry storm?

Amplification. A single check is one request every 15 or 30 seconds, which nobody notices; a fleet of checks with retries layered on top is a synchronized load generator pointed at whatever you decided was important enough to monitor. The math is unkind. Take 40 instances, a 5s interval, and 3 retries per failed attempt, and a dependency that normally handles a trickle of health traffic suddenly sees a couple thousand requests a minute — all of them arriving at the exact moment it's least able to absorb them. Retries stack on top of the polling interval rather than replacing it, and because every poller sees the same failure at the same time, they all back off together and return together. The Google SRE book calls out this shape under cascading failures, and the load pattern that comes out of it looks nothing like organic traffic: sawtooth spikes, perfectly aligned, growing until something sheds load.

The worst one I've dealt with wasn't even a real outage. Our checker read status off the top level of a health payload, a routine change moved it under checks[0].state, and the field I assumed was there came back as an empty string. Comparison against "ok" never matched, so every probe in the fleet concluded the service was sick. The log line we emitted said health check failed and nothing else — no field name, no body snippet, no status code. 38 pollers, 5s interval, 3 retries each, roughly 1,900 requests a minute against an endpoint that normally sees 120. The service was healthy the entire time. It took us about 25 minutes to work out that the fault was in the shape of the data, not the state of the system, and most of that time went into arguing about which deploy had "broken prod." I'm still not sure why we never logged the raw body on a parse miss; we do now.

That's the case a kill switch is for. Not for hiding a real failure, but for cutting the amplifier when your own tooling has become the loudest client on the box.

Put the kill switch where the check runs, not where it's configured

The instinct is to disable checks in the monitoring configuration — comment out the check, redeploy the config, move on. Mid-incident that's the wrong layer. Config lives on the deploy path, and the deploy path is exactly what you don't want to exercise while you're troubleshooting, because it's slow, it needs review, and it's a second source of change during an event you're already trying to explain.

The switch belongs in the polling client, evaluated on each tick, with the last known value cached in memory.

Where the switch lives	Time to take effect	Blast radius	Usable mid-incident
Config file plus redeploy	minutes to hours	whole service	no
Env var plus rolling restart	minutes	one deployment	rarely
Feature flag read per tick	one interval	one check, one cohort	yes
Circuit breaker in the client	milliseconds	one dependency	automatic, not aimable

A circuit breaker and a flag solve different halves of this. The breaker reacts in milliseconds to error rates it can measure locally, which is what you want for the common case; the flag is the aimable one, the thing a human points at a specific check when the automation's model of "healthy" is wrong. Run both.

Two properties matter for the rollout. First, the flag decision has to be stable per probe: hash the probe ID into a bucket so that a 25% rollout means the same quarter of the fleet stays disabled tick after tick, rather than a random quarter each time — flapping suppression is worse than no suppression because it makes the graphs unreadable. Second, a flag fetch that doesn't land must keep the previous decision instead of reverting to a default. A safety valve that quietly re-arms itself when the flag store is unreachable isn't a safety valve.

The safe implementation: one flag read per tick

Here's the core of the Go poller we run. The ticker is the retry — one attempt per interval, no inner retry loop, because an inner loop is how you turn a 15s cadence into a 3s cadence without anyone noticing.

package main

import (
    "context"
    "encoding/json"
    "hash/fnv"
    "math/rand"
    "net/http"
    "time"
)

type flags struct {
    UptimeChecksDisabled bool   `json:"uptime_checks_disabled"`
    DisabledPercent      int    `json:"disabled_percent"` // 0..100, cohort by probe ID
    Reason               string `json:"reason"`
}

type checker struct {
    probeID string
    client  *http.Client
    last    flags // last known decision; a fetch that doesn't land keeps it
}

func (c *checker) loadFlags(ctx context.Context) flags {
    req, _ := http.NewRequestWithContext(ctx, http.MethodGet, "http://flags.internal/state", nil)
    resp, err := c.client.Do(req)
    if err != nil {
        return c.last
    }
    defer resp.Body.Close()

    var f flags
    if err := json.NewDecoder(resp.Body).Decode(&f); err != nil {
        // Shape changed under us. Keep the decision we already had and say so.
        logDecode(c.probeID, err)
        return c.last
    }
    c.last = f
    return f
}

// Stable cohort: the same probe lands in the same bucket every tick.
func inCohort(probeID string, percent int) bool {
    h := fnv.New32a()
    h.Write([]byte(probeID))
    return int(h.Sum32()%100) < percent
}

func (c *checker) tick(ctx context.Context) {
    f := c.loadFlags(ctx)
    if f.UptimeChecksDisabled && inCohort(c.probeID, f.DisabledPercent) {
        // Silence is ambiguous. Emit a heartbeat so "suppressed" and "dead poller" look different.
        heartbeat(c.probeID, "suppressed", f.Reason)
        return
    }
    probeOnce(ctx, c.client)
}

func main() {
    c := &checker{probeID: hostProbeID(), client: &http.Client{Timeout: 3 * time.Second}}
    ctx := context.Background()
    for {
        c.tick(ctx)
        // Jitter, so a fleet-wide restart doesn't align every poller on the same second.
        time.Sleep(15*time.Second + time.Duration(rand.Intn(3000))*time.Millisecond)
    }
}

heartbeat, probeOnce, logDecode and hostProbeID are elided; the only interesting thing about them is that the heartbeat fires whether or not the probe ran. Our Node.js pollers implement the same contract — same flag names, same heartbeat, same one-attempt-per-tick rule — because two different meanings of "disabled" across one fleet is how half your probes keep hammering after you thought you'd stopped them.

One more habit from queue work: make the alert side idempotent too. If the alert dispatcher keys on check name plus a coarse time bucket, a suppressed check that comes back doesn't replay a backlog of notifications at whoever is holding the pager.

Verify the flip, then plan the way back

Flipping the switch is the easy part. Confirming it did what you meant takes about three minutes and saves the argument later, so I do it in a fixed order: request rate at the target should drop within one polling interval, the suppressed heartbeat should appear for exactly the cohort you aimed at, and at least one canary probe should still be reporting so you haven't traded noise for blindness. If the rate doesn't move, the usual causes are a poller that caches flags for longer than its interval, a second checker nobody remembered (ours lived in a synthetic-monitoring job on a different schedule), or a cohort percentage that rounds to fewer instances than you expected.

The way back deserves as much thought as the way out.

Re-enabling a suppressed fleet all at once is its own thundering herd, so walk the percentage down in steps and watch the target's latency between them. Put an expiry on the flag with an owner attached — a kill switch with no removal date turns into permanent, undocumented blindness, and six months later nobody can tell you whether that check was retired on purpose. Write the flag name and the flip time into the incident timeline while you're at it. Half of postmortem archaeology is working out which suppression was still active when the next thing broke.

When a flag is the wrong tool

If the checks are noisy because the threshold is wrong, fix the threshold; a kill switch just moves the bad signal out of view and the trade-off is that you'll rediscover it during the next incident. Flags are also not suitable when the amplification needs sub-second correction, or when the checker can't reach the flag store from wherever it runs — stick with a client-side circuit breaker and a hard request budget for those. And if you find yourself adding a switch per check, that's a sign the monitoring surface itself needs pruning rather than more toggles.

References

Google SRE Book — Monitoring Distributed Systems: https://sre.google/sre-book/monitoring-distributed-systems/
Google SRE Book — Addressing Cascading Failures: https://sre.google/sre-book/addressing-cascading-failures/
AWS Builders' Library — Timeouts, retries, and backoff with jitter: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
Martin Fowler — Feature Toggles (aka Feature Flags): https://martinfowler.com/articles/feature-toggles.html