Two Completely Different Failures, One Monitoring Blind Spot

Most monitoring setups I've seen treat "is my service healthy" as a single question. It isn't. There are at least two distinct failure modes that look nothing alike, require different detection tools, and cause different kinds of damage if you miss them. Mixing them up — or worse, picking one and assuming it covers both — is how you end up with a cron job that silently stopped running three weeks ago and nobody noticed.

The Loud Failure vs. The Silent One

An HTTP uptime monitor does one thing well: it pokes your endpoint and checks whether it responds. If your checkout health route returns a 500, your monitor fires, your phone buzzes, and someone fixes it. That's a loud failure. It announces itself. The monitor just has to be listening.

A silent failure works differently. Your cron job doesn't crash. It doesn't return an error. It simply never runs. Maybe the scheduler got into a bad state. Maybe a deploy wiped an environment variable the job depends on. Maybe a background worker queue backed up and the job has been sitting there for six hours waiting for a slot. From the outside, nothing looks broken. Your uptime monitor is perfectly happy because there's nothing to ping.

This is the gap. A dead-man switch (sometimes called a heartbeat monitor) inverts the detection model. Instead of your monitoring system reaching out to your service, your job reaches out to the monitoring system on every successful run. If the check-in stops arriving, that silence is the alert. You configure an expected interval, and if that interval passes without a ping, you get paged.

These two tools are not redundant. They cover orthogonal failure modes.

Why This Gets Complicated in Multi-Tenant SaaS

The stakes go up when you're running tenant cohort experiments or rolling deployments across multiple customer regions. Now you don't just need to know whether something failed. You need to know where and for whom it failed before you decide whether to roll back.

Imagine a nightly billing job that runs per cohort. You deploy a change and the job starts running more slowly for cohort B but completes fine for everyone else. An uptime monitor on your billing health endpoint won't catch this at all. A single global dead-man switch might still get its check-in because the job technically finished. But if you have a dead-man switch per job per cohort, plus custom metrics tracking job duration and record count per cohort, you now have two independent signals that can disagree in useful ways.

A missed heartbeat is a fire. Page someone now. A job that checked in but took 4x longer than usual in one cohort is a warning. It's probably the rollback evidence you need, but it's not the thing that wakes someone up at 2am.

Keeping those two concerns separate in your alerting design matters a lot.

A Practical Setup That Actually Works

For a small SaaS with customers in multiple regions, a reasonable setup looks like this:

One external HTTP probe per customer region pointed at your health endpoint. Not a synthetic check from inside your own infrastructure. External, so it catches network-level and infrastructure-level failures you'd otherwise be blind to.

One dead-man switch per cron job per cohort. The job pings the switch as its last step when it completes successfully. If the switch doesn't hear from the job within the expected window plus a reasonable grace period, it pages.

import httpx

def run_billing_job(cohort_id: str):
    # ... do the actual work ...
    process_cohort_billing(cohort_id)

    # Dead-man check-in at the end, only on success
    heartbeat_url = f"https://your-monitor.example.com/ping/{cohort_id}"
    httpx.get(heartbeat_url, timeout=5)

Custom metrics (job duration, records processed, error rate per cohort) sit alongside this but serve a different purpose. They're your rollback decision support, not your primary alert trigger. When the dead-man switch fires and you're deciding whether to roll back versus hotfix, the metrics tell you whether this is widespread or isolated, gradual or sudden.

The paging path stays simple: missed heartbeat pages on-call, 500 on health endpoint pages on-call. Everything else feeds a dashboard that a human looks at after they've already been woken up.

What This Actually Protects You From

A rollback is a high-stakes decision made under pressure. The worst version of that situation is rolling back based on incomplete information, finding out the rollback made things worse, and now you've doubled your incident duration.

The monitoring setup above gives you two independent confidence signals before you act. If both the heartbeat and the error metrics look bad in the same cohort after a deploy, that's a clear rollback signal. If the heartbeat fired but the metrics look normal, you might have a scheduler problem unrelated to the deploy. If the metrics look bad but the heartbeat is fine, you have a partial failure worth investigating before pulling the trigger on a rollback.

The concrete takeaway: pick your monitoring tools based on failure mode, not convenience. Uptime monitors and dead-man switches solve different problems. Running both, scoped to the right granularity (per region, per cohort, per job), is what gives you the signal quality you actually need when something goes wrong.