Who Watches Your Watchdog? (And Why It Probably Isn't You)

Been thinking about this after reading about someone who built a cron-job monitoring service and had it silently stop running. The watchdog stopped watching. No alerts, no notifications. Just quiet failure.

It's one of those things that sounds obvious in hindsight but catches a lot of people.

The setup

You've got some jobs running on a schedule. Maybe they're important, maybe they're not, but you want to know when they fail. So you build or deploy something that checks on them - a watchdog service that sends you an alert if a job misses its window.

This works fine. Until the watchdog itself stops working.

What went wrong

The person running PulseWatch (a Python-based cron watchdog SaaS) had it running on GitHub Actions as a scheduler. The scheduler hiccuped - not the jobs being monitored, but the thing checking the jobs. The watchdog just... stopped.

Here's the thing: if your watchdog dies, you don't get notified. That's the whole point of having a watchdog, and it's also the failure mode.

The fix

The solution was to chain the watchdog to an external probe - specifically, a Healthchecks.io endpoint. The watchdog pings Healthchecks.io on a schedule, and if Healthchecks.io doesn't hear from it, that triggers an alert.

Now you have two layers:

• Your watchdog monitors your jobs
• Healthchecks.io monitors your watchdog

The failure domains are separated. If your watchdog dies, an external service notices.

The grace period trap

There's another lesson here about how grace periods work. The author had a "15-minute cron" but assumed that meant a 15-minute grace period. It doesn't.

A cron expression of */15 * * * * means "run every 15 minutes" but doesn't guarantee exactly every 15 minutes. GitHub Actions might run it at :00, :15, :30, :45 - or it might be :02, :17, :32, :47. The scheduler has its own timing.

If your grace period is tied to your cron interval, you're going to get false alerts when the scheduler drifts. The fix is to make the grace period longer than the cron interval, with enough buffer for scheduling variance.

What to check

If you're running any kind of monitoring or alerting system:

1. How do you know it's still running? Is there an external check on the check?
2. What's your grace period? Is it longer than your check interval plus scheduling variance?
3. Where does the alert go? If your primary alerting channel is down, is there a fallback?

The boring truth is that monitoring systems fail too. The question is whether you'll notice.

---

Original research from Saijo. Source: https://dev.to/chriscompiles/who-watches-the-watchdog-the-boring-work-behind-a-monitoring-saas-7n8