Who watches the watchdog? The boring work behind a monitoring SaaS

My last post about PulseWatch ended with a satisfying discovery: the monitoring bug I was investigating wasn’t actually a bug.

A GitHub Actions workflow configured to run every 15 minutes was sometimes going missing for well over an hour. PulseWatch noticed the absence, raised a MISSING alert and reported recovery when GitHub eventually ran the workflow again.

The smoke alarm wasn’t broken. There really was smoke.

That was useful validation for the product. It also exposed several less comfortable questions about the product itself.

Who watches the PulseWatch watchdog?

How should users configure grace periods when schedulers are unreliable?

And if someone needs help, can they actually contact me?

So, for the past couple of weeks, I haven’t been building a clever new feature. I’ve been working through the unglamorous jobs that make a monitoring product less fragile.

1. The watchdog now has its own watchdog

PulseWatch has a separate scheduled service that checks every monitor and decides whether it is OK, MISSING, FAILED or STUCK.

That separation is fundamental to how the product works. A monitored script cannot report that it never started, so something outside the script has to notice its absence.

But there was an awkward flaw in my architecture:

If the PulseWatch watchdog stopped running, nobody would receive an alert — including me.

The web application could still be online. Jobs could continue sending pings. Run history could continue accumulating in the database.

But the process responsible for evaluating those runs and sending alerts could be dead.

For a monitoring product, that’s a fairly serious blind spot.

I’ve now instrumented the watchdog using Healthchecks.io, which runs outside the PulseWatch infrastructure.

Each watchdog cycle sends:

• a start signal when checking begins
• a success signal after the full cycle completes
• a failure signal if the cycle exits with an unhandled exception

Healthchecks also knows how frequently the watchdog should run, so it can raise an alert if the process never starts at all.

The integration is deliberately best-effort and non-blocking. If Healthchecks is unavailable, that must not stop PulseWatch from checking its users’ monitors. Monitoring the monitor should never break the thing being monitored.

It creates a simple chain:

User's job → PulseWatch watchdog → external watchdog

Nothing is perfectly self-monitoring, but the failure domains are now separated. PulseWatch is no longer responsible for noticing the death of its own alerting process.

2. The scheduler incident became documentation

The GitHub Actions incident also showed me that PulseWatch’s configuration needed much better explanation.

A monitor has three important time settings:

• Expected interval: how often the job should complete
• Grace: additional time allowed before a missing run triggers an alert
• Max runtime: how long an active run may remain unfinished before it is considered stuck

These sound obvious until a scheduler configured for 15-minute intervals produces gaps of 90 minutes or more.

A 15-minute schedule does not mean a 15-minute grace period is safe.

GitHub explicitly documents that scheduled workflows can be delayed during periods of high load and that, under sufficiently high load, some queued jobs may be dropped.

The right grace period therefore depends on the behaviour you actually observe, not just the cron expression you wrote.

That lesson is now captured in a new public PulseWatch documentation page.

It includes:

• a 60-second getting-started guide
• the /start → /success or /fail lifecycle
• dependency-free Python integration
• a Bash/curl example
• expected interval, grace and max-runtime guidance
• a concrete warning about scheduler jitter
• definitions of MISSING, STUCK, FAILED, flapping and NEVER_RUN

I also made the examples defensive.

Monitoring requests use short timeouts and do not replace the original job’s error. The Bash example preserves the job’s actual exit code. Failure messages are URL-encoded. The example ping URLs are deliberately fake so nobody accidentally publishes a real monitor token.

The page is linked from the main navigation and footer, and the public route now has its own focused test.

None of this is particularly exciting engineering. It may be more valuable than another feature.

A monitoring tool that is configured incorrectly can behave exactly as designed and still be useless.

3. “Priority support” now has an actual support channel

PulseWatch’s paid Indie tier listed priority support.

Until recently, there was no obvious way to request it.

That is the sort of gap that survives when you’re concentrating on the application itself: the pricing page makes a promise, but the operational plumbing behind the promise doesn’t exist.

There is now a live support link and a working domain email address at [e-mail].

I tested the complete loop rather than stopping once outgoing mail worked:

1. Send from the PulseWatch address
2. Receive it in an external inbox
3. Reply to it
4. Confirm the reply arrives in the correct PulseWatch inbox

Again, boring work. But if someone is deciding whether to trust a tiny monitoring service run by one person, “can I reach that person when something breaks?” is not a minor detail.

The pattern I keep finding

The first version of PulseWatch was mostly about capability:

• Can it receive pings?
• Can it detect a missing job?
• Can it send an alert?
• Can it show run history?

The work after launch has increasingly been about trust:

• Will its own watchdog failure be noticed?
• Can a user configure it without creating false alerts?
• Are the examples safe to copy?
• Is there a real person behind the support link?

Those jobs don’t produce impressive launch screenshots. They don’t make the feature list much longer. They do make the product less likely to fail in an embarrassing way.

There are still gaps.

Automated database backups, including proving that a restore actually works, are next. Terms of Service and Privacy Policy pages also need shipping. More ambitious work, including detecting abnormal behaviour in unattended and AI-driven workflows, comes after that.

For now, I think a monitoring product has to earn the right to become clever by first becoming dependable.

If you were deciding whether to trust a tiny monitoring service with your unattended jobs, what evidence would you want to see next?