My last post about PulseWatch ended with a satisfying discovery: the monitoring bug I was investigating wasn’t actually a bug.
A GitHub Actions workflow configured to run every 15 minutes was sometimes going missing for well over an hour. PulseWatch noticed the absence, raised a MISSING alert and reported recovery when GitHub eventually ran the workflow again.
The smoke alarm wasn’t broken. There really was smoke.
That was useful validation for the product. It also exposed several less comfortable questions about the product itself.
Who watches the PulseWatch watchdog?
How should users configure grace periods when schedulers are unreliable?
And if someone needs help, can they actually contact me?
So, for the past couple of weeks, I haven’t been building a clever new feature. I’ve been working through the unglamorous jobs that make a monitoring product less fragile.
1. The watchdog now has its own watchdog
PulseWatch has a separate scheduled service that checks every monitor and decides whether it is OK, MISSING, FAILED or STUCK.
That separation is fundamental to how the product works. A monitored script cannot report that it never started, so something outside the script has to notice its absence.
But there was an awkward flaw in my architecture:
If the PulseWatch watchdog stopped running, nobody would receive an alert — including me.
The web application could still be online. Jobs could continue sending pings. Run history could continue accumulating in the database.
But the process responsible for evaluating those runs and sending alerts could be dead.
For a monitoring product, that’s a fairly serious blind spot.
I’ve now instrumented the watchdog using Healthchecks.io, which runs outside the PulseWatch infrastructure.
Each watchdog cycle sends:
- a start signal when checking begins
- a success signal after the full cycle completes
- a failure signal if the cycle exits with an unhandled exception
Healthchecks also knows how frequently the watchdog should run, so it can raise an alert if the process never starts at all.
The integration is deliberately best-effort and non-blocking. If Healthchecks is unavailable, that must not stop PulseWatch from checking its users’ monitors. Monitoring the monitor should never break the thing being monitored.
It creates a simple chain:
User's job → PulseWatch watchdog → external watchdog
Nothing is perfectly self-monitoring, but the failure domains are now separated. PulseWatch is no longer responsible for noticing the death of its own alerting process.
2. The scheduler incident became documentation
The GitHub Actions incident also showed me that PulseWatch’s configuration needed much better explanation.
A monitor has three important time settings:
- Expected interval: how often the job should complete
- Grace: additional time allowed before a missing run triggers an alert
- Max runtime: how long an active run may remain unfinished before it is considered stuck
These sound obvious until a scheduler configured for 15-minute intervals produces gaps of 90 minutes or more.
A 15-minute schedule does not mean a 15-minute grace period is safe.
GitHub explicitly documents that scheduled workflows can be delayed during periods of high load and that, under sufficiently high load, some queued jobs may be dropped.
The right grace period therefore depends on the behaviour you actually observe, not just the cron expression you wrote.
That lesson is now captured in a new public PulseWatch documentation page.
It includes:
- a 60-second getting-started guide
- the /start → /success or /fail lifecycle
- dependency-free Python integration
- a Bash/curl example
- expected interval, grace and max-runtime guidance
- a concrete warning about scheduler jitter
- definitions of MISSING, STUCK, FAILED, flapping and NEVER_RUN
I also made the examples defensive.
Monitoring requests use short timeouts and do not replace the original job’s error. The Bash example preserves the job’s actual exit code. Failure messages are URL-encoded. The example ping URLs are deliberately fake so nobody accidentally publishes a real monitor token.
The page is linked from the main navigation and footer, and the public route now has its own focused test.
None of this is particularly exciting engineering. It may be more valuable than another feature.
A monitoring tool that is configured incorrectly can behave exactly as designed and still be useless.
3. “Priority support” now has an actual support channel
PulseWatch’s paid Indie tier listed priority support.
Until recently, there was no obvious way to request it.
That is the sort of gap that survives when you’re concentrating on the application itself: the pricing page makes a promise, but the operational plumbing behind the promise doesn’t exist.
There is now a live support link and a working domain email address at chris@pulsewatch.ai.
I tested the complete loop rather than stopping once outgoing mail worked:
- Send from the PulseWatch address
- Receive it in an external inbox
- Reply to it
- Confirm the reply arrives in the correct PulseWatch inbox
Again, boring work. But if someone is deciding whether to trust a tiny monitoring service run by one person, “can I reach that person when something breaks?” is not a minor detail.
The pattern I keep finding
The first version of PulseWatch was mostly about capability:
- Can it receive pings?
- Can it detect a missing job?
- Can it send an alert?
- Can it show run history?
The work after launch has increasingly been about trust:
- Will its own watchdog failure be noticed?
- Can a user configure it without creating false alerts?
- Are the examples safe to copy?
- Is there a real person behind the support link?
Those jobs don’t produce impressive launch screenshots. They don’t make the feature list much longer. They do make the product less likely to fail in an embarrassing way.
There are still gaps.
Automated database backups, including proving that a restore actually works, are next. Terms of Service and Privacy Policy pages also need shipping. More ambitious work, including detecting abnormal behaviour in unattended and AI-driven workflows, comes after that.
For now, I think a monitoring product has to earn the right to become clever by first becoming dependable.
If you were deciding whether to trust a tiny monitoring service with your unattended jobs, what evidence would you want to see next?
Top comments (5)
Solid Python walkthrough. In production, I've found that combining this with structured logging (using structlog or the standard logging module with formatters) makes debugging distributed systems much more tractable.
Thanks! That feels like the right pairing: the external monitor tells you something needs attention; structured logs help explain what happened inside the runs that actually executed.
A consistent monitor ID/run ID, outcome and duration would make tracing that path much easier. Useful suggestion — particularly as the workflows get more involved.
Exactly, tying the external alert to the internal execution context is where the real debugging magic happens. Injecting that run ID into every log line from the start saves so much pain when you are trying to correlate a failed cron job with a specific database timeout. Have you found any specific logging libraries that make propagating that context automatically without too much boilerplate?
Your own rule, that a script cannot report it never started, has a second edge the external watchdog does not cover. The success ping is emitted by the watchdog after the cycle completes, so a cycle that runs on schedule and evaluates zero monitors (empty result set, a filter that stopped matching, a migration that renamed a status) still pings success, and Healthchecks reports a healthy chain because it is checking for presence of the cycle rather than presence of work. That one is also more durable than process death, since it survives deploys and restarts instead of being cleared by them. The cheapest separation with what you already have is to put the number of monitors evaluated into the success ping body and alert when it is zero or below the row count you can read straight from the database, so the signal is conditioned on work done rather than on the cycle returning.
That’s a really useful distinction. The external watchdog closes the “checking process stopped running” gap, but a successful cycle isn’t proof that it evaluated everything it should have. Your empty-result-set example makes that very concrete.
One wrinkle with PulseWatch: the cron runs every five minutes, but monitors are evaluated at different intervals depending on the plan. So zero evaluated can be legitimate, and comparing against the total database row count would generate false alarms.
I think the stronger check is whether monitors that should have been evaluated are being left overdue, with that expectation established independently of the selection logic being checked. Otherwise the same broken filter could report both “zero expected” and “zero processed” and still look healthy.
Including counts in the ping would be useful diagnostic context, although the comparison and failure signal would need explicit logic too — putting a count in the body alone wouldn’t trigger an alert.
Fair challenge to the post: I’ve improved detection of process failure, not yet proved that a healthy process means healthy monitoring coverage.