DEV Community: Chris Healey

A month of PulseWatch: billing, better setup and deliberately broken scripts

Chris Healey — Wed, 07 Oct 2026 09:26:05 +0000

On 1 September, I wrote Who watches the watchdog? about the less glamorous work behind PulseWatch: monitoring its own watchdog, explaining scheduler delays and making sure the support link reached a real person.

I’ve been quiet here since then. The project has been rather less quiet.

Over the past month, I’ve worked on the interface, billing, terms and privacy pages, and a simpler way to connect an existing script. Yesterday, I deliberately failed, stalled and stopped disposable jobs to check that the alerts actually reached my inbox.

That last part was particularly satisfying. A monitoring product ought to be able to demonstrate what happens when something goes wrong.

For anyone new to the project: PulseWatch monitors unattended scripts and scheduled jobs from outside the system running them. Your job sends a /start, then a /success or /fail. A separate watchdog notices when expected work goes missing or a started run takes too long.

The original motivation still holds: a script that never starts cannot report its own absence.

Making the interface useful when something breaks

One of the bigger September changes was bringing the landing page, dashboard, monitor details, documentation and admin screens into a consistent interface.

The useful part was deciding what belonged at the top.

When an account already has monitors, their health should come before the form for creating another one. When a job fails, the monitor page should show the current problem and recent run evidence before asking you to think about configuration.

That’s obvious when written down. It wasn’t how every screen was arranged.

Private ping URLs are also masked by default, with reveal and copy controls. They’re credentials: anyone holding one can send signals to that monitor.

The same pass included browser-form security work, server-side validation and checks at narrow mobile widths. Long error messages and tokens are very good at finding the weak spots in a layout.

It was a useful reminder that a UI refresh can improve how someone investigates a failure, as well as how the product looks in a screenshot.

Billing became a proper piece of application logic

The early payment setup involved Stripe links and manually changing someone’s plan.

That was manageable at a tiny scale, but it left too much dependent on me matching a payment to an account and remembering what should happen next.

The newer flow links Checkout to the signed-in PulseWatch account. Confirmed payment is what grants paid access; returning to a success page is not enough.

I’ve also worked through renewal handling and made the purchase pages clearer about the monthly price, tax treatment, when paid access begins and how to stop renewal. Terms, privacy and support pages are now published.

In the sandbox, I verified an initial purchase and a successful renewal, including the extension of the account’s paid-through date.

There is still billing verification to finish. The failed-renewal simulation hit a restriction in the Managed Payments setup, so that particular lifecycle test remains outstanding. Getting one successful payment through doesn’t prove every subscription edge case.

I also investigated what initially looked like a payment bug. An account with a manually assigned plan was being stopped from starting a conflicting purchase, exactly as intended.

The awkward part was the explanation. I improved the wording so the user could understand why they’d been stopped.

A fairly representative solo-builder afternoon: investigate a possible bug, find the guard working, then discover that the message still needs fixing.

A free tool to get the first script connected

The newest addition is Monitor my script, a free Python/Bash wrapper generator.

It creates the start and outcome handling around an existing job. The Python version uses the standard library; the Bash version runs your command and preserves its exit status.

The private ping URL comes from an environment variable.

The important constraint is that monitoring should preserve the job’s result. If your script raises an exception, a failed monitoring request must not replace it. If your command exits with an error, the wrapper must return that error.

The generated pings have five-second timeouts and guarded failure handling.

This came from thinking about the first integration. Someone arriving with a working script shouldn’t have to piece together several examples before they can try monitoring it.

I’ve also published a Python and Bash setup guide, covering the wrapper, schedule settings, the first successful run and checking failure and recovery.

Writing those steps down is its own product review. It exposes every place where you’ve assumed the reader already understands something.

Then I tried breaking it

Yesterday’s checks used disposable monitors and jobs to exercise the generated code against the live service.

Test	What I checked
An ordinary Python exception or a Bash command exiting with code 7	Failure was recorded, the original job outcome was preserved, and failure and recovery emails arrived.
A started job left unfinished beyond its maximum runtime	STUCK was detected, followed by recovery and both emails.
A job stopped after establishing a successful baseline	MISSING was detected, followed by recovery and both emails.

I checked the dashboard and the actual iCloud inbox. An email provider accepting a request is useful evidence, but seeing the intended message arrive completes another part of the chain.

The missing-run test needed its initial schedule setup correcting before it exercised the intended scenario. Worth recording: the test setup needs scrutiny too.

Those checks didn’t introduce new failure states. They verified existing behaviour through the new integration path.

They also reinforced two details the guide now makes explicit: finish a successful run to establish the baseline, and expect alerts on the next applicable watchdog check. The check cadence depends on the plan.

What does “success” actually mean?

Feedback here has also pushed me to think harder about a job that runs, finishes and produces nothing useful.

An export can exit cleanly while returning zero records. A report can complete without generating the file anyone needed.

PulseWatch receives the outcome your integration reports. It doesn’t inspect arbitrary business output and decide whether the work was correct.

I’ve extended the documentation with an example of checking useful output before sending /success. For a daily export where an empty result is a failure, a small local rule can raise an exception and take the existing /fail path.

The rule has to fit the job. Zero records might be perfectly valid for an import that only processes new entries.

That seems a useful next step to explore with real users: help them define a meaningful success condition, then make the signal reflect it.

Keeping the priorities proportionate

One correction to the previous post: I had put automated database backups next on the list. After reviewing the paid Render Postgres recovery provision, I accepted its existing point-in-time recovery as proportionate for the current stage and deferred additional backup work.

I haven’t completed a separate restore drill. That distinction matters.

There is still plenty to do, including finishing billing verification and getting more consequential workflows connected.

The engineering progress is tangible. The commercial question still needs evidence from people who keep using it.

For now, there’s a clearer interface, a more complete buying path, and a shorter route from an existing script to a verified first run. You can try it with two free monitors and no credit card.

If you maintain scheduled jobs, have you ever had one finish “successfully” and still leave you with nothing useful?

Who watches the watchdog? The boring work behind a monitoring SaaS

Chris Healey — Tue, 01 Sep 2026 10:36:48 +0000

My last post about PulseWatch ended with a satisfying discovery: the monitoring bug I was investigating wasn’t actually a bug.

A GitHub Actions workflow configured to run every 15 minutes was sometimes going missing for well over an hour. PulseWatch noticed the absence, raised a MISSING alert and reported recovery when GitHub eventually ran the workflow again.

The smoke alarm wasn’t broken. There really was smoke.

That was useful validation for the product. It also exposed several less comfortable questions about the product itself.

Who watches the PulseWatch watchdog?

How should users configure grace periods when schedulers are unreliable?

And if someone needs help, can they actually contact me?

So, for the past couple of weeks, I haven’t been building a clever new feature. I’ve been working through the unglamorous jobs that make a monitoring product less fragile.

1. The watchdog now has its own watchdog

PulseWatch has a separate scheduled service that checks every monitor and decides whether it is OK, MISSING, FAILED or STUCK.

That separation is fundamental to how the product works. A monitored script cannot report that it never started, so something outside the script has to notice its absence.

But there was an awkward flaw in my architecture:

If the PulseWatch watchdog stopped running, nobody would receive an alert — including me.

The web application could still be online. Jobs could continue sending pings. Run history could continue accumulating in the database.

But the process responsible for evaluating those runs and sending alerts could be dead.

For a monitoring product, that’s a fairly serious blind spot.

I’ve now instrumented the watchdog using Healthchecks.io, which runs outside the PulseWatch infrastructure.

Each watchdog cycle sends:

a start signal when checking begins
a success signal after the full cycle completes
a failure signal if the cycle exits with an unhandled exception

Healthchecks also knows how frequently the watchdog should run, so it can raise an alert if the process never starts at all.

The integration is deliberately best-effort and non-blocking. If Healthchecks is unavailable, that must not stop PulseWatch from checking its users’ monitors. Monitoring the monitor should never break the thing being monitored.

It creates a simple chain:

User's job → PulseWatch watchdog → external watchdog

Nothing is perfectly self-monitoring, but the failure domains are now separated. PulseWatch is no longer responsible for noticing the death of its own alerting process.

2. The scheduler incident became documentation

The GitHub Actions incident also showed me that PulseWatch’s configuration needed much better explanation.

A monitor has three important time settings:

Expected interval: how often the job should complete
Grace: additional time allowed before a missing run triggers an alert
Max runtime: how long an active run may remain unfinished before it is considered stuck

These sound obvious until a scheduler configured for 15-minute intervals produces gaps of 90 minutes or more.

A 15-minute schedule does not mean a 15-minute grace period is safe.

GitHub explicitly documents that scheduled workflows can be delayed during periods of high load and that, under sufficiently high load, some queued jobs may be dropped.

The right grace period therefore depends on the behaviour you actually observe, not just the cron expression you wrote.

That lesson is now captured in a new public PulseWatch documentation page.

It includes:

a 60-second getting-started guide
the /start → /success or /fail lifecycle
dependency-free Python integration
a Bash/curl example
expected interval, grace and max-runtime guidance
a concrete warning about scheduler jitter
definitions of MISSING, STUCK, FAILED, flapping and NEVER_RUN

I also made the examples defensive.

Monitoring requests use short timeouts and do not replace the original job’s error. The Bash example preserves the job’s actual exit code. Failure messages are URL-encoded. The example ping URLs are deliberately fake so nobody accidentally publishes a real monitor token.

The page is linked from the main navigation and footer, and the public route now has its own focused test.

None of this is particularly exciting engineering. It may be more valuable than another feature.

A monitoring tool that is configured incorrectly can behave exactly as designed and still be useless.

3. “Priority support” now has an actual support channel

PulseWatch’s paid Indie tier listed priority support.

Until recently, there was no obvious way to request it.

That is the sort of gap that survives when you’re concentrating on the application itself: the pricing page makes a promise, but the operational plumbing behind the promise doesn’t exist.

There is now a live support link and a working domain email address at chris@pulsewatch.ai.

I tested the complete loop rather than stopping once outgoing mail worked:

Send from the PulseWatch address
Receive it in an external inbox
Reply to it
Confirm the reply arrives in the correct PulseWatch inbox

Again, boring work. But if someone is deciding whether to trust a tiny monitoring service run by one person, “can I reach that person when something breaks?” is not a minor detail.

The pattern I keep finding

The first version of PulseWatch was mostly about capability:

Can it receive pings?
Can it detect a missing job?
Can it send an alert?
Can it show run history?

The work after launch has increasingly been about trust:

Will its own watchdog failure be noticed?
Can a user configure it without creating false alerts?
Are the examples safe to copy?
Is there a real person behind the support link?

Those jobs don’t produce impressive launch screenshots. They don’t make the feature list much longer. They do make the product less likely to fail in an embarrassing way.

There are still gaps.

Automated database backups, including proving that a restore actually works, are next. Terms of Service and Privacy Policy pages also need shipping. More ambitious work, including detecting abnormal behaviour in unattended and AI-driven workflows, comes after that.

For now, I think a monitoring product has to earn the right to become clever by first becoming dependable.

If you were deciding whether to trust a tiny monitoring service with your unattended jobs, what evidence would you want to see next?

The monitoring bug that turned out not to be a bug

Chris Healey — Thu, 13 Aug 2026 15:21:41 +0000

I spent the last couple of days debugging what looked like a fairly nasty problem in PulseWatch, the dead-man’s-switch monitoring tool I’ve been building.

My inbox was filling up with alerts:

MISSING → OK → MISSING → OK

Overnight, I received 29 of them.

Not ideal behaviour for a product whose job is to reduce monitoring noise.

What I thought was happening

PulseWatch monitors unattended jobs from the outside.

A job sends a simple HTTP request when it starts and another when it completes. PulseWatch stores those runs and a server-side watchdog checks whether a successful run has arrived within the expected interval plus a configurable grace period.

If not, the monitor becomes MISSING.

When a new successful run arrives, it becomes OK again.

The canary I use to test PulseWatch is itself a scheduled GitHub Actions workflow. It is configured to run every 15 minutes. At first I assumed my alert state logic was broken.

I increased the PulseWatch threshold to an hour plus a 30-minute grace period but the flip-flopping continued.

Typical alerts looked like this:

05:10 — MISSING — last success 1h34m ago
05:30 — OK — new successful run
07:00 — MISSING — last success 1h32m ago
07:05 — OK — new successful run
08:35 — MISSING — last success 1h33m ago
08:50 — OK — new successful run

That looked to me to be suspiciously precise.

Following the state transitions

So, what did I do? I got Codex to work tracing the relevant code.

PulseWatch declares a monitor missing when:

now - latest_success > expected_interval + grace_period
With the monitor configured for 60 minutes plus 30 minutes of grace, that gives a 90-minute boundary, so those MISSING alerts were correct.

More importantly, the watchdog cannot recover a monitor by itself. An OK recovery requires PulseWatch to receive a new /success request.

So the state machine wasn't oscillating, something really was sending successful pings after each MISSING alert and that moved the investigation upstream.

Then I looked at GitHub Actions

The canary workflow is configured to run every 15 minutes.

Its actual run history looked more like this:

00:05
00:53
03:35
05:27
07:00
08:45
09:55
11:16
12:12
13:01
14:36

One gap was 2 hours 42 minutes.

Another was 1 hour 52 minutes.

Another was 1 hour 45 minutes.

The interesting part was that the executions GitHub did run were green. The canary itself generally completed in around 10–20 seconds.

There weren't corresponding failed 15-minute runs filling those gaps. The scheduled executions simply weren't happening reliably.

GitHub documents that scheduled Actions can be delayed and, under sufficiently high load, some queued jobs may be dropped.

The smoke alarm wasn't broken

This completely changed my interpretation of the incident. PulseWatch wasn't generating false positives, it was detecting a genuine failure of the system it was monitoring.

Take the 07:00 example; the previous GitHub Actions execution had occurred at approximately 05:27. Nothing successfully ran for the next 90 minutes and so PulseWatch crossed its configured threshold and raised MISSING.

GitHub eventually executed the workflow again at 07:00 which is when the canary sent its success ping...PulseWatch reported recovery exactly as designed.

The more interesting monitoring lesson

This highlighted a failure mode that's easy to overlook, that is logs are excellent at telling you what happened inside a process that ran.

GitHub Actions was similarly quite capable of showing me green ticks for the workflows it executed successfully.

Neither helps much with the more awkward question:

What about the job that never ran at all?

There is no exception. There is no failed execution. There may be no application log. Nothing inside the job can report the failure because the job never started.

That's precisely where an external dead-man's switch becomes useful.

It doesn't ask:

"Did the job report an error?"

It asks:

"Did I hear from the job when I expected to?"

Those turn out to be quite different questions.

I thought I’d found a bug in PulseWatch. Instead, I’d found the exact failure PulseWatch was built to catch.

Two Bugs, Two Strangers, One Week: What Shipping Early Actually Buys You

Chris Healey — Fri, 17 Jul 2026 09:29:10 +0000

A week ago I put a rough, honestly-a-bit-thin version of PulseWatch in front of real people for the first time. Within days, two different strangers — independently, unprompted — found two real gaps in it. Neither was catastrophic. Both were exactly the kind of thing you only find by watching someone else use the thing you built.

This is the story of both, and the fixes.

Bug one: the run that never ends

This first bug came from a friend testing it on a real script. His question was simple:

"What happens if start fires twice before end?"

Good question. At the time: nothing good. Here's why.

PulseWatch works on two pings — a job calls /start when it begins and /success (or /fail) when it's done. The server tracks whichever run is currently "open" for a monitor. The bug: if a job's process restarts mid-run — a crash-and-retry, a redeploy that catches it mid-flight, a scheduler firing twice — you get a second /start before the first run ever closes. The old run just sits there, open forever, an orphan with no ending. Worse, because the watchdog was still waiting on that run's expected finish time, it could fire a false "still running" alert for a run that was, for all practical purposes, dead and abandoned.

The fix is a small rule with an outsized effect: a new /start supersedes whatever run is currently open. The old run gets marked superseded — a terminal, non-alerting status — and a fresh run begins clean. The watchdog was updated to treat superseded as a dead end: nothing to wait on, nothing to alert about, and it never shows up in a user's run history. It's not a failure and it's not a success. It's just "this run doesn't matter anymore, a newer one replaced it."

The logic, roughly:

def handle_start(monitor):
    open_run = monitor.get_open_run()
    if open_run is not None:
        open_run.status = "superseded"
        open_run.finished_at = now()

    new_run = Run(monitor=monitor, status="running", started_at=now())
    db.session.add(new_run)
    db.session.commit()

Simple once you see it. Invisible until someone actually restarts a job mid-flight and asks the right question.

Bug two: the failure that isn't a failure, exactly

The second came from a comment on my first post here on Dev.to, from someone whose day job is infrastructure monitoring. He'd read the piece on catching jobs that die silently and asked the harder follow-up:

"How does PulseWatch deal with jobs that aren't dead, just flaky, before alert fatigue kicks in?"

That's a different failure mode entirely, and at the time I only had half an answer.

PulseWatch already alerts on state transitions, not conditions — one email when a monitor goes down, one when it recovers, not a repeat every check cycle. That solves the "job has been down for six hours, stop emailing me about it" problem. What it didn't solve: a job that's actually unstable — fails, recovers, fails, recovers, each transition genuinely real. Alert on every transition, and you get exactly what it sounds like: fail, recover, fail, recover, five emails in twenty minutes, and you mute the whole channel by Thursday. Technically correct alerting, practically useless.

The naive fix — just suppress alerts if a monitor's been unstable recently — has an obvious failure mode of its own: it delays the one alert you actually wanted immediately, buried under a threshold meant for noise. That tradeoff is exactly what I flagged as unsolved in my reply to him at the time.

Here's where I landed. Rather than alerting on every up/down transition, PulseWatch now watches a rolling window and counts transitions within it. Cross a threshold — four or more state changes within two hours, by default — and the monitor enters a distinct flapping state. Two things change once it's flapping:

One alert fires on entry — "this monitor is unstable," not "this monitor is down," a meaningfully different signal.
Ordinary up/down alerts are suppressed while flapping continues, and one more alert fires when it settles back below the flap threshold.

So a genuinely flaky job gets exactly two emails — "it's flapping" and "it's settled" — no matter how many times it actually flipped in between. A job that's just plain down still gets its immediate down alert, because that's a single transition, not a pattern — the threshold only kicks in once instability itself becomes the story.

The state machine, simplified:

FLAP_WINDOW_MINUTES = 120   # rolling window to look back over
FLAP_THRESHOLD = 4          # this many state changes in the window = flapping

def is_currently_flapping(monitor):
    """Count how many times status changed across recent runs.
    Flapping if it crosses the threshold within the window."""
    runs = monitor.recent_terminal_runs(minutes=FLAP_WINDOW_MINUTES)  # oldest → newest
    changes = sum(1 for prev, cur in zip(runs, runs[1:])
                  if prev.status != cur.status)
    return changes >= FLAP_THRESHOLD

def check_monitor(monitor):
    currently = is_currently_flapping(monitor)

    if currently and not monitor.is_flapping:
        monitor.is_flapping = True
        alert(monitor, "unstable — flapping between states")

    elif not currently and monitor.is_flapping:
        monitor.is_flapping = False
        alert(monitor, "stabilised")

    elif not monitor.is_flapping:
        # normal single-transition alerting, unchanged
        handle_normal_transition(monitor)

    db.session.commit()

The one new piece of state this needed — a boolean is_flapping column — became, almost by accident, the first schema change to go through a proper migration pipeline I'd just set up on the live database. Small feature, useful excuse to prove the plumbing works before something bigger needs it.

The actual point

Neither of these bugs was findable by staring at the code longer. The double-start case only shows up when a real process actually restarts mid-run. The flapping case only matters once you've watched enough alert emails pile up to feel the fatigue yourself. Both needed someone else's script, someone else's failure pattern, someone else's patience to ask "but what about—".

That's the actual case for shipping something rough and putting it in front of people early, not the version of the advice that's become a cliché. It's not just "get feedback." It's that some categories of bug are structurally invisible to the person who wrote the code, because the code matches their own mental model of how it'll be used — and the whole value of a stranger is that their mental model is different.

Both fixes are live now. If you're running unattended jobs — cron, scheduled scripts, agent pipelines — and any of this sounds familiar, PulseWatch is free to try, and I'd genuinely like to know what breaks next.

How I Built a Dead-Man's Switch for My AI Trading Pipeline in Python

Chris Healey — Thu, 09 Jul 2026 06:58:39 +0000

The morning I noticed my trading bot hadn't traded in three days.

I run a small trading bot. Every morning before the market opens it pulls signals, weighs them, and decides whether to act. It had been humming along for weeks, so I'd mostly stopped watching it.

Then one morning I opened the dashboard out of idle curiosity and something was off. No trades. Not "it decided not to trade", just crickets. I scrolled back. No trades the day before either. Or the day before that.

Three days. The bot had silently stopped running and I had no idea.

Here's the part that stuck with me: there was no error to find. No red text, no stack trace, no alert in my inbox. The overnight process that was supposed to kick everything off had just… not kicked off. A scheduling hiccup on my machine, most likely. The code was fine. The code simply never ran.

And every single tool I had for catching problems was completely blind to it.

Why everything I had was looking the wrong way

Think about how we normally catch failures. We wrap risky things in try/except. We log errors. We wire up something to shout when an exception gets thrown:

try:
    run_morning_routine()
except Exception as e:
    logging.error("Morning routine failed: %s", e)
    send_alert(e)

This is good practice and you should do it. But notice the assumption baked into every line: it assumes the code ran. The try block has to execute for the except to ever fire. The logger has to be reached for it to log. The alert has to be triggered by a process that is, by definition, alive.

None of that helps you when the process never starts.

The server rebooted and the scheduler didn't come back up.
The cron entry got wiped in a config change.
The disk filled and the job couldn't even launch.
A dependency upstream hung and your job was never invoked.

In every one of these cases there is no exception, because there is no running code to throw one. There's no log line, because nothing got far enough to write it. Your monitoring is sitting there patiently waiting for a signal that will never come, and interpreting the silence as "all good."

That's the trap. Absence of an error is not the same as success. For unattended jobs like cron tasks, scheduled scripts, overnight pipelines, AI agents running on a timer etc. the most dangerous failure is the one that produces no output at all.

Inverting the model: a dead-man's switch

The fix is to stop waiting for bad news and start expecting good news.

A dead-man's switch (the name comes from the pedal that stops a train if the driver goes unresponsive) flips the logic around. Instead of your code reporting when it fails, it reports when it succeeds and something external watches for that report to arrive. If the expected check-in doesn't show up on time, the watcher raises the alarm.

The crucial word is external. The thing doing the watching has to live somewhere your job doesn't. If the watchdog runs on the same machine as the job, then when that machine dies, the watchdog dies with it and a dead watchdog can't tell you anything is wrong. You need something running elsewhere whose entire purpose is to notice an absence.

Concretely, three pieces:

Your job sends a ping when it starts, and another when it finishes.
A server somewhere records those pings and knows roughly when to expect the next one.
If the expected ping doesn't arrive inside a grace window, the server alerts you.

The silence becomes the signal. You just need something that's actually listening for it.

Building it in Python

Let me show the shape of it. This is deliberately minimal. The point is the pattern, not a framework.

The client side (what your job adds)

The job needs to announce itself at the start and confirm at the end. Pure standard library, no new dependencies:

import urllib.request

PING = "https://your-watchdog.example/ping/<your-token>"

def ping(event):
    try:
        urllib.request.urlopen(f"{PING}/{event}", timeout=10)
    except Exception:
        # Monitoring must never crash the job it monitors.
        pass

ping("start")
try:
    run_morning_routine()
    ping("success")
except Exception as e:
    ping("fail")
    raise

Three things worth calling out:

The ping is wrapped so it can never take down the real job. Monitoring that crashes the thing it's monitoring is worse than no monitoring.
start and success bracket the work, so the watcher can tell "never started" apart from "started but never finished" (a hang) apart from "finished cleanly."
It's a plain HTTP GET, so this works from any language. A shell job can do the same with curl:

curl -fsS https://your-watchdog.example/ping/<token>/start
./run_backup.sh && curl -fsS https://your-watchdog.example/ping/<token>/success \
                || curl -fsS https://your-watchdog.example/ping/<token>/fail

The receiving side

On the server, receiving a ping is just recording that it happened and when:

from datetime import datetime, timezone
from flask import Flask

app = Flask(__name__)

@app.route("/ping/<token>/<event>")
def receive_ping(token, event):
    monitor = Monitor.query.filter_by(token=token).first_or_404()
    monitor.last_event = event
    monitor.last_ping_at = datetime.now(timezone.utc)
    if event in ("success", "fail"):
        monitor.last_finished_at = monitor.last_ping_at
    db.session.commit()
    return "", 200

Nothing clever here, and that's the point. The receiver's only job is to remember the last time it heard from you.

The watchdog — the part that actually matters

The interesting logic is the part that runs on its own schedule, independent of any job, and asks: is anyone overdue?

from datetime import datetime, timezone, timedelta

def check_all():
    now = datetime.now(timezone.utc)
    for monitor in Monitor.query.all():
        deadline = monitor.last_ping_at + timedelta(
            minutes=monitor.period_minutes + monitor.grace_minutes
        )
        is_late = now > deadline
        new_status = "DOWN" if is_late else "UP"

        # Only act on a *change* of state.
        if new_status != monitor.status:
            monitor.status = new_status
            db.session.commit()
            if new_status == "DOWN":
                alert(monitor, "Your job hasn't checked in — it may have stopped.")
            else:
                alert(monitor, "Your job is reporting again — recovered.")

Two design decisions here earned their keep, and both are about not being annoying:

1. State-transition alerting. You alert on the change, not on the condition. A job that's been down for six hours should generate exactly one "it's down" email, not one every time the watchdog loops. And exactly one "it's back" email when it recovers. Alert on the edge, not the level. This is the single biggest thing standing between "useful monitoring" and "inbox noise you'll mute within a day."

2. A grace window. Jobs are never perfectly punctual; a run that usually takes ten minutes occasionally takes twenty. The grace_minutes buffer stops those normal wobbles from firing false alarms. A decent rule of thumb is to set the grace to roughly 20–30% of the expected run time, with a sensible floor of a few minutes.

Run check_all() on its own timer - its own cron entry, a scheduled worker, whatever just somewhere separate from the jobs it's watching. That separation is the whole game. The watchdog has to outlive the thing it watches.

From "fix for me" to "thing other people can use"

I built the version above for exactly one job: my trading bot. It worked. The next morning the bot failed to start again, and this time my phone buzzed instead of me finding out three days later by accident.

But once I had it, I started pointing it at other things almost reflexively. A nightly database backup which, it turned out, had quietly failed a week earlier and I'd never noticed. A scraper on a schedule. A content pipeline. Every unattended job I owned had the same blind spot, and I'd just been getting lucky that none of them had bitten me yet.

That's when it stopped feeling like a personal script and started feeling like something other solo devs probably needed too. Especially the growing pile of us running AI agents and LLM pipelines on timers, where "did the agent actually run this morning?" is a real and slightly unnerving question. So I cleaned it up, put a proper dashboard on it, and turned it into a small product called PulseWatch.

The whole idea is what's above: your job sends a start and a finish ping, a server watches for them, and you get one email when something goes quiet and one when it comes back. There's a free tier with no card required if you want to point it at a job and see it work.

The one thing to take away

Even if you never use a tool for this, even if you go build the twenty lines yourself, internalise the core idea, because it changes how you think about reliability:

Your monitoring is only watching for the failures it can see. The failure that produces no output is invisible to everything that waits for output.

Go look at your most important unattended job right now and ask one question: if this silently stopped running tonight, what would tell me?

If the honest answer is "I'd notice eventually, when something downstream breaks" then that's the gap. And the fix is genuinely about twenty lines of code away.