DEV Community: Chris Healey

The monitoring bug that turned out not to be a bug

Chris Healey — Thu, 13 Aug 2026 15:21:41 +0000

I spent the last couple of days debugging what looked like a fairly nasty problem in PulseWatch, the dead-man’s-switch monitoring tool I’ve been building.

My inbox was filling up with alerts:

MISSING → OK → MISSING → OK

Overnight, I received 29 of them.

Not ideal behaviour for a product whose job is to reduce monitoring noise.

What I thought was happening

PulseWatch monitors unattended jobs from the outside.

A job sends a simple HTTP request when it starts and another when it completes. PulseWatch stores those runs and a server-side watchdog checks whether a successful run has arrived within the expected interval plus a configurable grace period.

If not, the monitor becomes MISSING.

When a new successful run arrives, it becomes OK again.

The canary I use to test PulseWatch is itself a scheduled GitHub Actions workflow. It is configured to run every 15 minutes. At first I assumed my alert state logic was broken.

I increased the PulseWatch threshold to an hour plus a 30-minute grace period but the flip-flopping continued.

Typical alerts looked like this:

05:10 — MISSING — last success 1h34m ago
05:30 — OK — new successful run
07:00 — MISSING — last success 1h32m ago
07:05 — OK — new successful run
08:35 — MISSING — last success 1h33m ago
08:50 — OK — new successful run

That looked to me to be suspiciously precise.

Following the state transitions

So, what did I do? I got Codex to work tracing the relevant code.

PulseWatch declares a monitor missing when:

now - latest_success > expected_interval + grace_period
With the monitor configured for 60 minutes plus 30 minutes of grace, that gives a 90-minute boundary, so those MISSING alerts were correct.

More importantly, the watchdog cannot recover a monitor by itself. An OK recovery requires PulseWatch to receive a new /success request.

So the state machine wasn't oscillating, something really was sending successful pings after each MISSING alert and that moved the investigation upstream.

Then I looked at GitHub Actions

The canary workflow is configured to run every 15 minutes.

Its actual run history looked more like this:

00:05
00:53
03:35
05:27
07:00
08:45
09:55
11:16
12:12
13:01
14:36

One gap was 2 hours 42 minutes.

Another was 1 hour 52 minutes.

Another was 1 hour 45 minutes.

The interesting part was that the executions GitHub did run were green. The canary itself generally completed in around 10–20 seconds.

There weren't corresponding failed 15-minute runs filling those gaps. The scheduled executions simply weren't happening reliably.

GitHub documents that scheduled Actions can be delayed and, under sufficiently high load, some queued jobs may be dropped.

The smoke alarm wasn't broken

This completely changed my interpretation of the incident. PulseWatch wasn't generating false positives, it was detecting a genuine failure of the system it was monitoring.

Take the 07:00 example; the previous GitHub Actions execution had occurred at approximately 05:27. Nothing successfully ran for the next 90 minutes and so PulseWatch crossed its configured threshold and raised MISSING.

GitHub eventually executed the workflow again at 07:00 which is when the canary sent its success ping...PulseWatch reported recovery exactly as designed.

The more interesting monitoring lesson

This highlighted a failure mode that's easy to overlook, that is logs are excellent at telling you what happened inside a process that ran.

GitHub Actions was similarly quite capable of showing me green ticks for the workflows it executed successfully.

Neither helps much with the more awkward question:

What about the job that never ran at all?

There is no exception. There is no failed execution. There may be no application log. Nothing inside the job can report the failure because the job never started.

That's precisely where an external dead-man's switch becomes useful.

It doesn't ask:

"Did the job report an error?"

It asks:

"Did I hear from the job when I expected to?"

Those turn out to be quite different questions.

I thought I’d found a bug in PulseWatch. Instead, I’d found the exact failure PulseWatch was built to catch.

Two Bugs, Two Strangers, One Week: What Shipping Early Actually Buys You

Chris Healey — Fri, 17 Jul 2026 09:29:10 +0000

A week ago I put a rough, honestly-a-bit-thin version of PulseWatch in front of real people for the first time. Within days, two different strangers — independently, unprompted — found two real gaps in it. Neither was catastrophic. Both were exactly the kind of thing you only find by watching someone else use the thing you built.

This is the story of both, and the fixes.

Bug one: the run that never ends

This first bug came from a friend testing it on a real script. His question was simple:

"What happens if start fires twice before end?"

Good question. At the time: nothing good. Here's why.

PulseWatch works on two pings — a job calls /start when it begins and /success (or /fail) when it's done. The server tracks whichever run is currently "open" for a monitor. The bug: if a job's process restarts mid-run — a crash-and-retry, a redeploy that catches it mid-flight, a scheduler firing twice — you get a second /start before the first run ever closes. The old run just sits there, open forever, an orphan with no ending. Worse, because the watchdog was still waiting on that run's expected finish time, it could fire a false "still running" alert for a run that was, for all practical purposes, dead and abandoned.

The fix is a small rule with an outsized effect: a new /start supersedes whatever run is currently open. The old run gets marked superseded — a terminal, non-alerting status — and a fresh run begins clean. The watchdog was updated to treat superseded as a dead end: nothing to wait on, nothing to alert about, and it never shows up in a user's run history. It's not a failure and it's not a success. It's just "this run doesn't matter anymore, a newer one replaced it."

The logic, roughly:

def handle_start(monitor):
    open_run = monitor.get_open_run()
    if open_run is not None:
        open_run.status = "superseded"
        open_run.finished_at = now()

    new_run = Run(monitor=monitor, status="running", started_at=now())
    db.session.add(new_run)
    db.session.commit()

Simple once you see it. Invisible until someone actually restarts a job mid-flight and asks the right question.

Bug two: the failure that isn't a failure, exactly

The second came from a comment on my first post here on Dev.to, from someone whose day job is infrastructure monitoring. He'd read the piece on catching jobs that die silently and asked the harder follow-up:

"How does PulseWatch deal with jobs that aren't dead, just flaky, before alert fatigue kicks in?"

That's a different failure mode entirely, and at the time I only had half an answer.

PulseWatch already alerts on state transitions, not conditions — one email when a monitor goes down, one when it recovers, not a repeat every check cycle. That solves the "job has been down for six hours, stop emailing me about it" problem. What it didn't solve: a job that's actually unstable — fails, recovers, fails, recovers, each transition genuinely real. Alert on every transition, and you get exactly what it sounds like: fail, recover, fail, recover, five emails in twenty minutes, and you mute the whole channel by Thursday. Technically correct alerting, practically useless.

The naive fix — just suppress alerts if a monitor's been unstable recently — has an obvious failure mode of its own: it delays the one alert you actually wanted immediately, buried under a threshold meant for noise. That tradeoff is exactly what I flagged as unsolved in my reply to him at the time.

Here's where I landed. Rather than alerting on every up/down transition, PulseWatch now watches a rolling window and counts transitions within it. Cross a threshold — four or more state changes within two hours, by default — and the monitor enters a distinct flapping state. Two things change once it's flapping:

One alert fires on entry — "this monitor is unstable," not "this monitor is down," a meaningfully different signal.
Ordinary up/down alerts are suppressed while flapping continues, and one more alert fires when it settles back below the flap threshold.

So a genuinely flaky job gets exactly two emails — "it's flapping" and "it's settled" — no matter how many times it actually flipped in between. A job that's just plain down still gets its immediate down alert, because that's a single transition, not a pattern — the threshold only kicks in once instability itself becomes the story.

The state machine, simplified:

FLAP_WINDOW_MINUTES = 120   # rolling window to look back over
FLAP_THRESHOLD = 4          # this many state changes in the window = flapping

def is_currently_flapping(monitor):
    """Count how many times status changed across recent runs.
    Flapping if it crosses the threshold within the window."""
    runs = monitor.recent_terminal_runs(minutes=FLAP_WINDOW_MINUTES)  # oldest → newest
    changes = sum(1 for prev, cur in zip(runs, runs[1:])
                  if prev.status != cur.status)
    return changes >= FLAP_THRESHOLD

def check_monitor(monitor):
    currently = is_currently_flapping(monitor)

    if currently and not monitor.is_flapping:
        monitor.is_flapping = True
        alert(monitor, "unstable — flapping between states")

    elif not currently and monitor.is_flapping:
        monitor.is_flapping = False
        alert(monitor, "stabilised")

    elif not monitor.is_flapping:
        # normal single-transition alerting, unchanged
        handle_normal_transition(monitor)

    db.session.commit()

The one new piece of state this needed — a boolean is_flapping column — became, almost by accident, the first schema change to go through a proper migration pipeline I'd just set up on the live database. Small feature, useful excuse to prove the plumbing works before something bigger needs it.

The actual point

Neither of these bugs was findable by staring at the code longer. The double-start case only shows up when a real process actually restarts mid-run. The flapping case only matters once you've watched enough alert emails pile up to feel the fatigue yourself. Both needed someone else's script, someone else's failure pattern, someone else's patience to ask "but what about—".

That's the actual case for shipping something rough and putting it in front of people early, not the version of the advice that's become a cliché. It's not just "get feedback." It's that some categories of bug are structurally invisible to the person who wrote the code, because the code matches their own mental model of how it'll be used — and the whole value of a stranger is that their mental model is different.

Both fixes are live now. If you're running unattended jobs — cron, scheduled scripts, agent pipelines — and any of this sounds familiar, PulseWatch is free to try, and I'd genuinely like to know what breaks next.

How I Built a Dead-Man's Switch for My AI Trading Pipeline in Python

Chris Healey — Thu, 09 Jul 2026 06:58:39 +0000

The morning I noticed my trading bot hadn't traded in three days.

I run a small trading bot. Every morning before the market opens it pulls signals, weighs them, and decides whether to act. It had been humming along for weeks, so I'd mostly stopped watching it.

Then one morning I opened the dashboard out of idle curiosity and something was off. No trades. Not "it decided not to trade", just crickets. I scrolled back. No trades the day before either. Or the day before that.

Three days. The bot had silently stopped running and I had no idea.

Here's the part that stuck with me: there was no error to find. No red text, no stack trace, no alert in my inbox. The overnight process that was supposed to kick everything off had just… not kicked off. A scheduling hiccup on my machine, most likely. The code was fine. The code simply never ran.

And every single tool I had for catching problems was completely blind to it.

Why everything I had was looking the wrong way

Think about how we normally catch failures. We wrap risky things in try/except. We log errors. We wire up something to shout when an exception gets thrown:

try:
    run_morning_routine()
except Exception as e:
    logging.error("Morning routine failed: %s", e)
    send_alert(e)

This is good practice and you should do it. But notice the assumption baked into every line: it assumes the code ran. The try block has to execute for the except to ever fire. The logger has to be reached for it to log. The alert has to be triggered by a process that is, by definition, alive.

None of that helps you when the process never starts.

The server rebooted and the scheduler didn't come back up.
The cron entry got wiped in a config change.
The disk filled and the job couldn't even launch.
A dependency upstream hung and your job was never invoked.

In every one of these cases there is no exception, because there is no running code to throw one. There's no log line, because nothing got far enough to write it. Your monitoring is sitting there patiently waiting for a signal that will never come, and interpreting the silence as "all good."

That's the trap. Absence of an error is not the same as success. For unattended jobs like cron tasks, scheduled scripts, overnight pipelines, AI agents running on a timer etc. the most dangerous failure is the one that produces no output at all.

Inverting the model: a dead-man's switch

The fix is to stop waiting for bad news and start expecting good news.

A dead-man's switch (the name comes from the pedal that stops a train if the driver goes unresponsive) flips the logic around. Instead of your code reporting when it fails, it reports when it succeeds and something external watches for that report to arrive. If the expected check-in doesn't show up on time, the watcher raises the alarm.

The crucial word is external. The thing doing the watching has to live somewhere your job doesn't. If the watchdog runs on the same machine as the job, then when that machine dies, the watchdog dies with it and a dead watchdog can't tell you anything is wrong. You need something running elsewhere whose entire purpose is to notice an absence.

Concretely, three pieces:

Your job sends a ping when it starts, and another when it finishes.
A server somewhere records those pings and knows roughly when to expect the next one.
If the expected ping doesn't arrive inside a grace window, the server alerts you.

The silence becomes the signal. You just need something that's actually listening for it.

Building it in Python

Let me show the shape of it. This is deliberately minimal. The point is the pattern, not a framework.

The client side (what your job adds)

The job needs to announce itself at the start and confirm at the end. Pure standard library, no new dependencies:

import urllib.request

PING = "https://your-watchdog.example/ping/<your-token>"

def ping(event):
    try:
        urllib.request.urlopen(f"{PING}/{event}", timeout=10)
    except Exception:
        # Monitoring must never crash the job it monitors.
        pass

ping("start")
try:
    run_morning_routine()
    ping("success")
except Exception as e:
    ping("fail")
    raise

Three things worth calling out:

The ping is wrapped so it can never take down the real job. Monitoring that crashes the thing it's monitoring is worse than no monitoring.
start and success bracket the work, so the watcher can tell "never started" apart from "started but never finished" (a hang) apart from "finished cleanly."
It's a plain HTTP GET, so this works from any language. A shell job can do the same with curl:

curl -fsS https://your-watchdog.example/ping/<token>/start
./run_backup.sh && curl -fsS https://your-watchdog.example/ping/<token>/success \
                || curl -fsS https://your-watchdog.example/ping/<token>/fail

The receiving side

On the server, receiving a ping is just recording that it happened and when:

from datetime import datetime, timezone
from flask import Flask

app = Flask(__name__)

@app.route("/ping/<token>/<event>")
def receive_ping(token, event):
    monitor = Monitor.query.filter_by(token=token).first_or_404()
    monitor.last_event = event
    monitor.last_ping_at = datetime.now(timezone.utc)
    if event in ("success", "fail"):
        monitor.last_finished_at = monitor.last_ping_at
    db.session.commit()
    return "", 200

Nothing clever here, and that's the point. The receiver's only job is to remember the last time it heard from you.

The watchdog — the part that actually matters

The interesting logic is the part that runs on its own schedule, independent of any job, and asks: is anyone overdue?

from datetime import datetime, timezone, timedelta

def check_all():
    now = datetime.now(timezone.utc)
    for monitor in Monitor.query.all():
        deadline = monitor.last_ping_at + timedelta(
            minutes=monitor.period_minutes + monitor.grace_minutes
        )
        is_late = now > deadline
        new_status = "DOWN" if is_late else "UP"

        # Only act on a *change* of state.
        if new_status != monitor.status:
            monitor.status = new_status
            db.session.commit()
            if new_status == "DOWN":
                alert(monitor, "Your job hasn't checked in — it may have stopped.")
            else:
                alert(monitor, "Your job is reporting again — recovered.")

Two design decisions here earned their keep, and both are about not being annoying:

1. State-transition alerting. You alert on the change, not on the condition. A job that's been down for six hours should generate exactly one "it's down" email, not one every time the watchdog loops. And exactly one "it's back" email when it recovers. Alert on the edge, not the level. This is the single biggest thing standing between "useful monitoring" and "inbox noise you'll mute within a day."

2. A grace window. Jobs are never perfectly punctual; a run that usually takes ten minutes occasionally takes twenty. The grace_minutes buffer stops those normal wobbles from firing false alarms. A decent rule of thumb is to set the grace to roughly 20–30% of the expected run time, with a sensible floor of a few minutes.

Run check_all() on its own timer - its own cron entry, a scheduled worker, whatever just somewhere separate from the jobs it's watching. That separation is the whole game. The watchdog has to outlive the thing it watches.

From "fix for me" to "thing other people can use"

I built the version above for exactly one job: my trading bot. It worked. The next morning the bot failed to start again, and this time my phone buzzed instead of me finding out three days later by accident.

But once I had it, I started pointing it at other things almost reflexively. A nightly database backup which, it turned out, had quietly failed a week earlier and I'd never noticed. A scraper on a schedule. A content pipeline. Every unattended job I owned had the same blind spot, and I'd just been getting lucky that none of them had bitten me yet.

That's when it stopped feeling like a personal script and started feeling like something other solo devs probably needed too. Especially the growing pile of us running AI agents and LLM pipelines on timers, where "did the agent actually run this morning?" is a real and slightly unnerving question. So I cleaned it up, put a proper dashboard on it, and turned it into a small product called PulseWatch.

The whole idea is what's above: your job sends a start and a finish ping, a server watches for them, and you get one email when something goes quiet and one when it comes back. There's a free tier with no card required if you want to point it at a job and see it work.

The one thing to take away

Even if you never use a tool for this, even if you go build the twenty lines yourself, internalise the core idea, because it changes how you think about reliability:

Your monitoring is only watching for the failures it can see. The failure that produces no output is invisible to everything that waits for output.

Go look at your most important unattended job right now and ask one question: if this silently stopped running tonight, what would tell me?

If the honest answer is "I'd notice eventually, when something downstream breaks" then that's the gap. And the fix is genuinely about twenty lines of code away.