How I Built a Dead-Man's Switch for My AI Trading Pipeline in Python

The morning I noticed my trading bot hadn't traded in three days.

I run a small trading bot. Every morning before the market opens it pulls signals, weighs them, and decides whether to act. It had been humming along for weeks, so I'd mostly stopped watching it.

Then one morning I opened the dashboard out of idle curiosity and something was off. No trades. Not "it decided not to trade", just crickets. I scrolled back. No trades the day before either. Or the day before that.

Three days. The bot had silently stopped running and I had no idea.

Here's the part that stuck with me: there was no error to find. No red text, no stack trace, no alert in my inbox. The overnight process that was supposed to kick everything off had just… not kicked off. A scheduling hiccup on my machine, most likely. The code was fine. The code simply never ran.

And every single tool I had for catching problems was completely blind to it.

Why everything I had was looking the wrong way

Think about how we normally catch failures. We wrap risky things in try/except. We log errors. We wire up something to shout when an exception gets thrown:

try:
    run_morning_routine()
except Exception as e:
    logging.error("Morning routine failed: %s", e)
    send_alert(e)

This is good practice and you should do it. But notice the assumption baked into every line: it assumes the code ran. The try block has to execute for the except to ever fire. The logger has to be reached for it to log. The alert has to be triggered by a process that is, by definition, alive.

None of that helps you when the process never starts.

• The server rebooted and the scheduler didn't come back up.
• The cron entry got wiped in a config change.
• The disk filled and the job couldn't even launch.
• A dependency upstream hung and your job was never invoked.

In every one of these cases there is no exception, because there is no running code to throw one. There's no log line, because nothing got far enough to write it. Your monitoring is sitting there patiently waiting for a signal that will never come, and interpreting the silence as "all good."

That's the trap. Absence of an error is not the same as success. For unattended jobs like cron tasks, scheduled scripts, overnight pipelines, AI agents running on a timer etc. the most dangerous failure is the one that produces no output at all.

Inverting the model: a dead-man's switch

The fix is to stop waiting for bad news and start expecting good news.

A dead-man's switch (the name comes from the pedal that stops a train if the driver goes unresponsive) flips the logic around. Instead of your code reporting when it fails, it reports when it succeeds and something external watches for that report to arrive. If the expected check-in doesn't show up on time, the watcher raises the alarm.

The crucial word is external. The thing doing the watching has to live somewhere your job doesn't. If the watchdog runs on the same machine as the job, then when that machine dies, the watchdog dies with it and a dead watchdog can't tell you anything is wrong. You need something running elsewhere whose entire purpose is to notice an absence.

Concretely, three pieces:

1. Your job sends a ping when it starts, and another when it finishes.
2. A server somewhere records those pings and knows roughly when to expect the next one.
3. If the expected ping doesn't arrive inside a grace window, the server alerts you.

The silence becomes the signal. You just need something that's actually listening for it.

Building it in Python

Let me show the shape of it. This is deliberately minimal. The point is the pattern, not a framework.

The client side (what your job adds)

The job needs to announce itself at the start and confirm at the end. Pure standard library, no new dependencies:

import urllib.request

PING = "https://your-watchdog.example/ping/<your-token>"

def ping(event):
    try:
        urllib.request.urlopen(f"{PING}/{event}", timeout=10)
    except Exception:
        # Monitoring must never crash the job it monitors.
        pass

ping("start")
try:
    run_morning_routine()
    ping("success")
except Exception as e:
    ping("fail")
    raise

Three things worth calling out:

• The ping is wrapped so it can never take down the real job. Monitoring that crashes the thing it's monitoring is worse than no monitoring.
• start and success bracket the work, so the watcher can tell "never started" apart from "started but never finished" (a hang) apart from "finished cleanly."
• It's a plain HTTP GET, so this works from any language. A shell job can do the same with curl:

curl -fsS https://your-watchdog.example/ping/<token>/start
./run_backup.sh && curl -fsS https://your-watchdog.example/ping/<token>/success \
                || curl -fsS https://your-watchdog.example/ping/<token>/fail

The receiving side

On the server, receiving a ping is just recording that it happened and when:

from datetime import datetime, timezone
from flask import Flask

app = Flask(__name__)

@app.route("/ping/<token>/<event>")
def receive_ping(token, event):
    monitor = Monitor.query.filter_by(token=token).first_or_404()
    monitor.last_event = event
    monitor.last_ping_at = datetime.now(timezone.utc)
    if event in ("success", "fail"):
        monitor.last_finished_at = monitor.last_ping_at
    db.session.commit()
    return "", 200

Nothing clever here, and that's the point. The receiver's only job is to remember the last time it heard from you.

The watchdog — the part that actually matters

The interesting logic is the part that runs on its own schedule, independent of any job, and asks: is anyone overdue?

from datetime import datetime, timezone, timedelta

def check_all():
    now = datetime.now(timezone.utc)
    for monitor in Monitor.query.all():
        deadline = monitor.last_ping_at + timedelta(
            minutes=monitor.period_minutes + monitor.grace_minutes
        )
        is_late = now > deadline
        new_status = "DOWN" if is_late else "UP"

        # Only act on a *change* of state.
        if new_status != monitor.status:
            monitor.status = new_status
            db.session.commit()
            if new_status == "DOWN":
                alert(monitor, "Your job hasn't checked in — it may have stopped.")
            else:
                alert(monitor, "Your job is reporting again — recovered.")

Two design decisions here earned their keep, and both are about not being annoying:

1. State-transition alerting. You alert on the change, not on the condition. A job that's been down for six hours should generate exactly one "it's down" email, not one every time the watchdog loops. And exactly one "it's back" email when it recovers. Alert on the edge, not the level. This is the single biggest thing standing between "useful monitoring" and "inbox noise you'll mute within a day."

2. A grace window. Jobs are never perfectly punctual; a run that usually takes ten minutes occasionally takes twenty. The grace_minutes buffer stops those normal wobbles from firing false alarms. A decent rule of thumb is to set the grace to roughly 20–30% of the expected run time, with a sensible floor of a few minutes.

Run check_all() on its own timer - its own cron entry, a scheduled worker, whatever just somewhere separate from the jobs it's watching. That separation is the whole game. The watchdog has to outlive the thing it watches.

From "fix for me" to "thing other people can use"

I built the version above for exactly one job: my trading bot. It worked. The next morning the bot failed to start again, and this time my phone buzzed instead of me finding out three days later by accident.

But once I had it, I started pointing it at other things almost reflexively. A nightly database backup which, it turned out, had quietly failed a week earlier and I'd never noticed. A scraper on a schedule. A content pipeline. Every unattended job I owned had the same blind spot, and I'd just been getting lucky that none of them had bitten me yet.

That's when it stopped feeling like a personal script and started feeling like something other solo devs probably needed too. Especially the growing pile of us running AI agents and LLM pipelines on timers, where "did the agent actually run this morning?" is a real and slightly unnerving question. So I cleaned it up, put a proper dashboard on it, and turned it into a small product called PulseWatch.

The whole idea is what's above: your job sends a start and a finish ping, a server watches for them, and you get one email when something goes quiet and one when it comes back. There's a free tier with no card required if you want to point it at a job and see it work.

The one thing to take away

Even if you never use a tool for this, even if you go build the twenty lines yourself, internalise the core idea, because it changes how you think about reliability:

Your monitoring is only watching for the failures it can see. The failure that produces no output is invisible to everything that waits for output.

Go look at your most important unattended job right now and ask one question: if this silently stopped running tonight, what would tell me?

If the honest answer is "I'd notice eventually, when something downstream breaks" then that's the gap. And the fix is genuinely about twenty lines of code away.