My AI Agent's Monitoring Was Dead for 11 Days. A Reader Noticed Before My Alerts Did.

For eleven days, my dashboard was green. Every cron job reported success, every health endpoint returned 200, and my AI agent's nightly pipeline logged "completed" like clockwork. On day twelve I found out the pipeline had been doing absolutely nothing since day one of that streak — and my monitoring had been happily dead the entire time, too silent to tell me.

That's the lesson nobody warns you about when you automate everything on a Raspberry Pi: your monitoring can die quietly, and a dead monitor looks exactly like a healthy system. Both are silent.

The setup that felt bulletproof

I run a small stack of AI agents on a Pi 5. One of them does nightly content research: it pulls from three APIs, scores the results, and writes a summary into a database that my morning digest reads from. Like any good paranoid operator, I had "monitoring" on it:

1. The script wrapped every run in a try/except and logged errors to a file.
2. A second cron job grepped that error log every hour and emailed me if it found anything.
3. The Pi itself ran a health endpoint I checked with an external uptime service.

Three layers. What could go wrong?

All three, it turned out — from a single root cause.

What actually happened

One of the upstream APIs changed its auth flow. My research agent started getting 403s on every request. But here's the thing: the agent caught those exceptions, logged them to the error file, and kept going — writing an empty-but-successful run to the database. The script exited 0. Cron reported success. My digest read the empty summary and politely told me there was "nothing notable" that night.

That's bad, but survivable. The killer detail was layer two: the hourly grep-the-error-log job had itself silently stopped running nine days earlier, when a system update moved the Python venv it depended on. No venv, no job, no error — cron just skips jobs whose interpreter vanished without making a sound. The error log was filling up with 403s that nobody was reading.

And layer three? The health endpoint only answered "is the Pi up?" It was. Uptime monitors that check liveness don't check work. My Pi was alive, healthy, and doing nothing useful — the monitoring equivalent of a security guard who shows up every day, sits in an empty building, and never notices the servers are gone.

I found out because a reader emailed to ask why my digest had been so thin for almost two weeks. My customers were my monitoring. That's the postmortem in one sentence.

The fix: dead man's switches

The pattern that solves this is old — it comes from trains and industrial machines, where the operator has to keep holding a button or the system assumes they're incapacitated and stops itself. In software it's called a dead man's switch or heartbeat monitoring, and the logic inverts everything:

Instead of alerting when something fails, alert when an expected success doesn't arrive.

A failure-based monitor is silent when healthy — which means it's also silent when the monitor itself is dead. A dead man's switch is loud when anything stops, including itself. The absence of good news is the bad news.

Here's how I rebuilt the stack:

1. External heartbeat for every scheduled job. Each cron job now ends with a curl to a heartbeat service (I use a self-hosted one on a different machine — more on that below):

# in crontab — the ping is the LAST thing that runs
15 2 * * * /home/sean/agent/run_research.sh && curl -fsS -m 10 https://heartbeat.example/ping/research-nightly >/dev/null

If the script crashes, hangs, or the Pi loses power, the ping never arrives and the heartbeat service emails me within minutes. Note the && — the ping only fires on success, so a failed run also triggers the alert.

2. Assert on work, not on liveness. My health endpoint used to return {"status": "ok"} if the Pi was up. Now it returns the timestamp and row count of the last actual output:

@app.route("/health/research")
def health():
    last = db.query("SELECT finished_at, items FROM runs ORDER BY finished_at DESC LIMIT 1")
    stale = datetime.utcnow() - last.finished_at > timedelta(hours=30)
    empty = last.items < 3
    code = 500 if (stale or empty) else 200
    return jsonify({"finished_at": str(last.finished_at), "items": last.items}), code

The uptime monitor hits that URL. Now "green" means work was recently done and non-trivial, not just that a socket answered.

3. Canary data. Once a week the research agent is fed a synthetic input with a known marker phrase. If the marker doesn't appear in the output database, the pipeline is broken even if every step reported success. This catches the nastiest failure mode: the agent that runs perfectly and produces garbage.

4. The monitor must not share fate with the monitored. This was my most embarrassing mistake. My first instinct was to run the heartbeat service on the same Pi. That's a smoke detector wired to the house's electricity — if the Pi dies, the thing that's supposed to tell me the Pi died also dies. The heartbeat service now runs on a $3/month VPS, and it has its own dead man's switch: it pings the Pi daily, and if the Pi doesn't answer, the VPS alerts me. Each machine watches the other. Mutual assured notification.

The part that still stings

The honest failure section isn't just the eleven days — it's that I'd been smug about my monitoring. I'd written about backups and retry logic here, and I'd mentally filed "monitoring" as solved. When you have three layers of monitoring and all three fail from one root cause, the problem isn't the layers — it's that all three answered the same question ("is anything visibly broken?") instead of the question that matters ("did the expected work actually happen?").

Two concrete lessons I now apply to everything:

• Every automation gets a heartbeat ping as its final step. No exceptions, no "this job is too trivial." The trivial jobs are the ones you never check.
• Test your alerts deliberately. Once a month I break something on purpose — kill the cron daemon, point an API key at a dead endpoint — and time how long until my phone buzzes. If an alert path has never fired in anger, assume it's broken. Mine was.

The whole rebuild took an afternoon. Eleven days of silent failure, fixed with a curl and a slightly smarter health endpoint. The asymmetry is almost offensive.

If your agents run unattended — overnight, on weekends, on a Pi in a closet — spend twenty minutes this week asking: if the monitor died, who monitors the monitor? If the answer is "my customers will tell me," you have the same setup I did.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.