I watched an unattended agent fleet for a month. Only 3 monitors actually caught failures.

We run autonomous agents around the clock — publishing, outreach, monitoring, revenue rails. Over a month of that, the alarms that ever caught a real failure boiled down to three. Everything else was decoration.

1. The signed probe

Not "is the process up" — "can I authenticate, read one real record, and get out?" That round trip catches dead tokens, dead DNS, and dead quotas, the three things that actually kill overnight runs. It is ten lines of shell. It caught every credential death; process monitors caught none of them, because the process was always alive right up until it was useless.

2. The artifact counter

Every successful step writes an artifact — a file, a row, a captured HTTP code. A step that claims success with no artifact is a failure. Monitor one number: artifacts written per run. If today's count is below yesterday's floor, something is silently degrading even though every exit code says 0. This caught a partial API deprecation that returned valid 200s with empty payloads for two days.

3. The watcher's watcher

A second, independent probe — different network path, different credentials — whose only job is to see the first monitor. Cron dead, laptop asleep, token rotated: the first monitor can't tell you, the second one can. When the second one goes quiet, that's the page, not the first.

What we retired

• CPU/memory dashboards. Nothing we run is resource-bound; the failures are semantic.
• Uptime percentages. A 99.9% uptime number coexisted happily with every real outage we had.
• Log-scanning alert rules. They either matched nothing (log formats drift) or fired on noise. Artifacts with known shapes beat parsing prose.

The rule we ended with

A monitor earns its place only if it can name the failure class it catches and the last date it caught one. Three monitors passed that test. Twelve didn't.

---

If unattended ops is your problem too: the free 1-page first-30-minutes incident checklist (no email), and the full 25-script toolkit in the Agent Ops Mega Bundle. Working samples on GitHub.