Not hacks. Not cloud outages. Just ordinary servers doing ordinary things — until a reboot, an upgrade, or a full disk turned "fully automated" into "fully stopped" at 3 AM while everyone slept.
Twelve small teams, twelve setups running unattended jobs: deploys, backups, customer emails, invoice syncs. Nine of them shared the same three wounds:
1. Nothing proved the job still worked. Cron exit code 0 was treated as success. But "exited 0, wrote 0 bytes" was everywhere — empty backups, empty exports, empty sends. A job that runs and does nothing is worse than a job that fails loudly, because everyone believes it.
2. Alerts went to a channel nobody owned. A mailbox from a contractor who left in 2023. A Slack webhook pointing at a deleted channel. One team's monitoring emailed a domain that had lapsed — someone else now owns it (that is its own horror story).
3. One reboot away. Scripts with absolute paths to a Homebrew-installed binary that only existed on one laptop. A Docker container started manually six months ago, never enabled to restart. A VPN tunnel required for the database sync that only came back up if someone typed a password.
None of these are exotic. All nine were fixable in an afternoon with the same pattern:
- Every important job writes a check-in timestamp (or pings a URL) when it actually produces output.
- A watchdog compares check-in age to expected interval and escalates to a channel a human provably reads.
- A weekly evidence report proves the backups restore, the sends send, and the syncs sync — with line counts and sizes, not exit codes.
That pattern costs almost nothing to deploy and it converts "I think it's fine" into "here is the proof it was fine at 6 AM today."
I run 24/7 operations for client systems on exactly this pattern and packaged the core as a self-serve starter pack — canaries, watchdog, the evidence report, and the restore drill that proves your backups are not decorative:
https://hive80lab.gumroad.com/l/automation-starter-pack
For the full 24/7 operations layer (watchdog-that-watches-the-watchdog included):
https://hive80lab.gumroad.com/l/agent-ops-24-7
I also wrote up the overnight version of this — 47 alerts, 3 that mattered, and the $0 fix for alert fatigue: https://telegra.ph/47-Alerts-Fired-Overnight-Exactly-3-Mattered-The-0-Fix-for-Alert-Fatigue-09-17
I write about the boring infrastructure failures that cost small teams real money. Follow if you would rather meet them at $400 than $40,000.
Top comments (0)