I Took Over a 'Stable' Client Server and Found 11 Silent Failures in the First Hour

Last month a two-person agency handed me a server they'd been running "without issues for two years." Their words, not mine. One hour later I had a list of 11 things that were broken and nobody knew about.

None of them were hacking. None of them were cloud outages. All of them were the same species: something silently stopped doing its job, and every system that could have complained was pointed at a channel nobody reads.

The list, anonymized and reordered by how much each one would have cost them eventually:

1. Nightly backup writing to a full disk. Dying for ~6 weeks. The df alert went to a mailbox attached to a contractor's dead email.
2. TLS cert autorenew failing on ONE of three vhosts. The main site was fine. The staging domain would have gone browser-warning in 9 days.
3. A database dump that was 0 bytes. Cron ran nightly, exited 0, wrote nothing. ls -la was the forensic tool.
4. Redis filling memory and evicting the session store every 3 days. Users just got logged out; support blamed "the internet."
5. A deploy script pinned to an old Node version that a brew upgrade had removed. Deploys only worked from one specific laptop.
6. Log rotation gone — 22 GB of app logs. Disk pressure from #1 compounded by #6; the machine was one spike from a cascade.
7. A health-check that checked the wrong port. Monitoring showed green while the app behind the proxy was 502ing intermittently.
8. Uptime monitor watching the load balancer, not the app. Classic.
9. An API key hard-coded in a crontab — rotated 4 months ago by a "security improvement" that broke the job and fixed nothing.
10. A queue worker that had been down since March. Emails were "sending" into a dead queue. The sales team thought prospects were ignoring them.
11. No offsite copy of anything. The "backups" lived on the same disk array they were supposed to restore from.

The pattern behind all 11

Every one of these was invisible because success was silent and failure was silent too. The fix is not more dashboards. It's a dead-man's-switch habit:

• Every important job checks in when it succeeds (timestamp file or ping).
• A watchdog compares check-in age vs. expected interval and screams in a channel humans actually read when it's late.
• Exit codes, output line-counts, and runtime deltas get logged — a job that "ran" but did nothing is caught in minutes, not months.

That's maybe 30 minutes of setup per job and it would have caught 10 of the 11 above on day one.

If you're the client in this story

I packaged the exact scripts I use for this — canaries, watchdog, evidence report, the restore drill that proves your backups aren't decorative — as a self-serve starter pack. Same system I deploy on real client boxes, priced so a solo operator can afford it:

https://hive80lab.gumroad.com/l/automation-starter-pack

And if you want the full 24/7 operations layer (the watchdog plus the incident playbook that turns alerts into fixed problems instead of forwarded anxiety):

https://hive80lab.gumroad.com/l/agent-ops-24-7

Run the df check tonight. Not because I said so — because item #1 is currently happening to someone reading this.