Silent failures don't page you. They invoice you later.
The worst outages I've seen didn't start with an alert. They started three weeks earlier, quietly: a cron that kept "running," a backup that kept "succeeding," a cert that kept counting down. Nothing pages until the customer emails.
Here's the 10-minute audit I run every Monday. Six checks, all copy-paste. It has caught things my monitoring never did — because monitoring watches symptoms, and this watches assumptions.
The six checks
1. Do the crons actually do what they claim? (2 min)
# last 10 cron exits — anything 0 with tiny runtime is suspicious
grep cron /var/log/syslog | tail -10
An exit 0 in 0.4 seconds is not a success. It's a script that gave up politely. Check the biggest job by hand: did the output file actually change today? ls -la on the artifact beats reading the log.
2. Did the backup LAND anywhere? (1 min)
ls -la /backups | tail -3
Not "did the job run" — did bytes arrive, with today's date, on a different disk/box than the database. A backup into the same machine as the data is a copy, not a backup.
3. When does the cert expire? (1 min)
echo | openssl s_client -connect yoursite.com:443 2>/dev/null | openssl x509 -noout -enddate
Under 14 days? Fix it today, during business hours, with coffee — not at 2 a.m. Sunday with your phone at 4%.
4. Is the disk lying in wait? (1 min)
df -h | awk '$5+0 > 80 {print}'
Logs are the classic killer. If anything prints, rotate before it chooses its own moment.
5. Would you KNOW if the site were down right now? (2 min)
Open the site from your phone on cellular — not from the server. Monitoring that lives inside the box it watches dies with the box. External check or it doesn't exist.
6. Could a stranger restart it? (2 min)
One page: what this service is, how to tell it's alive, how to restart it, who to call. If that page doesn't exist, you are the single point of failure — and Mondays are when you find out, because Monday is when you're in a meeting.
Why this beats "better monitoring"
Monitoring tells you when something breaks. This audit finds the things that pretend to work — the exit-0 cron, the backup into a dead bucket, the cert nobody checked since setup. Those don't trip alerts; they trip invoices, refunds, and trust loss three weeks later.
Ten minutes, six checks, every Monday. It's the highest-leverage quarter-hour in small-team ops.
If you want the guards pre-built
I package the exact scripts and templates for this — cron/heartbeat guards, restore-drill checklist, runbook skeleton:
- Ops Starter Kit — monitoring + runbook framework for tiny teams (code LAUNCH50 = 50% off)
- Automation Starter Pack — the silent-failure guard patterns with copy-paste commands
- Agent Ops 24/7 — autonomous ops without a staff
Run the six checks this Monday. Write down what #1 and #2 find. That list is your quarter's real roadmap.
Top comments (0)