Run one test on your automation setup tonight: turn off every dashboard, every alert channel, every status page — and go to sleep. If anything breaks, does anyone find out before a customer does? If the honest answer is "probably not," you don't have automation. You have hope with a cron job.
Here's the standard I hold my own ops to, and the four checks that pass it.
The 2 a.m. test
Every critical job must be able to wake someone (or something) without a human remembering to look. No "I check it every morning." Morning is when customers already left.
Check 1: Does failure make noise?
A cron that fails silently is worse than no cron — you think you have coverage. Every scheduled job should emit a signal on failure: an email, a webhook, a log line something watches. If you can't name where the failure signal goes, it doesn't exist.
Check 2: Does recovery leave evidence?
When a job retries, does it log what it retried, when, and why? The cheapest postmortem tool is a plain text log you can grep. Fancy dashboards rot; grep -i error run.log never does.
Check 3: Can a stranger run your setup?
If only you understand the pipeline, it will die the week you're sick. A short README with the three commands that matter beats any internal wiki. (Every product I ship includes one for exactly this reason.)
Check 4: Is there a tripwire on the money path?
If your setup sells anything, a checkout break is the most expensive failure mode — and the quietest. Hit your own checkout weekly with a script, not your memory.
What this costs
Nothing but an afternoon. These are the exact patterns in my ops kits — runbooks, monitors, tripwires — packaged so you deploy them same-day:
- Ops Starter Kit — the base runbook + monitoring pack
- Agent-Ops 24/7 — keeping autonomous systems alive without babysitting
- Automation Starter Pack — the scripts behind checks 1-4
Run the test tonight. If you fail it, fix Check 1 first — it's the one that saves you at 2 a.m.
P.S. An automation you have to check manually is just a slower way of doing it by hand.
Top comments (0)