Automation is supposed to save time. But automation that was shipped once and never watched doesn't just fail — it fails silently, overnight, and it bleeds money every week it stays broken. Nobody notices until a customer does.
Here are the five failure modes I see most often in small teams, and the cheapest fix for each.
1. The silent death
A cron job dies. A webhook receiver gets decommissioned. Nothing screams, because nothing was ever wired to scream. The pipeline that 'runs every night' has been dead for three weeks and everyone still trusts its dashboard.
Cheapest fix: every automation gets one end-to-end health check that writes a timestamp somewhere you actually look. A dead heartbeat after N hours = page yourself. Ten lines of shell beats a monitoring platform you'll never finish configuring.
2. The retry storm
A failing job retries every minute forever — hammering a rate-limited API, padding your bill, and burying the real error in ten thousand identical log lines. Retries without backoff and without a max are a denial-of-service attack you're launching against yourself.
Cheapest fix: exponential backoff, hard cap on attempts, and a distinct alert when the cap trips. If the same job failed 50 times, that's not noise — that's one incident.
3. The single point of failure
One API key, in one account, on one laptop. The key expires, the laptop updates, the whole business process vanishes. Unattended automation multiplies the blast radius of any single credential.
Cheapest fix: inventory every secret your automation depends on, write down what breaks when each one expires, and set expiry reminders. Thirty minutes now saves the 2 AM scramble later.
4. The data drift
The automation 'works' — green checkmark every day — but the output has been subtly wrong for a month. An API response shape changed, a currency got parsed without a symbol, a timezone shifted. Silence + green checkmark is the most dangerous combination in ops.
Cheapest fix: assert on output, not exit codes. A row count, a checksum, a total that crosses-checked against yesterday. If the number moved more than X%, stop and tell a human.
5. The un-owned system
The script that keeps the lights on lives in someone's home directory, has no README, and the person who wrote it left. When it finally breaks, nobody even knows where to look.
Cheapest fix: every automation gets a one-page runbook: what it does, where it lives, what it depends on, how to restart it. Not a wiki. One page, next to the code.
The pattern behind all five
Every one of these is a detection gap, not a coding gap. The code mostly works. What's missing is the small, boring layer that notices when reality drifts away from the dashboard.
If your team runs anything unattended — even a single cron job — I put together a free playbook on the first 30 minutes after an outage: what to check, in what order, before you touch anything:
👉 Free: The First 30 Minutes After Ransomware / Outage
And if you want the full checklist system — runbooks, heartbeat scripts, retry rules, and the daily 10-minute ops drill — it's all in the Ops Starter Kit:
👉 Ops Starter Kit on Gumroad — $14, yours forever, updates included.
Ship the boring safety layer this week. It's the highest-ROI hour your team will spend all month.
Top comments (0)