Last Tuesday at 3:47 AM, my monitoring stack caught a disk-full condition on a client's staging box, opened a ticket, drafted the fix, and had the cleanup running before I woke up. I paid nothing for this. No APM tier upgrade, no managed service, no on-call contractor.
Here's the setup that made it happen — and the honest parts people leave out of these posts.
The architecture (boring on purpose)
Three pieces, all free:
The sentries. Cron-driven checks on the money path: heartbeat curls, disk/mem guards, log-noise tripwires, SSL expiry. Each one is 5-15 lines of shell. They don't "think" — they scream on anomaly and stay quiet otherwise. False-positive rate is the whole game: 2 consecutive failures before paging, severity tiers for the rest.
The dispatcher. A lightweight agent loop (I used an LLM API on a free tier for the first 3 months) that receives sentry alerts, classifies them against a runbook, and either (a) executes a pre-approved remediation script, (b) drafts the fix + posts it to a ticket, or (c) stays silent. The trick: it only ever runs remediations that a human reviewed once and signed off in a manifest. It has never invented a command.
The dead-man's switch. The sentries check in with a heartbeat service when they run. If they stop running, I get an alert. The watcher watching the watchers is the piece everyone skips — and the reason "my automation was down for a week" stories exist.
Total runtime cost: a $5 VPS I already had, and API spend that stayed under the free tier until traffic grew.
What it actually caught in month one
- Disk-full on staging (would have broken CI the next morning) — auto-cleaned.
- A cron job that silently died 6 days earlier — flagged, fixed, dead-man's-switch added.
- SSL cert 14 days from expiry — renewed before anyone noticed.
- 3 log-noise storms (a retry loop in someone's side project) — tripwires caught the anomaly curve, not the errors themselves.
None of these would have justified a $500/month APM invoice. All of them would have cost real hours.
The honest parts
- The runbook is 80% of the work. The AI dispatcher is only as good as the manifest of pre-approved actions behind it. I spent a weekend writing down "when X happens, do Y" — that document is the product. The model just executes it.
- Escalation discipline matters more than intelligence. The system's most valuable behavior is shutting up. Anything ambiguous goes to a human ticket with a drafted fix attached. Over-clever autonomous remediation is how you delete a database at 4 AM.
- "For $0" means no software licenses. Your time is not free. This took ~2 weekends to build. It then ran untouched for 4 months — that ROI math is the actual pitch.
If you want to steal this
I packaged the exact version I run — the sentry scripts, the dispatcher runbook manifest, the escalation rules, and the weekly review template — in the Agent-Ops 24/7 kit. If you're starting from zero ops discipline, the Ops Starter Kit is the on-ramp ($14, one-time, deploys in an afternoon). The dispatcher/automation layer lives in the Automation Starter Pack ($19). Everything is in the Ops Mega Bundle if you'd rather take the full stack at once.
No subscriptions. You own the scripts. They run on your box, not mine.
Question for the comments: what's the longest your "automated" system has run silently broken? I'm collecting war stories for a follow-up on dead-man's-switch design — best one gets featured.
Top comments (0)