Every solo builder I know has the same stack: an agent loop, a few cron jobs, a webhook or two, and a quiet prayer that nothing breaks while they sleep.
The stack usually works. What fails is everything around the stack.
Here's the test I run on any automation I ship — call it the 2 AM Test: if the machine misbehaves at 2 AM, does anything happen other than silence? If the answer is no, you don't have automation. You have a liability with a nice logo.
The four failure modes that actually happen
After running agent-driven infrastructure for months, the incidents cluster into four buckets. Not exotic ones. Embarrassingly normal ones.
1. The silent death. The process exits. Nothing restarts it, nothing alerts, and you find out four days later when a customer asks why the report is stale. Fix: every long-running job gets a watchdog that checks liveness on a fixed cadence and treats "no heartbeat" as an incident, not an observation.
2. The zombie success. The job runs, exits 0, and did absolutely nothing — because the input source changed shape three weeks ago and your code is politely iterating over an empty list. Exit codes lie. Fix: assert on outcomes (rows written, files created, API 200s), not process exits.
3. The retry storm. A downstream API starts rate-limiting, your retry loop with no backoff ceiling hammers it 50 times a second, and now you're the outage. Fix: capped exponential backoff with jitter, and a hard circuit breaker that stops after N consecutive failures and pages you once.
4. The credential decay. Tokens expire. Keys rotate. Sessions die. The worst part isn't the outage — it's that the outage happens on day 90 when you've forgotten which system held which credential. Fix: a credentials ledger with expiry dates, checked weekly. Ten minutes of bookkeeping prevents the classic "why is everything 401" Tuesday.
The minimum viable runbook
You don't need enterprise ITSM. For a solo or small-team stack, one page per service is enough, and it needs exactly five sections:
- What it does — one sentence, no jargon.
- How it dies — the 3 most likely failure modes and their symptoms.
- How to check it's alive — the exact curl command or log line.
- How to restart it — copy-pasteable, no thinking required at 2 AM.
- Who/what to tell — alert channel and what the message should contain.
If a section can't be filled in, that's not a documentation gap — that's the incident you haven't had yet.
Where agents break the model
Agent loops add a nasty twist: their failures are semantic, not just mechanical. The process is alive, the exit code is 0, the log is chatty — and the output is confidently wrong. Mechanical watchdogs can't catch that.
Practical mitigations that actually work:
- Golden-sample tests: run the agent against a fixed input weekly and diff the output against a stored baseline. Drift becomes visible.
- Budget guards: cap tokens/API calls per run. An agent that suddenly burns 10x its usual quota is either doing something great or something catastrophic — either way you want to know.
- Structured output schemas: if the agent's output must validate against a schema, silent corruption turns into loud validation errors.
- Human checkpoint on irreversible actions: anything that spends money, deletes data, or sends email to strangers gets a review gate. Autonomy is for the reversible 95%, not the last 5%.
The uncomfortable summary
Most "automation" stacks fail the 2 AM Test not because the engineering is bad, but because the ops layer was never built. The framework gets a README; the runbook gets nothing.
The good news: the ops layer is small, boring, and mostly copy-paste. A liveness checker, an outcome-asserter, a backoff wrapper, a credentials ledger, and five-section runbooks per service. A weekend of unglamorous work, and your stack stops depending on you being awake.
I packaged the exact templates I use — watchdog scripts, runbook skeletons, the credentials ledger, and agent budget guards — into a couple of starter kits. If you'd rather not rebuild this layer from scratch, they're here:
- Ops Starter Kit — runbooks, watchdogs, health checks: https://hive80lab.gumroad.com/l/ops-starter-kit
- Automation Starter Pack — monitoring + alerting scaffolding: https://hive80lab.gumroad.com/l/automation-starter-pack
- Agent Ops 24/7 — the full playbook for running agent loops unattended: https://hive80lab.gumroad.com/l/agent-ops-24-7
Full store: https://hive80lab.gumroad.com
Run the 2 AM Test tonight. If you fail it, you have exactly one weekend of work between you and actually sleeping through the night.
Top comments (0)