The 2AM page nobody can close
Every team that runs automation 24/7 eventually meets the same ghost: an incident that looks open, will never complete, and nobody has the authority to close. We call it a dangling event, and it is the most expensive line on any ops dashboard — because it taxes attention every single day forever.
Where dangling events come from
Almost every workflow engine has races: two workers pick up near-identical tasks, one wins, the other's acceptance sits in history forever. The engine moves on. Your dashboard does not. The loser has no terminal state, so it stays "in progress" until someone manually buries it — and if nobody does, your signal degrades: real in-progress work is now indistinguishable from dead work.
The fix costs one column
Add a superseded state to every lifecycle you operate. When a loser is detected — same task key, earlier timestamp, already-delivered — mark it superseded with a pointer to the winner. Ten lines of code. What you get back:
- Dashboards that mean "open" when they say "open"
- Pages that can always be closed
- Metrics that stop lying politely
Three questions to audit your own stack tonight
- Pick any event on your board stuck longest. Can a named human close it today, with a documented command? If not, it is a dangling event.
- What does your verifier actually prove — substance or structure? Write the distinction down. A green light that means "structurally plausible" is not a green light that means "correct".
- Do you reserve work for specific workers instead of racing general sweepers? Exclusive lanes with short poll intervals beat crowds every time — reserved work settles, contested work expires.
Make the dangerous states loud
The whole discipline of running agents and automations around the clock reduces to two moves: make the boring cases boring on purpose, and make the dangerous states LOUD. If your system can sit silent for six hours and nothing notices, you do not have monitoring — you have a screensaver.
We run our own stack this way and packaged the discipline — runbooks, paging rules, incident comms templates, the "five questions" lifecycle audit — into the Agent Ops 24/7 playbook:
→ https://hive80lab.gumroad.com/l/agent-ops-24-7
And if you are starting from zero, the Ops Starter Kit gives you the two-hour Sunday setup that keeps a solo SaaS observable from day one:
Top comments (0)