Most on-call pain isn't from incidents. It's from alerts that were never worth a page in the first place. A page should be a contract: something is broken, a human can fix it now, and waiting makes it worse. Everything else belongs in a dashboard.
We audited a year of pages across a few small teams and found the same three alert classes were the only ones that survived the "was this page useful?" review:
- Error budget burn rate. Not raw 5xx counts — burn rate against the budget you promised customers. It fires exactly when you're about to break an SLO, and it never fires on a Tuesday afternoon for no reason.
- Queue age, not queue depth. A queue can be huge and healthy, or tiny and dying. Depth lies; the age of the oldest message is the truth about whether someone's order will actually process.
- Restore-path heartbeat. Not "is the backup job green" — is the restore path actually exercisable right now. A backup you haven't restored from this quarter is a theory with a cron job.
Everything else we demoted to email or dashboards. Paging volume dropped ~70% and, the part that matters, nobody ignored a page since. Fewer pages is not the goal. Pages that are always worth answering is the goal.
How to move one alert today
Pick your noisiest alert. Ask three questions: can a human act on it right now? will waiting an hour change the outcome? did the last ten pages of it lead to action? Any "no" demotes it to dashboard-only. Do one per day and your on-call rotation is unrecognizable in two weeks.
We package this stuff as done-for-you runbooks + 24/7 monitoring kits: 24/7 Agent Ops Kit — self-serve, instant download.
Want to try before you buy? A free starter kit lives on the same page.
Top comments (0)