Monitoring bills creep because teams monitor everything. Senior teams monitor the few things that page correctly and drop the rest. Here are 7 alert rules that earn their keep — each one mapped to a real incident class.
1. The Checkout Canary (synthetic, not CPU)
A synthetic transaction that logs in and completes checkout every 60s. CPU at 90% with checkout fine = green. Checkout failing with CPU at 20% = page. You get paid when you catch revenue-level failure before a customer email does.
2. The Silent Death Detector
Services that die and stay dead cost more than flappy ones (nobody notices). Alert on: up == 0 for 2 minutes plus a rate-of-change guard so restart storms don't mask a true outage.
3. Error Budget Burn Rate
Not "5xx > 1%" — that fires all the time. Alert when you burn 2% of your monthly error budget in 1 hour. Google's SRE playbook calls this multi-window burn-rate alerting; it cuts noisy pages by an order of magnitude.
4. The Queue That Never Drains
Any queue (jobs, emails, webhooks): alert if oldest message age > 10 minutes. Queue depth is a lagging indicator; age is the leading one.
5. Disk Space With a Headroom Rule
90% full = warning, 95% = page. But the rule that actually saves you: alert if disk grew >5% in the last hour — that's a log fire, and you'll be full by morning otherwise.
6. The Expired Certificate Alarm
Still the #1 self-inflicted outage. Alert 30 days out. One line of config; one embarrassing incident avoided.
7. The Restore-Path Tripwire
Once a week, a cron job restores your latest backup into staging and runs a checksum. If the restore fails, you get an alert — not a surprise during a real incident. A backup you've never restored is Schrödinger's backup.
The Meta-Rule
For every alert that exists, someone must be able to answer: what do I do when this fires at 3AM? If the answer is "read the source code", delete the alert and write a runbook instead.
If you want these 7 rules as ready-to-paste configs (Prometheus/UptimeRobot/friendly cron variants) plus the runbook and incident-checklist templates, we packaged them as the Ops Starter Kit.
Which of the 7 are you missing today? Say it in the comments — I'll reply with the cheapest way to add it.
Top comments (0)