DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com on

Stop Alerting on CPU: How to Fix Your 3 AM On-Call Nightmare

One red alert in a sea of green. That's the only page worth waking up for.

One red alert in a sea of green. That’s the only page worth waking up for.

Pull up a chair, grab some chai, and let me tell you about a rookie mistake almost every engineer makes when they first carry the pager.

Years ago, early in my on-call rotations, my phone went off non-stop through the night. By 4:00 AM, my brain was so fried that I was instinctively snoozing PagerDuty — the incident dispatch tool that calls and wakes up the engineer on duty — without even reading what it was screaming about. I just swiped it away and buried my face back into the pillow.

Think about that for a second. That is the exact failure mode alerting systems are built to prevent. The system had cried wolf fifty times, so when a real disaster arrived, it blended right into the background static. Classic human conditioning.

That night etched the most important rule of systems monitoring into my head:

An alert that doesn’t require immediate action isn’t an alert. It’s spam with a pager attached.

The default everyone starts with (and why it’s wrong)

When teams set up observability for the first time, they almost always alert on internal causes. You know the drill:

  • CPU usage crossing 80% (the main processor working hard)

  • Disk usage crossing 85% (storage space filling up)

  • Memory running past 90% (RAM running short)

  • A single pod restarting (a pod is a tiny container running your application code in Kubernetes)

  • SSL security certificates expiring in 30 days

Setting these thresholds makes you feel responsible. You tell yourself, “Hey, I am staying ahead of trouble!” In reality, this is the number one reason your engineers dread being on-call.

Here is why: high CPU is not a customer-facing problem. It is merely a signal that something inside the engine is doing work. It could mean your system is under heavy load, but it could also mean a scheduled batch script kicked off, a routine daily deploy ran, or the Kubernetes Horizontal Pod Autoscaler (the mechanism that automatically spins up extra application containers when traffic jumps) is doing its job perfectly.

Precursors fire constantly during completely healthy, normal operations. If an alert rings every time an internal gear turns fast, you train your brain to treat every alarm as a false alarm.

I learned this the embarrassing way. We had an alert named HighCPUWarning. It buzzed our phones constantly, day and night, yet it never once helped us catch a real outage. One afternoon, we just deleted it. Guess what happened? Absolutely nothing. The systems ran fine, downtime did not increase, and the team finally got some peace. If deleting an alert changes nothing about your reliability, it was pure noise from day one.

Alert on symptoms, not causes

To fix this, you have to flip your mental model. It is a philosophy Google’s Site Reliability Engineering (SRE) teams popularized, and it holds up in every production environment I have ever seen: alert on what your users can actually feel.

Think of it like driving a car. If the oil temperature needle wiggles slightly above midpoint on a hot afternoon, you do not pull emergency brakes on the highway. But if thick black smoke billows out of the hood and the accelerator stops responding, you pull over immediately. One is an internal operating condition; the other is an unmistakable failure.

Your users do not know or care if your CPU is sitting at 82%, your disk is at 87%, or a single pod restarted silently in the background. What users feel is simple:

  • The checkout button throws an error page (requests failing).

  • The search bar takes 12 seconds to load (requests running slow).

  • The entire site won’t load at all (the service is completely down).

The only events that should wake a sleeping human are built around SLIs — Service Level Indicators, which are direct, quantifiable measurements of customer experience (like error percentage or response latency). Everything else belongs on an informational dashboard, or at most, inside a low-priority ticket for someone to look at after morning standup.

Alert pipeline: raw signals filtered into pages, tickets, and dashboards.

Every signal enters the pipeline. Only symptoms come out as pages.

Look at the difference between these two Prometheus alerting rules (the monitoring engine we use to inspect metrics and trigger alerts):

# BAD: pages on a cause
- alert: HighCPUUsage
  expr: cpu_usage > 80 for: 5m

# GOOD: pages on a symptom
- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total[5m])) > 0.01
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "Error rate above 1% for 10 minutes"
    runbook: "<link-to-runbook>"
Enter fullscreen mode Exit fullscreen mode

Let’s unpack what the second rule actually does:

  • http_requests_total{status=~"5.."} counts all server-side errors (HTTP 500, 502, 503 — the screens telling customers “Oops, something went wrong”).

  • rate(...[5m]) calculates the per-second rate of those errors over a 5-minute rolling window.

  • We divide those broken requests by the total volume of all requests arriving at the service.

  • If broken requests exceed 0.01 (meaning more than 1% of your real users are getting failures) continuously for 10 full minutes, only then does it ring the phone.

The first rule wakes you up because an arbitrary hardware metric crossed a number. The second rule wakes you up because real human beings are having a broken experience on your platform. Only one of those deserves your 3:00 AM attention.

The three tiers that fixed my on-call

After getting burned by alert storms, I reorganized our monitoring rules into three clear buckets. You can take this exact model and apply it to your team tomorrow:

The three alert tiers: page, ticket, dashboard.

The only question that matters: does a human need to wake up for this?

Tier 1 — Page (Wake someone up immediately). Reserved strictly for user-facing symptoms: spike in error rates, severe latency spikes, or complete system outages. An engineer’s sleep is sacred; you only interrupt it if customers are actively suffering. And here is a non-negotiable rule: every single Tier 1 alert must include a working link to a runbook — a concise cheat sheet detailing what the alert means, how to triage it, and who to escalate to. If you cannot write a simple runbook for the alert, you do not understand the failure mode well enough to be paging humans for it.

Tier 2 — Ticket (Business hours backlog). This is where causes and early warning signs belong. A disk that is projected to fill up in 10 days, an SSL certificate expiring in two weeks, or an error budget leaking slowly. Send these straight to Jira, Linear, or Slack during office hours. Nobody needs an adrenaline rush at midnight for a disk that still has 80 gigabytes left.

Tier 3 — Dashboard (Investigative context). Everything else. CPU spikes, memory utilization trends, network packet counts, garbage collection pauses. These are invaluable clues while you are debugging an active incident, but completely useless as standalone interruptions. Your CPU graphs belong on a Grafana dashboard (our visual metric graphing board). They always belonged there; we just stopped pretending they were emergencies.

One operational discipline I strictly enforce: if an alert pages an engineer and the response is, “Oh yeah, that always fires around this time, just resolve it,” that alert goes on trial. It gets rewritten or deleted within one week. Zero zombie alerts allowed. Every page must demand action, or it must die.

Burn-rate alerting: the upgrade

Once you have separated symptoms from causes, the natural next level is burn-rate alerting based on your Service Level Objectives (SLOs). An SLO is simply your agreed reliability target — for example, saying “99.9% of API requests must succeed over any 30-day window.”

The remaining 0.1% of permitted failures is called your error budget. Think of your error budget like a monthly spending allowance for downtime. Burn-rate alerting tracks how quickly you are spending that allowance:

  • Fast burn (spending your entire month’s budget in just 2 or 3 hours): this is a raging fire. Page the on-call engineer immediately.

  • Slow burn (spending your budget at a pace that will drain it over several days or weeks): create an automated ticket for the team to investigate during normal working hours.

Burn-rate alerting: fast burn pages, slow burn tickets.

A 1% error rate means different things at 99.9% vs 99% SLO. Burn-rate alerts know the difference; static thresholds don’t.

This approach beats static thresholds every single time because it dynamically adapts to your service’s promised reliability level. If you run a payment service with a strict 99.99% SLO, a 0.5% failure rate is an existential emergency. But if you run an internal experimental analytics tool with a 95% SLO, that same 0.5% blip is negligible. Burn-rate alerts understand that distinction automatically; static percentage limits are blind to it.

You can wire these up easily using Prometheus recording rules — precomputed metric queries that run in the background to calculate complex burn rates smoothly without slowing down your database — and display them cleanly on Grafana. Setting up the math takes an afternoon. Hunting down and retiring legacy alerts might take a couple of weeks, but the payoff is immense: when your phone finally chimes, your adrenaline spikes for a legitimate incident, not a phantom.

The weekly ritual nobody wants but everyone needs

Here is the unsexy secret behind teams with quiet pagers: they treat alerting maintenance like personal hygiene.

Every week, spend 15 minutes reviewing the past seven days of alerts. Look at what paged, what got immediately silenced, and what auto-resolved before anyone opened a terminal. Every ignored page is not a badge of honor for surviving on-call — it is a bug in your telemetry system.

Keep a tiny scoreboard:

  1. Total pages received per week.

  2. Percentage acknowledged within 5 minutes.

  3. Percentage of alerts that led to a concrete engineering fix.

Whenever you notice your team’s acknowledgement time creeping up from 3 minutes to 15 minutes, do not scold the team. That delay is textbook alert fatigue setting in. It means noise has crept back into the pipes, and it is time for a pruning session. Monitoring code rots just like production code. Keeping it clean is the job.

The takeaway

If your team dreads their turn holding the pager, don’t blame the people. Blame the alerts. Page strictly on symptoms, file tickets for internal causes, leave the hardware metrics on dashboards, and ruthlessly delete anything that cries wolf.

Your team’s sanity — and your own sleep at 3:00 AM — depends on it.


Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

  • Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

  • Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

I write about the unsexy work that keeps systems alive at 3am — monitoring, SRE, Terraform, and platform engineering.

Next time an alert wakes you up and you just hit ‘snooze’, do yourself a favor: delete the alert or turn it into a Jira ticket. Your future 3am self will thank you.

Examples are illustrative — validate in your own environment before running in production.

Top comments (0)