One Noisy Alert, Tuned Without Hiding a Real Outage

Your phone buzzes at 3:12 AM: "bus-times API down." By the time you open your laptop, it's back up. It happened twice last week too, and each time you went back to sleep. That's the dangerous part: one night it will be real, and you'll assume it isn't.

Fact-checked against the official sources on 29 October 2026.

An alert is a rule that sends a message (a page) to whoever is on call when something looks wrong. A noisy alert fires often for problems that fix themselves. The Prometheus team's advice is short: avoid pages where there's nothing to do, and allow some slack for small blips. Here's how to do that for one alert without making it blind to a real outage.

Pick the noisiest alert

1. Look at your pages from the last two to four weeks.
2. Count which alert fired most often with nothing for you to do.
3. Take just that one. Tuning everything at once makes it hard to tell which change helped.

Look at how long each blip lasted

For each time it fired, note how long the problem lasted before it cleared by itself. In our made-up example, the bus-times API dropped out for under a minute each time, usually during a nightly restart.

Then write down the shortest real outage you'd want to be woken for. Say it's anything over 5 minutes.

Read the rule as it is now

groups:
  - name: bus-times
    rules:
      - alert: BusTimesApiDown
        expr: up{job="bus-times-api"} == 0
        labels:
          severity: page

up is a series Prometheus stores for every target it scrapes (collects metrics from): 1 if the target was reachable, 0 if the scrape failed. This rule has no for: line. Per the Prometheus alerting rules docs, a rule without for becomes active on the very first check where the condition is true. One failed scrape is enough to page you.

Add a wait with for:

      - alert: BusTimesApiDown
        expr: up{job="bus-times-api"} == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "bus-times API unreachable for 5 minutes"

With for: 5m, Prometheus checks that the condition is still true on every evaluation for 5 minutes before it fires. While it waits, the alert is pending, not firing. A 40-second blip goes pending, then clears, and nobody gets woken. A real outage still pages you, 5 minutes in.

Choose the for value from your notes: longer than the blips, shorter than the outage you care about.

Stop it flapping if it bounces

Sometimes a service goes down, comes back for a moment, then goes down again. The alert fires, resolves, and fires again. That's called flapping. For this, the docs describe keep_firing_for. It holds the alert in the firing state for the length you choose, counted from the last moment the condition was true:

        for: 5m
        keep_firing_for: 2m

You only need this if you actually see flapping. Leave it out otherwise.

Check you can still catch a real outage

1. Find a past real incident for this service and check it lasted longer than your new for value. If it didn't, your wait is too long.
2. If your team uses it, Prometheus has unit testing for rules with promtool, so you can test the rule against made-up data before it goes live.
3. Write the change and the reason in the rule's annotations or your team channel, so the next person knows the 5 minutes was a choice.

Where this fits at DevGrid Staffing LLC

DevGrid Staffing LLC places engineers on teams that run production systems, and our handovers include runbook drills with the client's on-call people (how we work). In software engineering, an engineer who quiets one bad alert carefully, and can show why it's still safe, saves a whole rotation a lot of sleep.

One thing to try today

Find the alert that paged you most last month. Check whether it has a for: line, and how long its blips usually last.

---

From the DevGrid Staffing LLC notebook · devgridstaffing.com