Alertmanager Routing Fixes to Cut Prometheus Alert Fatigue

Originally published on kuryzhev.cloud

---

A pager goes off at 3 a.m. with dozens of notifications for the same disk-pressure event on one node, spread across three teams who all have the node exporter alert wired to their phones. By the time someone actually reads one of them, the disk has already recovered and the incident channel is full of "ack, resolved" messages nobody trusts anymore. This is alert fatigue in Prometheus Alertmanager routing, and it is a common reason on-call engineers start muting entire Slack channels instead of fixing root causes.

The frustrating part is that Alertmanager was built specifically to solve this problem. Grouping, inhibition, and silences exist precisely to stop duplicate noise. When teams still drown in alerts, it is commonly a routing configuration problem, not a Prometheus rule problem.

Failure scenario

Picture a mid-sized platform team running Prometheus with fifteen exporters across sixty nodes. Someone adds a new rules.yml file for disk space, memory pressure, and pod restarts, all firing independently. The route block in alertmanager.yml still uses the default configuration copied from a tutorial two years ago: a single receiver, no group_by customization beyond the default, and no inhibition rules.

During a real node failure, kubelet reports NotReady, the node exporter fires DiskPressure and MemoryPressure, and every pod scheduled on that node starts restarting. Each condition maps to a separate alert rule. With grouping left at the tool's default of grouping by all labels, near-identical alerts rarely merge, so Alertmanager ends up sending a batch of individual notifications instead of one grouped message covering the failing node.

The on-call engineer receives a wall of pings, most of which are symptoms of the same root cause. Over weeks, this repeats for every flapping node, every deployment rollout, and every cert renewal. Engineers start snoozing the pager app notification sound entirely, which is the exact failure mode alerting is supposed to prevent — a real incident gets treated the same as background noise.

Why it happens

Alert fatigue in Prometheus Alertmanager routing commonly traces back to a handful of root causes, and they compound each other.

First, grouping is too granular or missing entirely. If group_by is left unset, Alertmanager groups by all labels, which effectively treats every distinct label combination as its own notification. Even when group_by is set explicitly, including alertname means alerts with different names never merge into one notification regardless of what other labels they share — DiskPressure and MemoryPressure on the same node stay separate. Consolidating correlated symptoms into fewer notifications requires dropping alertname from group_by and grouping by scope labels like cluster and instance instead, accepting that alerts of different types on the same node will then land in one group.

Second, inhibition rules are absent. Inhibition tells Alertmanager to suppress lower-severity alerts when a related higher-severity alert is already firing. Without it, a NodeDown alert and every downstream symptom alert (pod restarts, service unavailable, high latency) all fire in parallel instead of NodeDown suppressing the rest. Alertmanager cannot synthesize a root-cause summary message on its own — the realistic outcomes are a single batched notification containing many related alerts, or fewer notifications overall through inhibition.

Third, severity labels are inconsistent or unused. If every alert rule uses severity: warning because nobody agreed on a taxonomy, the routing tree cannot distinguish "wake someone up" from "check this during business hours." Watch out for teams that add severity labels late and forget to backfill existing rules — half the alerts route correctly and half don't, which is worse than having no severity labels at all because it looks fixed when it isn't.

A fourth, quieter cause: repeat_interval left at its default. If it is shorter than the actual mean-time-to-resolve for a given alert class, the same unresolved incident re-notifies every hour and gets mentally filed as spam. Note that a child route's repeat_interval can be overridden independently, but it still inherits group_interval from its parent unless that is also set explicitly.

The fix (with code)

The fix has two parts: tighten grouping and timing, then add inhibition so related alerts collapse into fewer notifications. The example below is a partial config — a working file also needs global and fully defined receivers for every name referenced in route.

route:
  receiver: default-slack
  group_by: ['cluster', 'namespace', 'instance']  # correlate by scope, not by rule name
  group_wait: 30s        # wait to batch near-simultaneous firings
  group_interval: 5m     # minimum gap between updates to an existing group
  repeat_interval: 4h    # avoid re-paging for the same unresolved issue every hour
  routes:
    - matchers:
        - severity="critical"
      receiver: pagerduty-oncall
      group_wait: 10s
      repeat_interval: 1h   # overrides parent; still inherits group_interval: 5m
      continue: false       # default behavior for a matched route; listed for clarity
    - matchers:
        - severity="warning"
      receiver: slack-warnings
      repeat_interval: 12h
    - matchers:
        - alertname="Watchdog"   # always-firing heartbeat rule
      receiver: null-receiver    # confirms the pipeline is alive without paging anyone

receivers:
  - name: default-slack
    slack_configs:
      - api_url: "https://hooks.slack.com/services/PLACEHOLDER"
        channel: "#alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: "PLACEHOLDER_KEY"
  - name: slack-warnings
    slack_configs:
      - api_url: "https://hooks.slack.com/services/PLACEHOLDER"
        channel: "#alerts-warnings"
  - name: null-receiver   # no notifier configs: matched alerts are discarded

Grouping by scope instead of by alertname means DiskPressure and MemoryPressure notifications for the same node land in one message body rather than two, but different alert types on unrelated nodes stay in separate groups because cluster/namespace/instance differ.

Next, add inhibition so a node-level failure suppresses the symptom alerts it causes. This is the piece most teams skip, and label overlap between source and target matters more than it looks:

inhibit_rules:
  - source_matchers:
      - alertname="NodeDown"
    target_matchers:
      - severity="warning"
    equal: ['node']   # node exporter alerts share 'node', not 'instance', with symptom alerts — verify against your label schema

  - source_matchers:
      - alertname="KubeAPIDown"
    target_matchers:
      - alertname=~"KubePodCrashLooping|KubeDeploymentReplicasMismatch"
    equal: ['cluster']  # confirm these exact rule names exist in your kube-prometheus-stack ruleset

Verify with amtool config routes test --config.file=alertmanager.yml severity=critical against representative label sets before rolling this into production, and use amtool check-config alertmanager.yml to catch syntax and reference errors before reload — an undefined receiver name or a matcher targeting a label that doesn't exist in your rules will fail validation, not fail silently. Both source_matchers/target_matchers and the unified matchers list require Alertmanager 0.22 or later; earlier versions only understand source_match/target_match and match/match_re, which still work in current releases but are discouraged in favor of the newer syntax. The official Alertmanager configuration reference documents every matcher and timing field for the version in use.

Prevention checklist

Fixing one bad routing tree solves today's fatigue; a checklist keeps it from creeping back in over the next quarter of new alert rules.

• Standardize a severity taxonomy before writing new rules. Two or three tiers (critical, warning, info) is enough; document what each means for response time.
• Group by scope, and decide deliberately whether alertname belongs in group_by. Keeping it in keeps notifications per-rule; dropping it merges different alert types sharing a scope into one message.
• Add inhibition rules whenever a new "root cause" alert is introduced. Check that equal lists labels actually shared between source and target alerts — a mismatch means the rule silently suppresses nothing.
• Set repeat_interval per severity tier, not globally. Critical alerts can repeat hourly; warnings should not. Remember child routes inherit unset fields like group_interval from the parent.
• Run a quarterly noise audit. Pull notification counts per alertname from your paging tool and question anything firing repeatedly within a short window — that is usually a threshold problem, not a routing problem, but it belongs on the same review.
• Test config changes with amtool before reload, always passing --config.file and sample labels, and keep a staging Alertmanager instance if the routing tree is complex enough to warrant it.
• Watch out for silences that outlive their incident. A silence created during a maintenance window and never removed quietly disables alerting for that scope indefinitely — audit active silences alongside the noise audit.

None of this requires new tooling, only a deliberate pass over configuration that most teams write once and never revisit. For more on pairing this with dashboard-side alert visibility, see the Prometheus and Grafana setup notes on kuryzhev.cloud.

Related

• Prometheus and Grafana Docker Compose Setup That Sticks
• Loki Retention Tuning: 7 Fixes for Runaway Log Storage Costs
• Kubernetes HPA and VPA Rightsizing: Fixing Autoscaling Thrash