DEV Community

Cover image for Prometheus Alert Fatigue: Symptom-Based Alert Design
Mustafa ERBAY
Mustafa ERBAY

Posted on • Originally published at mustafaerbay.com.tr

Prometheus Alert Fatigue: Symptom-Based Alert Design

#observability #sistemmimarisi #software

In a monitoring infrastructure that generates hundreds of alerts a day, the moment engineers stop reading notifications and start muting them, system observability is completely lost. Alert fatigue is the direct consequence of writing a separate alert rule for every possible anomaly in the system. The solution is a symptom-based alert architecture that focuses not on "Why did CPU hit 90%?", but on "Can users actually get a response from the service right now?"

Symptom-based alert design makes the metrics measuring user-facing outages, latency, and error rates the primary notification triggers, rather than tracking the isolated states of subcomponents. Root causes are not used to fire alerts; instead, they serve as diagnostic dashboard metrics and log correlations once a symptom alert is triggered.

Diagram

Why Do Cause-Based Alerts Fail?

A cause-based alerting strategy stems from the engineering reflex of "writing a rule for every part that could possibly break." Waking up the on-call engineer whenever disk usage reaches 80% on a server, CPU consumption exceeds a specific threshold, or Redis memory usage climbs is a textbook example of this mindset. However, high CPU usage on a server does not necessarily mean the service provided by that server is degraded; often, it simply indicates that the system is utilizing its hardware resources efficiently.

The fundamental problem created by this approach is that most alerts are non-actionable. When an on-call engineer gets paged at midnight for "Server CPU 85%", checks the system, and sees that users are experiencing zero errors and normal latency, they will completely tune out that alert channel within two weeks. Consequently, when a real database deadlock or network outage occurs, the critical alert gets lost in the noise.

The table below summarizes the operational differences between both approaches:

Criterion Cause-Based Symptom-Based
Trigger Focus CPU, Memory, Disk, Thread count HTTP 5xx rate, p99 Latency, Queue backlog
Alert Volume High (Every component screams individually) Low (Fires only when users are impacted)
Actionability Ambiguous (Often requires no action) Clear (Users cannot use the service)
Maintenance Burden Thresholds must be constantly retuned Tied to SLO/SLI targets, stable
Coverage Limited to known failure modes Catches unknown/novel failures as well

Core Principles of Symptom-Based Alerting: RED and USE

When setting up symptom-based monitoring in a distributed system, two foundational methodologies serve as your guide: RED (Rate, Errors, Duration) for request-driven services, and USE (Utilization, Saturation, Errors) for resource-driven infrastructure components. Applying these two approaches to the appropriate architectural layers cuts out unnecessary noise at the source.

The RED method should be applied to layers directly exposing APIs to users or other services. A service's request rate per second (Rate), how many of those requests fail (Errors), and how long requests take to complete (Duration) directly reflect its health. Regardless of the memory consumption of the container behind the service, if the 5xx error rate exceeds the defined Service Level Objective (SLO) boundary, you have a real symptom that must be addressed.

       +-------------------------------------------------------+
       |                  RED Method (Services)                |
       |  - Rate (RPS)                                         |
       |  - Errors (HTTP 5xx / gRPC Errors) -> ALERT           |
       |  - Duration (p95/p99 Latency)      -> ALERT           |
       +---------------------------+---------------------------+
                                   |
                                   v
       +-------------------------------------------------------+
       |             USE Method (Infrastructure Resources)     |
       |  - Utilization (CPU/RAM Usage)     -> DASHBOARD       |
       |  - Saturation (Queue/Load Average) -> WARNING/DASHBOARD|
       |  - Errors (Disk I/O, Dropped Pkts) -> DIAGNOSIS       |
       +-------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

On the infrastructure side (disks, network interfaces, database connection pools), the focus shifts to the USE method. But here is the critical distinction: Utilization alone should rarely be an urgent pager alert. The true focus must be Saturation (queuing/backlog). While 85% disk usage belongs on a dashboard, I/O requests queuing up on disk (I/O saturation) or running out of TCP sockets and dropping packets is an actionable symptom.

đź’ˇ Golden Rule: Page Only When Users Are Affected

The only condition that should wake an on-call engineer from their sleep is when the system cannot meet the promised Service Level Objective (SLO) to users, or when it is certain to fail completely in a short window (e.g., within 1-2 hours, such as a disk filling up at a rapid write rate).

Writing User-Centric Alerts with PromQL

When writing symptom-based rules in Prometheus, instead of relying on simplistic instantaneous thresholds (metric > 90), calculations should be based on rates and distributions spanning a defined time window. A well-crafted PromQL query must filter out transient spikes (jitter) and confirm the persistence of the failure.

1. Error Rate Alert

Instead of a simple error count, the error percentage relative to total traffic should be calculated. To prevent alerts from firing due to a single isolated error during low-traffic periods, a minimum request rate condition is included in the query:

groups:
  - name: service_symptom_alerts
    rules:
      - alert: HighHTTPErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"^5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 2
          and
          sum(rate(http_requests_total[5m])) > 10
        for: 3m
        labels:
          severity: critical
          tier: api
        annotations:
          summary: "API service returning high rate of 5xx errors"
          description: "Over 2% of incoming requests failed with 5xx in the last 5 minutes. Current error rate: {{ $value | printf \"%.2f\" }}%"
Enter fullscreen mode Exit fullscreen mode

In this rule, the for: 3m clause prevents the on-call engineer from getting paged unnecessarily over momentary 10-second network blips; it requires the issue to persist consistently for at least 3 minutes.

2. Latency / Duration Alert

Percentiles should always be used instead of average latency. An average value masks severe slowdowns experienced by a significant fraction of users. Using the histogram_quantile function, we measure p99 or p95 latency:

      - alert: HighRequestLatencyP99
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
          > 1.5
        for: 5m
        labels:
          severity: warning
          tier: backend
        annotations:
          summary: "p99 Request latency exceeds threshold"
          description: "p99 response time for {{ $labels.service }} has been above 1.5s for 5 minutes. Current value: {{ $value | printf \"%.2f\" }}s"
Enter fullscreen mode Exit fullscreen mode

3. Fast-Depleting Resources (Multi-Window Burn Rate)

For disk space, instead of a static 85% threshold, we use the predict_linear function to estimate when the disk will run out of space based on its current write rate. This ensures no alerts are generated at night for a disk that will take a month to fill, while fast action is taken for a disk that will fill up in 2 hours:

      - alert: DiskFillingFast
        expr: |
          predict_linear(node_filesystem_free_bytes{mountpoint="/"}[4h], 4 * 3600) < 0
          and
          node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.2
        for: 15m
        labels:
          severity: critical
          tier: infra
        annotations:
          summary: "Root filesystem will fill within 4 hours"
          description: "Root disk on instance {{ $labels.instance }} will be completely exhausted within 4 hours at the current fill rate."
Enter fullscreen mode Exit fullscreen mode

Filtering Noise with Alertmanager: Routing, Grouping, and Inhibition

Even when Prometheus rules are designed around symptoms, a major infrastructure failure can still trigger hundreds of alerts simultaneously. When a core backbone switch goes down, receiving separate "Service Unreachable" alerts for 40 servers and 200 containers behind it represents peak noise. This is where Alertmanager's grouping and inhibition capabilities become indispensable.

Diagram

Setting up proper inhibit_rules in Alertmanager prevents downstream cascading alerts from flooding your notification channels when a primary outage occurs.

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default-slack'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true
    - match:
        severity: warning
      receiver: 'slack-warnings'

inhibit_rules:
  # If the host is completely down, mute service alerts on that host
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: 'InstanceDown|HighHTTPErrorRate|HighMemoryUsage'
    equal: ['node', 'instance']

  # If the database is unreachable, mute latency alerts for backend services depending on it
  - source_match:
      alertname: 'PostgresDead'
    target_match:
      tier: 'backend'
    equal: ['environment']
Enter fullscreen mode Exit fullscreen mode

Here, group_wait: 30s waits 30 seconds after the first alert fires to bundle other related alerts in the same group into a single consolidated notification. Meanwhile, repeat_interval: 4h controls how frequently an unresolved issue re-notifies the channel, avoiding unnecessary spam.

ℹ️ Label Matching in Inhibition Rules

The equal parameter in the inhibit_rules block is critical. If you apply a global inhibition rule without verifying that the source and target alerts share the same instance or cluster label, a failure on Host A could accidentally silence a legitimate alert on Host B.

Actionable Alerts and Runbook Architecture

When an alert triggers, the message displayed on the on-call engineer's screen should not merely state a status—it must serve as a concrete remediation guide. Every minute an engineer spends searching for documentation asking "What does this alert mean and what should I do?" directly inflates your Mean Time to Recovery (MTTR).

Every alert rule must include two essential links: a focused Grafana Dashboard URL to inspect the relevant service metrics and a Runbook URL outlining initial mitigation steps.

      - alert: OrderProcessingLagSpike
        expr: |
          rabbitmq_queue_messages_ready{queue="order_processing"} > 5000
        for: 10m
        labels:
          severity: critical
          service: order-pipeline
        annotations:
          summary: "Order processing queue is backing up"
          description: "The number of unprocessed orders in the queue has exceeded 5000 for 10 minutes. Consumer services may be falling behind."
          runbook_url: "https://wiki.internal/ops/runbooks/order-queue-lag"
          dashboard_url: "https://grafana.internal/d/orders/order-pipeline-metrics?var-queue=order_processing"
Enter fullscreen mode Exit fullscreen mode

A well-designed runbook should answer these 4 questions directly and concisely:

  1. What symptom occurred? (e.g., Order queue is backing up; users cannot receive order confirmations.)
  2. What is the initial verification step? (e.g., Check consumer pod logs for OOM kills or connection timeouts.)
  3. What is the temporary mitigation? (e.g., Scale up consumer pods: kubectl scale deployment order-consumer --replicas=10)
  4. Root cause diagnosis: (e.g., Inspect payment gateway response times, check for PostgreSQL lock contention.)

Alert Hygiene and Lifecycle Management

Building a symptom-based monitoring infrastructure is not a one-off project; it requires continuous maintenance. Evolving architectures, newly added microservices, or refactored components can quickly render old alert rules obsolete.

       +-------------------------------------------------------+
       |                  Weekly Alert Review                  |
       |  - Which top 5 alerts fired most in the last 7 days?  |
       |  - Which alerts required ZERO action?                 |
       +---------------------------+---------------------------+
                                   |
                                   v
       +-------------------------------------------------------+
       |              Classification and Improvement           |
       |  - Non-actionable      -> DELETE or move to Dashboard |
       |  - Incorrect threshold -> Adjust PromQL / Time window |
       |  - Genuine symptom     -> Update Runbook              |
       +-------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

To maintain alert hygiene, incorporate these rules into your team's operational routines:

  • The Zero-Action Rule: Any alert that fired more than 5 times in the past month without requiring a single code change, service restart, configuration fix, or incident triage must be deleted immediately or converted into a dashboard metric.
  • Silencing Discipline: Temporary silences created in the Alertmanager UI must always include an expiration (TTL) and a link to an active tracking ticket or task. Indefinite silences are the leading cause of blind spots in production.
  • Alert Review Meetings: During weekly on-call handovers, review the total alert count and noise ratio. Any rule that woke up the on-call engineer unnecessarily should be revised immediately via a pull request.

Wrapping Up

Achieving sustainable observability with Prometheus and Alertmanager isn't about writing more alert rules; it's about monitoring the right architectural layer with the right methodology. Instead of drowning in the noise of thousands of cause-based rules, implementing symptom-based alerting grounded in RED and USE principles restores confidence for on-call teams.

Keeping infrastructure metrics (CPU, RAM, connection counts) on Grafana dashboards as diagnostic aids when symptoms fire—while reserving pages exclusively for moments when the user experience is genuinely compromised—is the foundation of operational resilience.

Official Resources

Top comments (0)