DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com on

The Phantom Alert: Why Prometheus Stays Silent When Production Is on Fire

Metric streams flowing into a monitoring console — the pipeline your alerts depend on.

Metric streams flowing into a monitoring console — the pipeline your alerts depend on.

Pull up a chair, grab some chai, and let’s talk about a quiet nightmare sitting inside almost every Kubernetes cluster right now. Somewhere in your monitoring stack, there is an alert rule that will never wake anyone up. Ever.

Not because your code is rock solid. Not because your infrastructure is immortal. But because the metric your alert is looking for simply does not exist. Nobody botched the PromQL math (Prometheus Query Language, the query syntax used to ask Prometheus for data). Nobody set the threshold to an impossible number. The target was simply never scraped — meaning Prometheus never reached out over HTTP to grab the numbers — and when Prometheus checks a metric that isn’t being collected, it gets an empty answer. And in the monitoring world, “nothing to report” looks identical to “everything is perfectly fine.”

This is what we call the coverage gap. It is easily the most underrated failure mode in Prometheus setups. We spend hours in sprint reviews arguing over whether CPU alerts should fire at 80% or 85%, but almost nobody stops to ask the boring, unsexy question: Is Prometheus even talking to this service in the first place? Let’s fix that mindset today. We will walk through auditing coverage using up and absent(), dodge the classic Kubernetes scrape traps that silently drop your pods, build alerts that humans can actually understand at 3 AM, and decide what truly deserves to ring your phone.

The coverage gap: an alert that can never fire

Think of it like buying an expensive fire alarm, mounting it proudly on the ceiling, and forgetting to put in the batteries. There is no fire alarm sound, the room is quiet, so you assume you are safe — until the kitchen is literally burning.

Here is how this failure plays out under the hood, step by step. An alert rule in Prometheus is just a query evaluated on a regular timer (say, every 30 seconds). If your query looks for a metric, but Prometheus has zero time-series data stored for it, the engine returns an empty vector. It doesn’t throw a bright red syntax error. It doesn’t send a warning to Slack. It just gets zero rows back.

When you evaluate a condition like > 0.05 against literally nothing, the result is false. So the alert sits quietly in the INACTIVE state. On your Prometheus Alerts UI, it sits in a soothing sea of green rows, looking just as healthy as a hundred other working alerts.

Here is a classic setup that bites teams every single month. A squad ships a shiny new checkout API. Being responsible engineers, they write an alert: api_high_error_rate > 5%. That error metric is supposed to come from a ServiceMonitor (a custom Kubernetes resource from the Prometheus Operator that tells Prometheus which Kubernetes Services to scrape). But someone made a tiny typo in the ServiceMonitor’s label selector — it was looking for app: checkout-api, but the actual deployment had app.kubernetes.io/name: checkout-api.

Kubernetes doesn’t complain. The deploy turns green. But Prometheus discovers zero pods. The internal counter up{job="checkout-api"} has zero series. The alert rule evaluates against empty air, stays green, and the team only finds out on a Friday evening when payment errors hit 40%, customers are raging on social media, and nobody’s phone has buzzed once.

Before: alert against an unscraped metric stays silent. After: coverage verified with a companion check.

Before: alert against an unscraped metric stays silent. After: coverage verified with a companion check.

An alert that can never fire looks identical to a system that is healthy. You find out during the incident.

The engineering rule here is simple: for every critical alert you write on a scraped metric, you must write a companion alert that screams if that metric disappears. Verify coverage first; tune thresholds second. Let’s look at how to build that guardrail.

Auditing coverage: up, absent(), and the Targets page

Prometheus gives you two core primitives for verifying that targets exist, and they catch completely different kinds of breakage. You need to know the difference cold:

# Scrape is failing: the target exists in config but can't be reached.
# up is 1 on success, 0 on failure — per target.
up{job="checkout-api"} == 0
# Target vanished entirely: no series at all for this job.
# absent() returns a 1-element vector with value 1 when its input is empty,
# deriving labels from the equality matchers in the selector.
absent(up{job="checkout-api"})
# Same idea over a window: only fire if it's been gone the whole time,
# so one missed scrape doesn't page anyone.
absent_over_time(up{job="checkout-api"}[10m])
Enter fullscreen mode Exit fullscreen mode

Let’s break down what these queries actually do:

  • up{job="checkout-api"} == 0 catches a target that Prometheus knows about, but cannot talk to. Maybe your application container crashed, the port is blocked by a NetworkPolicy (firewall rules inside your Kubernetes cluster), or your TLS certificates expired. Prometheus knocked on the door, but nobody answered.

  • absent(up{job="checkout-api"}) catches an entirely different beast: the target that disappeared from Prometheus’s brain altogether. If your ServiceMonitor selects nothing, or someone renamed the job label during a Helm chart cleanup, Prometheus isn’t failing to scrape — it doesn’t even know it was supposed to scrape! In that scenario, up == 0 returns nothing at all, so it stays silent. absent() is built specifically to flip an empty result into a firing alert.

Here is a technical detail that catches people off guard: absent() can only copy labels from exact match operators (=). If you write absent(up{job="api", instance=~".*"}), the resulting alert only gets the label {job="api"}; regex matchers don’t populate labels in the output. Also, if you run absent() against a job with ten replicas, it only fires if all ten replicas vanish. If nine crash and one survives, absent() stays happy. That is why for per-pod health you use up{job="checkout-api"} == 0 with a duration threshold, while absent() guards the entire job pipeline.

The fastest audit in the world costs zero dollars: open the Prometheus web UI, click on Status → Targets , and check that your job shows UP with a healthy number of endpoints. Do this before you merge your PR, not while customers are complaining. Once verified, lock it down with a companion alert in Git:

# Companion coverage alert: fires if the scrape target disappears.
# Pair one of these with every alert that depends on this job's metrics.
- alert: CheckoutApiScrapeMissing
  expr: absent(up{job="checkout-api"})
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "No scrape targets for job checkout-api"
Enter fullscreen mode Exit fullscreen mode

This rule evaluates absent(up{job="checkout-api"}). If Prometheus doesn’t see a single active target for this job for 10 straight minutes, it escalates as a critical page. Now, if someone deletes a Service or breaks label selectors, your on-call gets notified immediately instead of sitting in the dark.

The silent killers in the scrape config

Most monitoring gaps aren’t caused by complex network failures. They happen because of tiny YAML mismatches that fail silently without producing a single error log:

  • ServiceMonitor selector matches nothing: The selector.matchLabels inside your ServiceMonitor does not match the labels on your Kubernetes Service object. Kubernetes and the Prometheus Operator won’t log an error — having a selector that matches zero services is completely valid syntax. Verify this right from your terminal by checking actual service labels: kubectl get svc --show-labels.

  • Endpoint port is a raw number instead of a port name: In Prometheus Operator CRDs (Custom Resource Definitions, which are custom Kubernetes object types), the endpoint’s port property expects the name of the port defined on your Service (like metrics or http-web), not a raw number like 8080. If you pass a bare number where a string name is expected, Prometheus quietly drops the target.

  • PrometheusRule labels don’t match the ruleSelector: If you use the popular kube-prometheus-stack collection, your Prometheus instance doesn’t just slurp up every PrometheusRule in the cluster. It checks a filter called ruleSelector (often looking for a label like release: kube-prom). If you forget that label, your rule file gets applied to the cluster cleanly via kubectl, but Prometheus ignores it completely. Always check the Rules tab in Prometheus to confirm your rule is actually loaded into memory.

  • Namespace scoping: If your Prometheus Operator is set up to only watch specific Kubernetes namespaces, placing your monitor or service in an unmonitored namespace makes it completely invisible. No error is thrown; the scrape simply never happens.

The full alert pipeline: scrape targets to Prometheus to Alertmanager to receivers.

The full alert pipeline: scrape targets to Prometheus to Alertmanager to receivers.

A ServiceMonitor that selects nothing produces no error. That’s the trap — the config is valid, the deploy is green, and the target was never scraped.

When an alert fails to fire, don’t guess. Trace the pipeline above from left to right. First, check the Targets page in Prometheus (are metrics being collected?). Second, run the raw PromQL query in the Graph console (are the numbers there?). Third, check the Rules page (is the rule loaded and evaluating?). Fourth, inspect Alertmanager (did the notification match the right routing tree?). Nine times out of ten, the problem dies at step one.

Anatomy of an alert rule

An alert rule that actually helps you survive an on-call rotation requires four specific ingredients: an accurate PromQL expression, a for: buffer, routing labels, and human-friendly annotations. Here is what a clean PrometheusRule looks like:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-api-alerts
  labels:
    release: kube-prom # must match Prometheus' ruleSelector, or the rule never loads
spec:
  groups:
    - name: checkout-api.availability
      interval: 30s
      rules:
        - alert: CheckoutApiHighErrorRate
          expr: |
            sum(rate(api_http_errors_total{job="checkout-api"}[5m]))
              /
            sum(rate(api_http_requests_total{job="checkout-api"}[5m]))
            > 0.05
          for: 5m
          labels:
            severity: page
            team: checkout
          annotations:
            summary: "Error rate above 5% on checkout-api ({{ $value | humanizePercentage }})"
            description: "Elevated 5xx ratio for 5 minutes on {{ $labels.instance }}."
            runbook_url: "https://wiki.example.com/runbooks/checkout-high-error-rate"
Enter fullscreen mode Exit fullscreen mode

Let’s explain what each section is doing:

  • Labels route; annotations inform: Remember this rule forever. Alertmanager uses labels (like severity: page and team: checkout) to decide where to send the alert (PagerDuty, Slack, or email). annotations (like summary, description, and runbook_url) are strictly for the human engineer reading the notification at 3 AM. Never use annotations for routing, and never put credentials or passwords in either!

  • The for: 5m window is load-bearing: If your error rate spikes for two seconds because a single pod restarted, you do not want your phone buzzing. The for: clause tells Prometheus: keep this alert in a PENDING state, and only escalate to FIRING if the bad condition persists for 5 consecutive minutes. It absorbs flaps and transient hiccups.

  • Templates provide immediate context: Notice {{ $value | humanizePercentage }} in the summary. When the alert hits your screen, it will literally say “Error rate above 5% on checkout-api (14.2%)” instead of making you log into Grafana just to see how bad things are.

Once Prometheus fires an alert, Alertmanager takes over routing. Alertmanager uses a tree structure. Note that modern Alertmanager syntax uses matchers: instead of the deprecated match: syntax:

# alertmanager.yml (excerpt)
route:
  receiver: 'default'
  group_by: ['alertname', 'team']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: 'pagerduty-critical'
      continue: true # also fall through to the ticket route below
    - matchers: [severity="ticket"]
      receiver: 'slack-alerts'
    - matchers: [severity="info"]
      receiver: 'email-digest'
Enter fullscreen mode Exit fullscreen mode

When an alert arrives, Alertmanager walks down this tree from the top. The first matching route wins, unless you set continue: true, which allows the alert to keep falling through to send notifications to secondary channels (like logging to Slack while also waking someone on PagerDuty). Meanwhile, group_by: ['alertname', 'team'] groups related failures together, so if 25 pods start throwing errors simultaneously, you get one consolidated alert instead of 25 separate notifications blowing up your phone.

Pageable vs. ticket-worthy: the fatigue guards

Coverage tells you whether your alert can fire. Severity tells you whether it should wake a human being in the middle of the night. Most alert fatigue (the dreaded state where engineers start ignoring pages because there is too much noise) isn’t caused by bad math — it is caused by bad classification. When non-urgent warnings are routed to the on-call pager, people stop taking alerts seriously.

The severity pyramid: critical pages, warning tickets, info goes to the dashboard.

The severity pyramid: critical pages, warning tickets, info goes to the dashboard.

Page less, mean it. A pager that only fires for real emergencies gets answered in minutes; a pager that fires for everything gets muted.

Three clean tiers are all you need to keep your sanity:

  • Critical / Page: Requires immediate human action right now, day or night. The production service is down, users cannot complete checkouts, data loss is imminent, or your Service Level Objective (SLO, your formal reliability target) is actively burning down.

  • Warning / Ticket: Action is needed, but it can comfortably wait until morning standup or office hours. A disk is at 75% capacity, a Kubernetes node is showing memory pressure, or an SSL/TLS certificate expires in 14 days. Do not wake people up for things that can wait four hours.

  • Info / Log: Contextual events meant for dashboards, daily digests, or Slack channels. A new deployment finished, an autoscaler added three pods, or a canary rollout shifted traffic.

Where do our coverage alerts fit in this hierarchy? Make no mistake: a vanished scrape target is a critical page. If Prometheus cannot scrape your checkout API, you are not just missing an error graph — you are completely blind. Every single alert sitting downstream of that service is currently useless decoration. Complete blindness in production is an emergency.

The alert-review checklist

Before you merge any PR that introduces new alerts, spend five minutes running through this checklist. It will save you from painful post-mortems later:

  • [] Coverage: The metric actually exists inside Prometheus right now — test it in the query browser, never assume.

  • [] Companion check: An absent() or up == 0 alert is configured to guard the scrape target this alert relies on.

  • [] Targets page: The job shows UP in Prometheus status with the expected number of active pods.

  • [] Selectors: ServiceMonitor or PodMonitor matchLabels match the exact labels on your Kubernetes Service.

  • [] Port naming: The endpoint port in your monitor references the Service’s named port, not an unlinked raw number.

  • [] Rule loading: The PrometheusRule labels match your Prometheus instance’s ruleSelector and show up on the Rules UI.

  • [] Severity label: Defined explicitly using your team’s agreed tiers (page, ticket, or info).

  • [] Pending window: A sensible for: duration is configured so transient spikes don’t cause notification flapping.

  • [] Runbook: The runbook_url annotation points to a live wiki page that explains remediation steps clearly.

  • [] Routing: The alert labels match a valid path in alertmanager.yml with an active receiver destination.

  • [] Page-worthy test: Is the problem urgent, actionable, and impacting users? If the answer to any of these is no, do not route it to the pager.

  • [] No secrets: Annotations and labels are free of authentication tokens, passwords, and private connection strings.

The takeaway

Bulletproof alerting follows a strict sequence: verify scrape coverage first, categorize severity second, configure Alertmanager routing third — and only then write your clever threshold formulas. A coverage companion alert is the easiest safety net you will ever write: absent(up{job="checkout-api"}) takes an invisible outage and turns it into an actionable alert with an exact runbook.

Do this today: pick your team’s five most important services. Verify that their metrics actually exist in Prometheus, confirm that their targets are showing UP, and verify you have a companion coverage rule armed. If any of those five fails the test, congratulations — you just caught a silent outage waiting to happen.


Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

  • Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

  • Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Practical guides on monitoring, SRE, Terraform, and platform engineering for high-scale systems.

Your turn: Run your five most important alerts through the coverage checklist above and fix the first gap you find. Leave a comment or reach out with the one that surprised you — the trickiest scrape-config traps make great follow-up teardowns.

Examples are illustrative — validate in your own environment before running in production.

Top comments (0)