DEV Community

jun liu
jun liu

Posted on

My Server Was Down for 4 Hours and I Didn't Know: A 15-Line Prometheus Alert That Finally Pings My Phone

#prometheus

My Server Was Down for 4 Hours and I Didn't Know: A 15-Line Prometheus Alert That Finally Pings My Phone

Last month I found out about a production outage the worst way possible: a client told me. "Your API has been timing out since the night shift." I checked uptime, grabbed my phone, and realized my monitoring dashboard had shown "all healthy" the whole time. The check I'd set up was only looking at whether the process was running. The process was running. It just wasn't answering requests.

That's the trap most of us fall into: we monitor the thing we know how to measure, not the thing that actually breaks. Here's the 15-line alert rule that closed that gap for me — and the mental model that made it work.

The gap that cost me 4 hours

My alert looked like this:

- alert: InstanceDown
  expr: up == 0
  for: 5m
Enter fullscreen mode Exit fullscreen mode

up is Prometheus's flag for "did the last scrape succeed?" And that's the problem. Scrapes kept succeeding. Prometheus was reaching the target, the exporter was answering /metrics. So up stayed 1. But the application was stuck — a stuck thread pool, a hung upstream — while the metrics endpoint (a separate, healthy goroutine) kept returning numbers.

The dashboard said green. The service said "try again later." Both could be true at the same time. That's the gap.

The fix: assert on behavior, not on liveness

Stop asking "is the process alive?" and start asking "is the thing I actually ship working right now?" For an HTTP service, that's a black-box check: a real request, end to end, from outside the box.

This is where blackbox_exporter comes in. It's Prometheus's probe-on-the-wire: you give it a URL, it does a real GET/TCP/ICMP, and it reports back probe_success. Now you can alert on what a user would actually feel.

The 15-line rule 

groups:
- name: blackbox
  interval: 30s
  rules:
  - alert: ServiceUnreachable
    expr: probe_success{job="blackbox-http"} == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} stopped answering real requests"
      description: "Blackbox probe failing for 2m. Last status {{ $value }}. Check the app, not just the process."
Enter fullscreen mode Exit fullscreen mode

Three things in these 15 lines do all the heavy lifting:

  1. probe_success == 0 — alert on the probe failing, not on up == 0. This is the whole point.
  2. for: 2m — give it two minutes so a single flaky packet or a brief DNS wobble doesn't page you. Two minutes of sustained failure is a real problem.
  3. annotations.description — this is the text that lands in your phone. Make it say what to do, not what broke. "Check the app, not just the process" is the exact lesson from the incident, baked into the alert.

Wiring it to a phone, in 30 minutes

An alert is only useful if it reaches a human. The chain is:

Prometheus (evaluates rule) → Alertmanager (routes, groups, silences) → your phone (push / SMS / webhook)
Enter fullscreen mode Exit fullscreen mode

Alertmanager's route config is where you say who and how. For a phone, I'd start with the cheapest thing that works:

  • Push (pushgateway not it — use a push service like ntfy, Bark, or a cheap IaaS SMS). ntfy is the fastest to test: one webhook URL, works on Android and iOS, no account. Point Alertmanager's webhook_configs or opsgenie/pagerduty at it.
  • Group the alerts (group_by: [job, instance]) so 40 failing instances don't send you 40 pings. You get one "these 40 are down," not 40 screams.
  • Silence the maintenance window before you touch the box, or you'll be mid-deploy watching your phone light up.

The mental model worth keeping

I keep a one-line rule taped to my monitor now:

Monitor the outcome your users feel, not the inputs you control.

up is an input I control. "A real HTTP 200 within 30s" is the outcome my users feel. Alert on the outcome. If you can't measure the outcome, build a check that approximates it (a real probe). The dashboard telling you "everything's fine" while a client is the one who tells you the site is down is the specific failure mode this whole setup is designed to kill.

A 15-line rule and 30 minutes of wiring saved me from the second 4-hour blind spot. It won't prevent every incident — but it ends the one where you're the last to know.

Tags: prometheus, monitoring, alertmanager, devops, ops

More on monitoring and field troubleshooting: jiunotes · ops notes

Top comments (0)