DEV Community

DevKit
DevKit

Posted on

A free runbook for 44 Prometheus alerts (Linux, Docker, Kubernetes, TLS)

#monitoring #prometheus #kubernetes #devops

Most Prometheus setups I see have alerts but no answer to the 3 a.m. question: "OK, it fired. Now what?" The fix is boring and effective: every alert gets a runbook_url annotation that points to one short page with the same structure every time.

I wrote such a runbook for 44 common alerts (Linux hosts via node-exporter, Docker via cAdvisor, Kubernetes via kube-state-metrics, websites and TLS via blackbox-exporter) and put it online for free:

👉 Prometheus alert runbook (free, English)

The format

Every section answers the same five questions:

  1. What it means: in plain words, including the exact condition.
  2. Common causes: the usual suspects, most likely first.
  3. Diagnostics: copy-paste commands.
  4. How to fix.
  5. When it is OK to silence: so silences get a reason and an expiry instead of becoming "monitoring switched off".

Example: disk will fill within 24 hours

A static "disk > 90%" alert is either too late or too noisy. A trend-based one catches the runaway log before it bites:

- alert: NodeDiskWillFill
  expr: |
    predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
    and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
      / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"} * 100 < 40
    and node_filesystem_readonly{fstype!~"tmpfs|overlay|squashfs"} == 0
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Disk {{ $labels.mountpoint }} will fill up within a day ({{ $labels.instance }})"
    runbook_url: "https://devkit-shop.surge.sh/runbook/en/#nodediskwillfill"
Enter fullscreen mode Exit fullscreen mode

The < 40 guard matters: without it, a big, mostly empty disk with a brief write burst will page you for nothing.

And the runbook section it links to starts with diagnostics you can paste straight away:

df -h <mountpoint>
du -xh --max-depth=2 <mountpoint> 2>/dev/null | sort -h | tail -n 20
find <mountpoint> -xdev -type f -mmin -60 -size +100M -exec ls -lh {} +
Enter fullscreen mode Exit fullscreen mode

Then the important judgement call: is the growth steady (rotation is missing, a leak) or a one-off jump (an import or a backup)? For a known one-off, silence it for a few hours, not days.

Example: CrashLoopBackOff 

kubectl -n <ns> describe pod <pod>              # Events and Last State
kubectl -n <ns> logs <pod> -c <container> --previous
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -n 20
Enter fullscreen mode Exit fullscreen mode

--previous is the flag people forget: it shows the log of the run that actually crashed, not the fresh one that is about to.

Tips that apply to any alert set

  • Put runbook_url in every rule and render it as a link in your Telegram/Slack template. It costs one line and saves minutes at night.
  • Use only two severities (warning, critical) and route them differently in Alertmanager.
  • Add an inhibition rule so NodeDown suppresses everything else from that host.
  • Unit-test your rules with promtool test rules. Thresholds and for: durations are easy to get subtly wrong, and a test catches that before production does.

The rules themselves

The runbook is free to read and bookmark. If you'd rather not write the 44 rules yourself, I sell the ready pack: rules in English and Russian, 113 promtool unit tests, a PrometheusRule for kube-prometheus-stack and an Alertmanager example with severity routing and inhibition. It costs 9 USDT via the Telegram shop. If you're just getting started, there's also a free Prometheus + Grafana + Telegram alerts kit for a single VPS.

Feedback welcome. If an alert you rely on is missing from the runbook, tell me in the comments.

Top comments (0)