Originally published on kuryzhev.cloud
Context
A standard Prometheus stack with cAdvisor, node_exporter, and Grafana is one of the most common ways teams monitor Docker hosts and containers. It's also one of the easiest setups to get subtly wrong, because Docker container alerting rules rarely fail loudly — they fail quietly, by never firing, firing constantly, or firing at the wrong time.
Part of the problem is copy-paste culture. cAdvisor and node_exporter together expose dozens of metrics per container and per host, and starter alert rules from dashboards or blog posts get dropped into production without adjusting for container lifecycle quirks: restarts, short-lived batch containers, overlay filesystem behavior, and metrics that reset when a container disappears and reappears under a new ID.
It also matters whether alerts live in classic Prometheus rule files, described in the Prometheus alerting rules documentation, or in Grafana-managed alerting, covered separately in Grafana's unified alerting documentation. The two persist alert state differently: classic rule files hold state in Prometheus memory and can lose in-flight alert transitions on restart, while Grafana-managed rules keep their own state in Grafana's database. Check which one is actually running before assuming alert history survives a restart or upgrade.
None of what follows describes a specific outage. These are documented behaviors in metric semantics and rule evaluation that repeatedly trip up teams building Docker container alerting rules from scratch or from templates.
Common failure 1: Alerting on the wrong memory metric
A common mistake is building memory alerts on container_memory_usage_bytes, the metric described in Docker's runtime metrics documentation. That number includes page cache and buffers, which the kernel reclaims automatically under pressure. A container can sit near its memory limit on this metric for hours with no real OOM risk. Threshold alerts built on it fire constantly, and repeated false alarms are the kind of thing that gets muted by on-call rather than acted on — a predictable outcome of alerting on the wrong number, not something that needs to be measured to be expected.
The metric that actually correlates with the OOM killer's behavior is closer to container_memory_working_set_bytes — usage minus total_inactive_file on cgroup v1, or inactive_file on cgroup v2. Active page cache is still counted inside working set, which matters when tuning a threshold: a container doing heavy sequential reads can push working set up without being anywhere near a real memory emergency. Alert thresholds should compare working set against container_spec_memory_limit_bytes, not a fixed number, since limits vary per service.
Watch out for containers with no memory limit configured. Dividing by that limit doesn't produce NaN — in PromQL, only 0/0 evaluates to NaN; a nonzero numerator over a zero limit returns +Inf. Depending on how the expression is written, that +Inf clears almost any threshold comparison, so the failure mode is a permanently firing alert on an unlimited container, not a silent gap. The fix is to filter for a nonzero limit explicitly, at the alert layer, and add a separate low-severity alert that flags containers with no limit set at all, so the gap is visible instead of implicit.
Common failure 2: Missing crash-loop and restart detection
A common pattern for "container is down" alerting is up == 0. This is a category error, not just a slow-to-fire check: up reflects whether Prometheus could scrape the target — cAdvisor itself — not whether any individual container on that host is running. Checking for the absence of container_last_seen gets closer to a real per-container signal, but it often still never fires for a container stuck in a fast restart loop, because Docker restarts the process every few seconds and the target reappears under a new container ID before the next evaluation cycle catches the gap.
A more reliable signal is changes(container_start_time_seconds{name!=""}[15m]) > 3. Counting how many times a container's start-time series changes within a window tracks restarts exactly. changes() counts discrete resets rather than extrapolating like increase() would on a counter, and it survives a container being replaced under a new ID, which a naive restart counter can't handle. cAdvisor has no container_restart_count metric — on Kubernetes, kube_pod_container_status_restarts_total from kube-state-metrics is the more direct source for the same signal.
A related mistake compounds the first one: setting the for: duration long to suppress noise, then discovering it also delays detection of exactly the crash-loop pattern the rule was written to catch. If a container restarts every 10 seconds, a 15-minute for: clause on top of a presence check means the alert may never transition to firing, because the target is technically "up" at almost every individual evaluation.
The for: duration needs to be sized to the failure mode, not copied from a template — short for restart detection, longer for gradual trends like memory growth.
Common failure 3: Alert fatigue from high-cardinality, flap-prone rules
Rules scoped per container_id or a container name that includes a random suffix generate a new alert series every time a deploy or autoscaling event replaces containers. Over time this inflates Prometheus TSDB series cardinality, a documented cost and performance problem per the Prometheus metric and label practices guide. It also drives up active alert counts and Alertmanager notification volume — an operational consequence of that same cardinality growth, though not something the guide itself addresses.
Rules without a for: clause fire on single-scrape spikes: a brief CPU throttling event, a garbage collection pause, a momentary blip in disk I/O. Left unaddressed, repeated false positives on rules like these teach on-call to treat pages as noise rather than signal — a predictable consequence of paging on transient spikes, not a measured outcome from any specific team.
Watch out for label mismatches between node_exporter (host-level) and cAdvisor (container-level) metrics too. Inconsistent instance or job labeling breaks correlation in Grafana panels and can misroute alerts to the wrong team or dashboard entirely. This is easy to miss during initial setup and expensive to untangle later, since dashboards and alert routing both depend on consistent label schemas.
Safer operating pattern
A more durable baseline starts with recording rules: pre-compute expensive expressions like working-set-to-limit ratios or restart rates once, then reference the recorded series from every alert and dashboard that needs it. This centralizes the "which metric is correct" decision in one place and cuts repeated evaluation cost.
# prometheus rules: container memory + restart alerting
groups:
- name: docker-container-fixed.rules
rules:
# Recording rule: record the raw ratio, unfiltered.
# Filtering here would drop containers with a legitimately-zero or
# unmeasurable ratio from dashboards with no indication anything is wrong.
# ignoring(id, image) + group_left works around cAdvisor emitting multiple
# series per container name (id/image differ) — verify against your own
# label set, this varies by cAdvisor version and cgroup driver.
- record: container:memory_working_set_ratio
expr: |
container_memory_working_set_bytes{name!=""}
/ ignoring(id, image) group_left
container_spec_memory_limit_bytes{name!=""}
- alert: ContainerMemoryPressure
# limit=0 guard lives here, at the alert, not in the recording rule
expr: |
container:memory_working_set_ratio > 0.85
and on(name, instance) container_spec_memory_limit_bytes{name!=""} > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.name }} working set > 85% of limit for 10m"
- alert: ContainerCrashLooping
# changes() on start time — cAdvisor exposes no container_restart_count
expr: changes(container_start_time_seconds{name!=""}[15m]) > 3
for: 0m # window already smooths noise; fire as soon as threshold crosses
labels:
severity: critical
annotations:
summary: "{{ $labels.name }} restarted >3 times in 15m"
- alert: ContainerMemoryUnlimited
# some cAdvisor/cgroup combinations report a sentinel near total machine
# memory instead of 0 for "no limit" — verify against your own version
expr: |
container_spec_memory_limit_bytes{name!=""} == 0
or on(instance) container_spec_memory_limit_bytes{name!=""} >= (machine_memory_bytes * 0.99)
for: 30m
labels:
severity: info
annotations:
summary: "{{ $labels.name }} has no memory limit set"
High-cardinality labels like container_id or ephemeral name suffixes should be dropped before they hit long-term storage, using metric_relabel_configs as documented in the Prometheus relabeling configuration. Group related container alerts by service label rather than by instance so a rolling deploy doesn't produce a wall of individual pages.
Inhibition rules in Alertmanager help too: if a rollout is expected to restart containers, an inhibition rule can suppress the resulting restart alert instead of relying on someone recognizing "this is expected" mid-deploy. And because alert payloads often carry internal hostnames and label values, Alertmanager webhook receivers should sit behind TLS with authentication — an unauthenticated receiver forwarding to a third-party chat integration is a quiet way to leak infrastructure details.
Alert-design checklist for Docker/container metrics
----------------------------------------------------
[ ] Memory alerts use working_set_bytes, not usage_bytes
[ ] Ratio alerts guard against limit=0 explicitly at the alert (e.g. `and container_spec_memory_limit_bytes > 0`)
[ ] Restart/crash-loop detection uses changes() on start time, or kube_pod_container_status_restarts_total on Kubernetes — not up==0
[ ] Every threshold rule has a deliberately-chosen `for:` window
[ ] High-cardinality labels (container_id, ephemeral name) dropped via relabeling
[ ] Recording rules record raw values; filtering happens at the alert layer
[ ] node_exporter and cAdvisor label schemas reconciled for routing/dashboards
[ ] Inhibition rules cover expected deploy/restart windows
[ ] Alertmanager receivers use TLS + auth, no secrets in annotations
[ ] Disk alerts include container log growth, not just host-level %used
That last item matters more than it looks: Docker's default json-file log driver without rotation configured can silently fill an overlay filesystem, and a disk alert based only on host-level percent-used will flag the symptom without pointing at the container generating the log volume.
None of this requires abandoning Prometheus, cAdvisor, or Grafana — the stack is well-documented and widely deployed for good reason. Start by auditing whatever rule set is already running in production against the checklist above before adding anything new; a rule set that's wrong in a way nobody's paging on is worse than no rule set at all. For broader coverage of Kubernetes and observability tooling patterns, see the DevOps_DayS archive.
Top comments (0)