DEV Community

Dmitriy (s8kur)
Dmitriy (s8kur)

Posted on

Why your monitor says a crash-looping Docker container is up, and how to catch it

#devops #monitoring #selfhosted #docker

A container that crashes every few seconds can sit behind a green uptime check for hours. This post is about why that happens and the free ways I now use to catch it, whatever monitoring tool you run.

What a crash loop looks like

Most compose files set a restart policy:

services:
  app:
    image: myapp:latest
    restart: unless-stopped   # or: always
Enter fullscreen mode Exit fullscreen mode

When the process exits, Docker starts it again. If it crashes on startup, for example because a database is unreachable or a config value is wrong, you get a loop: start, run for a few seconds, exit, wait, start again. Docker increases the delay between restarts (it starts at 100 ms and doubles, up to a cap), and resets it once the container has stayed up for 10 seconds.

If you run docker ps at the wrong moment, you see this:

STATUS
Up 2 seconds
Enter fullscreen mode Exit fullscreen mode

Run it again a bit later and you might see Restarting (1) 5 seconds ago. The container is broken, but a lot of the time it looks alive.

Why normal checks miss it or flap

"Is it running" checks. A check that asks Docker whether the container state is running will say yes for every sample that lands in the short window between start and crash. Depending on your interval and how long the app lives, you get mostly green with the occasional red.

Plain HTTP checks. These do better, but not always. If the app serves a page or a health endpoint for a few seconds before it dies, some probes succeed. If you also require two or three consecutive failures before alerting, which is a sensible way to avoid noise, the successful probes in between keep resetting that counter. You end up with a check that flaps or never alerts at all.

The useful signal is not "is it up right now" but "how many times has it restarted lately". Docker already tracks that.

Free way 1: watch RestartCount

Every container has a restart counter:

docker inspect -f '{{.RestartCount}}' app
Enter fullscreen mode Exit fullscreen mode

The number only goes up while that container exists. If you recreate it (for example docker compose up after an image change), the new container starts at 0. A healthy service should keep this number flat. If it climbs a few times in ten minutes, something is wrong, even if every HTTP check you sampled was fine.

Free way 2: the events stream

Docker emits an event every time a container dies:

docker events --filter event=die \
  --format '{{.Time}} {{.Actor.Attributes.name}} exit={{.Actor.Attributes.exitCode}}'
Enter fullscreen mode Exit fullscreen mode

This is good for seeing a loop live, or for piping into a log. It is a stream, so you need something that keeps running and counts events. If that watcher stops, you miss events, so I prefer polling RestartCount for alerting and use events for debugging.

Free way 3: HEALTHCHECK plus autoheal

If your image defines a HEALTHCHECK, Docker reports starting, healthy or unhealthy:

docker inspect -f '{{.State.Health.Status}}' app
Enter fullscreen mode Exit fullscreen mode

Docker itself does not restart a container because it is unhealthy. The popular willfarrell/autoheal container watches for unhealthy containers that carry an autoheal=true label (or all of them, if you configure it) and restarts them.

This is useful, but know its limits:

  • It only helps when the process keeps running but stops working. A crash loop is the opposite case: the process exits, and the restart policy is already restarting it.
  • It restarts, it does not tell you. Something that keeps getting fixed every few minutes can stay broken for weeks without anyone noticing.
  • It needs access to the Docker socket, which is effectively root on the host.

So I treat healthchecks as a signal to alert on, and autoheal as an optional extra.

A small script that alerts on a restart loop

This script polls RestartCount and prints an alert when the count rises N or more times within M minutes. I tested it against a container started with --restart always and a command that exits after 2 seconds; it alerted within a minute.

#!/bin/sh
# restart-watch.sh CONTAINER [N] [M_MINUTES] [INTERVAL_SECONDS]
# Alerts when RestartCount rises N or more times within M minutes.
c="$1"; n="${2:-3}"; m="${3:-10}"; every="${4:-30}"
[ -n "$c" ] || { echo "usage: $0 container [n] [minutes] [interval]"; exit 2; }
hist=""   # lines of "epoch count"
alerted=0
while :; do
  now=$(date +%s)
  cnt=$(docker inspect -f '{{.RestartCount}}' "$c" 2>/dev/null) || { echo "cannot inspect $c"; sleep "$every"; continue; }
  hist=$(printf '%s\n%s %s\n' "$hist" "$now" "$cnt" | awk -v cut=$((now - m*60)) 'NF==2 && $1>=cut')
  oldest=$(printf '%s\n' "$hist" | head -n1 | cut -d' ' -f2)
  rise=$((cnt - oldest))
  [ "$rise" -lt 0 ] && { hist="$now $cnt"; rise=0; }   # container recreated
  if [ "$rise" -ge "$n" ]; then
    if [ "$alerted" -eq 0 ]; then
      msg="$c: restart loop, $rise restarts in ${m} min (RestartCount=$cnt)"
      echo "$msg"
      # curl -fsS -d "$msg" https://ntfy.sh/your-topic
      alerted=1
    fi
  else
    alerted=0
  fi
  sleep "$every"
done
Enter fullscreen mode Exit fullscreen mode

Run it as ./restart-watch.sh app 3 10 30: alert if app restarts 3 times in 10 minutes, checking every 30 seconds. Replace the echo with whatever you use for notifications (the commented line posts to ntfy). It alerts once per loop and re-arms when the count goes quiet. If the count goes backwards, the container was recreated and the history resets. Run it under systemd or in a small container with the socket mounted read-only, and keep the same caveat about socket access as above.

Disclosure: I build Vigil

I am the developer of Vigil, a self-hosted monitoring and incident tool, so read this part with that in mind.

I added it to Vigil Core 1.31.0. The Docker check now samples RestartCount, and by default 3 restarts within 10 minutes marks the monitor down with a message like restart loop: 3 restarts in 10 min. A HEALTHCHECK status of unhealthy also counts as down by default. Both are configurable under Advanced (the threshold, the window, and turning either off). This is in the free, Apache 2.0 Core. Automatically restarting the container through a hook is part of the paid Commercial edition.

Someone also reported this exact case for Uptime Kuma in issue #7956, so it is a common gap, not a quirk of one tool.

Whatever you use, the main point stands: alert on the restart count, not only on whether the container is up right now.

This post was drafted with help from an AI assistant and reviewed by me. The script was tested on a real container before publishing.

Top comments (0)