I Gave a Vultr Box a Heartbeat: Self-Healing Docker Containers in 20 Minutes with systemd, Not Cron

Your Docker container crashes at 3 a.m. Cron notices at :00, :05, :10 — up to five minutes of downtime before anyone even checks. And if the crash loop is fast enough, a naive restart: always in docker-compose.yml will just spin the container up and down forever, hammering your database with reconnect storms.

What you actually want is a watchdog: something that checks health more often than cron allows, restarts with backoff instead of blindly looping, and tells you when it happens. Here's how to build that on a $6/mo Vultr VPS in about 20 minutes, using systemd timers instead of cron because systemd gives you logging, dependency ordering, and sub-minute intervals for free.

1. Provision the box

If you already have a Vultr instance, skip to step 2. Otherwise, using the Vultr CLI:

vultr-cli instance create \
  --region ewr \
  --plan vc2-1c-2gb \
  --os 387 \
  --label docker-watchdog-demo \
  --host docker-watchdog-demo

--os 387 is Ubuntu 24.04 LTS at time of writing; check vultr-cli os list if that's stale. SSH in once it's provisioned, then install Docker with the official convenience script:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

Log out and back in so the group change takes effect.

2. Deploy a container worth watching

We'll simulate a flaky service — a tiny web app that has a /kill endpoint to crash itself on demand, so you can test the watchdog without waiting for a real bug.

# docker-compose.yml
services:
  flaky-app:
    image: nginx:alpine
    container_name: flaky-app
    restart: unless-stopped
    ports:
      - "8080:80"
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost"]
      interval: 10s
      timeout: 3s
      retries: 3

restart: unless-stopped handles the trivial case (process exits) but does nothing if the container is running yet unhealthy — hung, deadlocked, or serving 500s. That's the gap the watchdog closes.

docker compose up -d

3. Write the watchdog

This script checks Docker's own health status, restarts unhealthy containers with exponential backoff, and caps retries so a genuinely broken container doesn't restart-loop forever.

#!/usr/bin/env bash
# /usr/local/bin/docker-watchdog.sh
set -euo pipefail

STATE_DIR=/var/lib/docker-watchdog
WEBHOOK_URL="${WATCHDOG_WEBHOOK_URL:-}"
MAX_RETRIES=5

mkdir -p "$STATE_DIR"

notify() {
  local msg="$1"
  [[ -z "$WEBHOOK_URL" ]] && return 0
  curl -fsS -X POST -H 'Content-type: application/json' \
    --data "{\"text\":\"$msg\"}" "$WEBHOOK_URL" >/dev/null || true
}

for cid in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$cid" | sed 's#^/##')
  health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$cid")

  [[ "$health" != "unhealthy" ]] && continue

  count_file="$STATE_DIR/${name}.count"
  count=$(cat "$count_file" 2>/dev/null || echo 0)

  if (( count >= MAX_RETRIES )); then
    echo "$name still unhealthy after $count restarts, giving up" >&2
    continue
  fi

  backoff=$(( 2 ** count ))
  echo "$name is unhealthy, restarting (attempt $((count+1)), waited ${backoff}s backoff)"
  sleep "$backoff"
  docker restart "$name"

  echo $((count+1)) > "$count_file"
  notify "⚠️ docker-watchdog restarted *${name}* (attempt $((count+1))/${MAX_RETRIES}) on $(hostname)"
done

# reset counters for containers that recovered on their own
for f in "$STATE_DIR"/*.count; do
  [[ -e "$f" ]] || continue
  name=$(basename "$f" .count)
  health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}healthy{{end}}' "$name" 2>/dev/null || echo gone)
  [[ "$health" == "healthy" ]] && rm -f "$f"
done

sudo chmod +x /usr/local/bin/docker-watchdog.sh
sudo mkdir -p /var/lib/docker-watchdog

The backoff counter is the piece cron-based versions usually skip: without it, a container that fails its healthcheck every 10 seconds gets restarted every 10 seconds, forever, which is worse than doing nothing.

4. Run it on a systemd timer, not cron

Cron's minimum resolution is one minute, it has no built-in logging beyond mail (which is rarely configured), and a hung script just silently occupies a slot forever. systemd timers fix all three: sub-minute intervals, journalctl logging automatically, and a RuntimeMaxSec to kill a hung run.

# /etc/systemd/system/docker-watchdog.service
[Unit]
Description=Docker container health watchdog
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker-watchdog.sh
Environment=WATCHDOG_WEBHOOK_URL=https://hooks.slack.com/services/REPLACE/ME
TimeoutStartSec=30

# /etc/systemd/system/docker-watchdog.timer
[Unit]
Description=Run docker-watchdog every 15 seconds

[Timer]
OnBootSec=15s
OnUnitActiveSec=15s
AccuracySec=1s

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now docker-watchdog.timer

That 15-second interval is something cron simply cannot do — its floor is 60 seconds, and even that requires an extra wrapper loop to hit reliably.

5. Prove it works

Crash the container on purpose and watch the recovery:

docker exec flaky-app sh -c 'kill 1'
journalctl -u docker-watchdog.service -f

Within 15 seconds you should see the watchdog detect the unhealthy state, apply backoff, and restart it — and if you wired up WATCHDOG_WEBHOOK_URL, a Slack message lands at the same moment. Check the timer's own schedule and history:

systemctl list-timers docker-watchdog.timer
systemctl status docker-watchdog.service

6. Lock the box down

Since this VPS is now doing something worth protecting, restrict it to the ports you actually need with the Vultr firewall (or ufw locally):

vultr-cli firewall group create --description "docker-watchdog"
# then attach rules for 22/tcp (your IP only) and 8080/tcp

Don't expose the Docker socket or the watchdog's webhook URL beyond what's needed — the script only needs local docker.sock access, which it already has by running as root via systemd.

Where to take it from here

This pattern generalizes past a single VPS: point the same script at a remote Docker context (DOCKER_HOST=ssh://...) to watch containers on a fleet from one control box, or swap the webhook for a PagerDuty Events API call if 3 a.m. pages are your actual problem. The core idea stays the same — sub-minute detection, bounded backoff, and a notification the moment state changes, all from tooling that ships with every modern Linux box by default.