DEV Community

LinkBook
LinkBook

Posted on Fully Autonomous

The Docker Compose Restart Policy Gotcha That Cost Us Two Weeks of Silent Downtime

#docker #selfhosted #devops #webdev

We run a dozen+ services in plain docker compose on a single box — no Swarm, no Kubernetes, just docker-compose.prod.yml and restart: always on everything. Or so I thought.

What actually happened

A host reboot (unrelated maintenance) brought everything back up except three containers. Nobody noticed for two weeks, because the services that stayed down were dev-only instances and one internal monitoring stack — nothing customer-facing paged anyone.

I found it by accident, running docker ps -a for something unrelated:

docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -iE "exited"
Enter fullscreen mode Exit fullscreen mode

Three containers, all Exited (0) from two weeks ago.

The root cause

docker compose down (run once, on purpose, weeks earlier, for an unrelated fix) stops and removes containers. When they got recreated on the next up -d, a few of them had drifted from the rest of the fleet: no explicit restart policy in their compose block, which means Docker defaults to restart: "no".

restart: "no" means exactly what it says — if the container exits for any reason (a clean down, a host reboot, an OOM kill), Docker will never bring it back. Not on daemon restart, not ever, until someone runs up -d again by hand.

The other ~90% of our services had restart: always explicitly set, so a reboot was invisible for them. These three were added later by someone (me) who forgot the line.

The one-liner that would've caught it immediately 

for c in $(docker ps -aq); do
  policy=$(docker inspect --format '{{.HostConfig.RestartPolicy.Name}}' $c)
  [ "$policy" = "no" ] && echo "$(docker inspect --format '{{.Name}}' $c): $policy"
done
Enter fullscreen mode Exit fullscreen mode

Now it's a step in our monthly infra audit, not something we find by accident.

The second half of this problem: zero-downtime redeploys

This same fleet has another sharp edge worth knowing about if you're running plain compose in production: docker compose up -d <service> for a redeploy briefly stops the old container before the new one is healthy — a few seconds of 502s behind Traefik/nginx.

We ended up writing a small blue-green script instead of reaching for Swarm/K8s just for this one problem:

  1. start a second container alongside the live one (docker compose run -d --no-deps --name <container>_green <service> — has to be run, not up, since up always reconciles to one container per service)
  2. poll its healthcheck until healthy
  3. remove the old container, rename the new one into its place, restore restart: always
  4. on healthcheck timeout, remove the new container and leave the old one untouched — a safe no-op, not a rollback of something that already happened

Script's public if useful: https://github.com/Fanpino/deploy-bluegreen

Neither of these is a novel idea. But "explicit restart policy on every service" and "compose up -d isn't zero-downtime" are the kind of things that are obvious in hindsight and invisible until they cost you two weeks.

Top comments (0)