DEV Community

jidonglab
jidonglab

Posted on

Docker Compose depends_on Didn't Wait: 11 of 40 Fresh Starts Crashed

My API worked every time on my laptop. A new teammate cloned the repo, ran docker compose up, and got ECONNREFUSED 172.18.0.2:5432 on the first try. CI hit the same error about once a week. When I ran it again it was fine, and that's the most annoying kind of bug there is.

The cause was Docker Compose depends_on, which does less than its name suggests. Then I fixed it with a Postgres healthcheck, and that healthcheck lied too, but only on a fresh volume. That explained why it never broke for me.

I wrote a loop that ran docker compose down -v && docker compose up 40 times for each config. Here's what I found.

TL;DR

  • depends_on: [db] (short syntax) waits until the db container is started. It does not wait for Postgres to accept connections. With that config, 11 of my 40 fresh starts crashed.
  • To wait for readiness, use the long syntax with condition: service_healthy and give the db service a healthcheck.
  • pg_isready with no -h checks the Unix socket. During first-boot init, the official Postgres image runs a temporary server that listens only on the socket. So the check passes before TCP is open, and before your init SQL has finished. Use pg_isready -h 127.0.0.1.
  • Healthcheck interval defaults to 30s, so a basic healthcheck adds about 30 seconds to every up. Set it to 1s.
  • Run migrations as a one-shot service and gate the app on condition: service_completed_successfully.

What does Docker Compose depends_on actually wait for?

The short form of Docker Compose depends_on controls start order. Compose creates and starts db, then starts api right after. It doesn't check whether the process inside db is ready to serve anything.

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: app
  api:
    build: .
    depends_on:
      - db
Enter fullscreen mode Exit fullscreen mode

A Postgres container is "started" when its entrypoint process is running. On a fresh volume, that entrypoint still has to run initdb, start a temporary server, run your init scripts, shut the temporary server down, and start the real one. That takes a few seconds. My Node API connects within milliseconds of starting, so it gets there first.

The short syntax is the same as writing:

depends_on:
  db:
    condition: service_started
Enter fullscreen mode Exit fullscreen mode

service_started is the default, and it isn't the same as "ready".

My numbers for the short syntax, fresh volume each run:

Config Crashed on startup
depends_on: [db] 11 / 40

The other 29 runs only worked because Docker happened to be slow enough that time.

How do you make Docker Compose wait until Postgres is ready?

Add a healthcheck to the database and use condition: service_healthy on the service that depends on it. Compose then won't start api until Docker marks db as healthy.

Most tutorials give you something like this:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: app
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres -d app"]
  api:
    build: .
    depends_on:
      db:
        condition: service_healthy
Enter fullscreen mode Exit fullscreen mode

That fixed the crashes. It also made every docker compose up take about 31 seconds, and I hadn't changed anything else.

Why does my healthcheck make docker compose up take 30 seconds?

Docker's healthcheck interval defaults to 30 seconds, and the first probe runs one interval after the container starts. So Postgres is ready in about 2 seconds, Docker waits until second 30 to check, and your app waits along with it.

Here are the defaults you get when you leave the fields out:

Field Default
interval 30s
timeout 30s
retries 3
start_period 0s

Set the interval explicitly:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres -d app"]
  interval: 1s
  timeout: 3s
  retries: 30
Enter fullscreen mode Exit fullscreen mode

up went from about 31s back to about 3s. Running pg_isready once a second is cheap, so I see no reason to run it less often in dev.

Then I ran my 40-run loop again, and 4 runs still crashed.

Why does pg_isready say healthy while connections are still refused?

pg_isready with no -h flag connects through the local Unix domain socket, not TCP. The official Postgres image's entrypoint does its first-boot setup on a temporary server started with listen_addresses='', which means socket only and no TCP at all. So while init is running, the socket check reports "accepting connections" and port 5432 isn't open yet.

Here's what happens on a fresh volume:

  1. initdb creates the data directory.
  2. A temporary server starts, listening on the socket only.
  3. Everything in /docker-entrypoint-initdb.d/ runs against it: your schema, seed data, extensions.
  4. The temporary server shuts down.
  5. The real server starts and listens on TCP.

If the healthcheck runs during step 2 or 3, it passes. Docker marks the container healthy, Compose starts the API, and the API tries TCP and gets refused. Or it connects in the short gap after step 5 and finds a database whose init scripts haven't finished.

This also explains why it never broke on my machine. My volume already existed, so the entrypoint skipped init completely. There was no temporary server and no gap. It only failed on a fresh volume, which is exactly what a new teammate or a CI runner has.

The fix is one flag. Make the check use TCP, the same way your app connects:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U postgres -d app"]
  interval: 1s
  timeout: 3s
  retries: 30
Enter fullscreen mode Exit fullscreen mode

The temporary server has no TCP listener, so this check can't pass until step 5, and by then init is done.

Config Crashed on startup Avg up time
short depends_on 11 / 40 ~2s
healthcheck, default interval 0 / 40 ~31s
healthcheck, interval: 1s, socket 4 / 40 ~3s
healthcheck, interval: 1s, -h 127.0.0.1 0 / 40 ~3s

What about the $POSTGRES_USER trap in the healthcheck?

Compose interpolates $VARIABLES in your YAML from your host shell and .env file before the container exists. If you write pg_isready -U $POSTGRES_USER, Compose tries to fill it in from your laptop's environment, and you'll get a warning like "The POSTGRES_USER variable is not set. Defaulting to a blank string."

To pass a literal $ through so the container's shell expands it, double it:

test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
Enter fullscreen mode Exit fullscreen mode

This one rarely breaks anything, because pg_isready only checks whether the server accepts connections and doesn't care whether the user exists. That's why it's easy to miss. A healthcheck that passes for the wrong reason will eventually let something through that it shouldn't.

How do you run migrations before the app starts in Docker Compose?

Make migrations a separate one-shot service, and gate the app on condition: service_completed_successfully. Compose waits for that container to exit with code 0. If it exits non-zero, the app never starts, which is what you want.

Here's the full file I ended up with:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./init:/docker-entrypoint-initdb.d:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 1s
      timeout: 3s
      retries: 30

  migrate:
    build: .
    command: npm run migrate
    depends_on:
      db:
        condition: service_healthy

  api:
    build: .
    ports:
      - "3000:3000"
    depends_on:
      db:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully

volumes:
  pgdata:
Enter fullscreen mode Exit fullscreen mode

In CI, use docker compose up -d --wait. It blocks until the services are running and healthy, and exits non-zero if one fails, so your test step doesn't race the stack.

Should the app still retry its database connection?

Yes. Docker Compose depends_on only applies at startup. If Postgres restarts later because of an OOM kill, a config reload, or someone running docker compose restart db, Compose won't hold your API back, and the API will hit the same refused connection.

I added a small retry with backoff to the connection pool: 10 attempts, starting at 200ms and doubling up to 2s. It's about fifteen lines. The healthcheck handles the common case of a cold start, and the retry handles the database restarting later. You want both. Don't use restart: on-failure as your only fix. It hides the crash loop in docker compose logs and makes your startup timing random.

Reproduce it yourself

for i in $(seq 1 40); do
  docker compose down -v >/dev/null 2>&1
  docker compose up -d >/dev/null 2>&1
  sleep 8
  docker compose ps api --format '{{.State}}' | grep -q running \
    && echo ok || echo CRASH
done | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode

The -v matters. Without it the volume survives between runs, init gets skipped, and the socket bug goes away. That's the same reason I didn't catch it on my own machine.

So does Docker Compose depends_on wait for the database?

No. On its own, Docker Compose depends_on only waits for the dependency's container to start, not for the service inside to accept connections. To actually wait, add a healthcheck to the database with a short interval (the 30s default makes every up slow), use depends_on with condition: service_healthy, and with Postgres, run pg_isready -h 127.0.0.1 so the check goes over TCP. The socket-only temporary server the image uses during first-boot init will pass a socket check before the database is really ready. Put migrations in a one-shot service behind service_completed_successfully, and keep a connection retry in the app for restarts that happen after startup.


Written by the developer behind Preterview, an interview prep platform.

Top comments (0)