Nine of my last 47 CI runs died on the same line:
Error: connect ECONNREFUSED 172.19.0.3:5432
Same commit. Same image. Same compose file. On my laptop, green every single time. In CI, a coin flip with a bad attitude.
The cause was four characters of YAML I had copy-pasted for years without ever reading:
depends_on:
- db
Docker Compose depends_on doesn't wait for your database. It waits for a container. Those are two completely different events, and the gap between them is where your flaky integration tests live.
TL;DR
-
depends_onin short list syntax only waits for the dependency container to start (condition: service_started). It knows nothing about the process inside. - To actually wait, use the long syntax with
condition: service_healthyand define ahealthcheckon the dependency. No healthcheck means no waiting, silently. -
pg_isreadywith no-htalks to the Unix socket. On a fresh volume the official Postgres image runs a temporary server on that socket while init scripts execute, so your healthcheck turns green before TCP 5432 is open. Usepg_isready -h 127.0.0.1. - That's why it only breaks in CI: CI starts with an empty volume, your laptop has a warm one, so your laptop never sees the init phase.
- Healthchecks fix startup ordering. They don't fix reconnects. Keep retry logic in the app anyway.
What does depends_on actually wait for in Docker Compose?
It waits for the container to reach the running state, and nothing more. Docker Compose depends_on in its short form expands to condition: service_started, which means "the container process has been launched." Postgres launching and Postgres accepting a connection on port 5432 are separated by anywhere from 200ms to about 15 seconds, depending on whether the data directory already exists.
depends_on gives you three real guarantees, and it's worth knowing exactly what they are:
- Start order. Dependencies come up first.
- Stop order. Dependents go down first.
- Nothing else.
It is not a readiness gate. It's a topological sort.
You can watch the gap yourself:
docker compose up -d db
docker compose exec db pg_isready -h 127.0.0.1
# pg_isready: no response ← container is "up", DB is not
The container is running. docker compose ps says running. Your API container already started, already dialed 5432, already crashed, and if you set restart: no it's already gone.
Why does adding a healthcheck still not fix it?
Because the healthcheck can be subtly wrong in two ways, and Docker will report both of them as "healthy."
Wrong tool. This is the classic:
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
Most slim and alpine images don't ship curl. The probe fails forever, the container sits in starting then flips to unhealthy, and your dependent service never boots at all. You get a hang instead of a crash, which is somehow worse to debug. Also note CMD vs CMD-SHELL: with CMD the list is exec'd directly, so ||, $VARS, and pipes do nothing. If you want shell semantics, you must say CMD-SHELL.
Wrong endpoint. This one is the actual villain in my CI logs:
test: ["CMD-SHELL", "pg_isready -U app -d app"]
Looks perfect. Ships with the image. Exits 0 when Postgres is accepting connections. It's also wrong, and here's the mechanic.
When the official Postgres image boots with an empty data directory, it does a two-phase startup. Phase one: initdb creates the cluster, then the entrypoint starts a temporary server to run everything in /docker-entrypoint-initdb.d. That temporary server deliberately does not listen on TCP — the entrypoint starts it with empty listen_addresses, so it's reachable only over the Unix socket inside the container. Phase two: the temp server is shut down and the real one starts, this time on 0.0.0.0:5432.
pg_isready with no -h connects over the Unix socket. So during phase one it gets a happy answer, exits 0, Docker marks the container healthy, Compose releases your API container, and your API dials TCP 5432 into a closed port.
The fix is one flag:
test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"]
-h 127.0.0.1 forces a TCP connection to the container's own loopback, which is exactly the thing your dependent service is about to do. During phase one it returns non-zero, so the container stays starting, so Compose keeps waiting. That's the whole bug.
And this is why it only ever broke in CI. My laptop had a populated pgdata volume from months ago, so the init phase never ran and phase one didn't exist. CI creates a fresh volume every run. The flake rate wasn't random — it tracked how long the seed scripts took that day.
MySQL has the same two-phase shape, by the way. mysqladmin ping can answer during the init window too. Same class of bug, same class of fix: probe the thing over the network path your app will actually use.
What's the Docker Compose config that actually works?
Long syntax, real healthcheck, migrations as their own gate:
services:
db:
image: postgres:16
environment:
POSTGRES_USER: app
POSTGRES_PASSWORD: app
POSTGRES_DB: app
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
# -h 127.0.0.1 forces TCP. Without it this passes during
# the init-scripts phase, when TCP 5432 is still closed.
test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"]
interval: 2s
timeout: 3s
retries: 30
start_period: 5s
migrate:
build: .
command: ["npm", "run", "migrate"]
depends_on:
db:
condition: service_healthy
api:
build: .
depends_on:
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
volumes:
pgdata:
Three things to read carefully there.
interval: 2s with retries: 30 is a 60-second patience budget. Default interval is 30s, which in a fresh-volume CI run means you either wait half a minute for a database that was ready in four seconds, or you blow the retry count. Short interval, generous retries.
start_period: 5s is not a sleep. Probes still run during it — a success inside the start period marks the container healthy immediately. What it changes is that failures during that window don't count against retries and don't mark the container unhealthy. It's a grace period, not a delay.
service_completed_successfully is the one condition people never discover. The migrate service runs, exits 0, and only then does api start. If migrations exit non-zero, api never launches and docker compose up fails loudly instead of starting an app against a half-migrated schema.
What still breaks after you fix this?
Healthy is not the same as correct, and a few sharp edges survive the fix:
-
Healthy ≠ migrated ≠ seeded.
pg_isreadysays the server accepts connections. It says nothing about whether your tables exist. That's what themigrategate above is for. -
Healthy ≠ healthy forever. A database that passes at t=0 can OOM at t=90.
depends_onis a startup gate, fired once. Your app still needs connection retry with backoff. This is the part people skip after adding healthchecks, and it's the part that pages you at 3am. -
Swarm ignores it.
docker stack deploydropsdepends_onentirely. If you're targeting Swarm, the ordering you wrote is decorative. -
CI needs
--wait.docker compose up -dreturns as soon as containers are created. Usedocker compose up -d --wait(with--wait-timeout) so the command doesn't exit until everything is healthy, then run your tests. -
Check that the probe exists at all. If you define no healthcheck,
docker inspect --format '{{json .State.Health}}' <container>printsnull, andcondition: service_healthyon that service will error out rather than quietly wait. Good — but it means a typo'd healthcheck key fails in a completely different way from a missing one.
So does Docker Compose depends_on wait for the database?
No. Docker Compose depends_on in short list syntax only waits for the dependency's container to start, which for a database is several seconds before it accepts connections. To make it actually wait, define a healthcheck on the dependency and switch to the long syntax with condition: service_healthy. Then make sure the healthcheck probes the same network path your app uses: pg_isready without -h checks the Unix socket, which answers during the Postgres image's init-scripts phase while TCP 5432 is still closed, so use pg_isready -h 127.0.0.1 instead. Pair it with a migration service gated by service_completed_successfully, keep interval short and retries high, and keep reconnect logic in your application, because startup ordering and runtime resilience are different problems.
Written by the developer behind Preterview, an interview prep platform.
Top comments (0)