Amorizz

Posted on Oct 11

Write a Docker Compose healthcheck that actually checks your app

#docker #devops #webdev #javascript

I wrote what looked like a careful Compose file for a small Node + Postgres app: a pg_isready healthcheck on Postgres and depends_on: condition: service_healthy on the app. On a fresh volume the app died with ECONNREFUSED while docker compose ps showed the database as (healthy).

Then I stopped Postgres to see what my readiness endpoint would say, and the Node process crashed instead. Six restarts in 40 seconds. Both bugs came from the same mistake: I was checking that something was running, not that it could do its job.

This is the setup I ended up with. Everything below was run on Docker Engine 26.1 with Compose 2.26, postgres:16.15-alpine and node:22.23.3-alpine, and the outputs are pasted from those runs, trimmed to the relevant lines (ps cut down to the SERVICE and STATUS columns, Start/End timestamps dropped from inspect).

TL;DR

pg_isready with no -h talks to the Unix socket, and during first-boot init the official Postgres image runs a temporary server that only listens on that socket. Add -h 127.0.0.1 so the check uses TCP like your app does.
Split your app's probes: /healthz answers "is the process alive", /readyz answers "can I serve a real request right now" (boot finished, database answers SELECT 1 within 1 second).
Plain Docker never restarts a container for being unhealthy. Pick what the healthcheck probes based on what reacts to it in your setup.

What does a healthcheck actually prove?

A healthcheck proves only what its command touches. "Process up", "port open", "app booted" and "dependencies reachable" are four different facts, and most Compose files check the first or second one and assume the rest.

Check	What it proves	What it misses
Container is `Up`	PID 1 hasn't exited	Everything else
Port accepts a TCP connection	Something is listening	App may still be booting, DB may be down
`pg_isready` (socket)	A Postgres server answers on the socket	First-boot temp server, TCP, auth, your database
`GET /healthz` returns 200	Event loop runs, HTTP stack works	Database, migrations, config
`GET /readyz` returns 200	Boot finished and the DB answered `SELECT 1` in 1s	Slow queries, other dependencies you didn't add

Kubernetes splits this into liveness and readiness probes. Docker gives you one healthcheck per container, so you have to decide which question it answers.

Why did pg_isready say healthy too early?

Because it asked the wrong server. The official image's entrypoint runs initdb, starts a temporary Postgres that listens only on the Unix socket, runs your /docker-entrypoint-initdb.d scripts, then shuts it down and starts the real server on TCP. pg_isready -U app uses the socket by default, so it reports "accepting connections" while the temp server is up.

My repro has a slow init script (an 8 second sleep standing in for a schema load):

#!/bin/sh
# initdb/01-slow-init.sh: stands in for a first boot that does real work
echo "init: loading schema (slow on purpose)"
sleep 8
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "CREATE ROLE reporting NOLOGIN;"

And a Compose file that trusts plain pg_isready:

# compose.lie.yml: the version that lies
services:
  db:
    image: postgres:16.15-alpine
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: app
    volumes:
      - ./initdb:/docker-entrypoint-initdb.d:ro
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "app"]
      interval: 1s
      timeout: 3s
      retries: 30
  app:
    build: ./app
    environment:
      DATABASE_URL: postgres://app:app@db:5432/app
    depends_on:
      db:
        condition: service_healthy

$ docker compose -f compose.lie.yml up -d
$ docker compose -f compose.lie.yml ps -a
SERVICE   STATUS
app       Exited (1) 3 seconds ago
db        Up 5 seconds (healthy)
$ docker inspect --format '{{json .State.Health}}' lab-2026-10-11-healthcheck-db-1
{"Status": "healthy", "FailingStreak": 0, "Log": [{"ExitCode": 0, "Output": "/var/run/postgresql:5432 - accepting connections\n"}]}

Look at the Output: /var/run/postgresql:5432. That's the socket. The logs show the order (timestamps trimmed):

db-1   13:23:59.918  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1   13:24:00.056  init: loading schema (slow on purpose)
app-1  13:24:00.901  listening on :3000
app-1  13:24:00.904  boot failed: ECONNREFUSED
db-1   13:24:08.065  CREATE ROLE
db-1   13:24:08.190  listening on IPv4 address "0.0.0.0", port 5432

The app gave up about 7 seconds before Postgres listened on TCP. Adding psql -c 'SELECT 1' over the socket doesn't fix it either. I tried: the temp server answers that too, and the app still got ECONNREFUSED. The fix is to probe the way your app connects, over TCP.

The Compose file that tells the truth

Probe Postgres over TCP, give the app its own healthcheck against /readyz, and gate startup on both. This is the full file.

# compose.yml
services:
  db:
    image: postgres:16.15-alpine
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: app
    volumes:
      - ./initdb:/docker-entrypoint-initdb.d:ro
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      # -h 127.0.0.1 forces TCP. The first-boot temp server only listens on the socket.
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app && psql -h 127.0.0.1 -U app -d app -tAc 'SELECT 1' | grep -q 1"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 30s
      start_interval: 1s

  app:
    build: ./app
    restart: unless-stopped
    environment:
      DATABASE_URL: postgres://app:app@db:5432/app
    ports:
      - "127.0.0.1:3000:3000"
    healthcheck:
      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000/readyz').then(r=>process.exit(r.ok?0:1),()=>process.exit(1))"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 20s
      start_interval: 1s
    depends_on:
      db:
        condition: service_healthy
        restart: true

volumes:
  pgdata:

What each knob does:

start_period: failures in this window don't count toward retries, but the first success marks the container healthy right away. Size it for your slowest cold boot.
start_interval (Engine 25+): how often to probe during start_period. 1s here means up --wait returns as soon as things are ready instead of waiting out a 10s interval.
timeout: kept above the app's own 1s DB timeout, so /readyz returns a clean 503 before Docker kills the probe.
restart: true under depends_on: Compose restarts the app when you restart db with a Compose command. It doesn't fire when the container runtime restarts a crashed db.

The /healthz and /readyz handlers

/healthz should be cheap and depend on nothing, so it only fails when the process itself is broken. /readyz should fail fast when the app can't serve a real request: boot not finished, or the database not answering within a second.

{ "name": "health-lab", "private": true, "dependencies": { "pg": "8.13.1" } }

// server.js: /healthz says "the process works", /readyz says "I can do real work"
const http = require("node:http");
const { Pool } = require("pg");

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 5,
  connectionTimeoutMillis: 1000, // a hung DB must fail the probe, not hang it
});

// Without this, a DB restart kills the whole process (unhandled 'error' event).
pool.on("error", (err) => console.error("pg idle client error:", err.code || err.message));

let booted = false; // flips after startup work (migrations, caches) is done

async function boot() {
  await pool.query("CREATE TABLE IF NOT EXISTS hits (id serial PRIMARY KEY, at timestamptz DEFAULT now())");
  booted = true;
  console.log("boot: migrations done");
}

function send(res, code, body) {
  res.writeHead(code, { "content-type": "application/json" });
  res.end(JSON.stringify(body));
}

const server = http.createServer(async (req, res) => {
  if (req.url === "/healthz") return send(res, 200, { status: "ok" });

  if (req.url === "/readyz") {
    if (!booted) return send(res, 503, { status: "starting" });
    try {
      await pool.query({ text: "SELECT 1", query_timeout: 1000 });
      return send(res, 200, { status: "ready", db: "ok" });
    } catch (err) {
      return send(res, 503, { status: "not_ready", db: err.code || err.message });
    }
  }

  try {
    await pool.query("INSERT INTO hits DEFAULT VALUES");
    send(res, 200, { hello: "world" });
  } catch (err) {
    send(res, 500, { error: err.code || err.message });
  }
});

server.listen(3000, "0.0.0.0", () => console.log("listening on :3000"));

// Retry boot instead of exiting: stay live (/healthz 200), report not ready (/readyz 503).
(async function bootLoop() {
  for (let attempt = 1; !booted; attempt++) {
    try {
      await boot();
    } catch (err) {
      console.error(`boot attempt ${attempt} failed: ${err.code || err.message}`);
      await new Promise((r) => setTimeout(r, Math.min(1000 * attempt, 10000)));
    }
  }
})();

FROM node:22.23.3-alpine
WORKDIR /app
COPY package.json ./
RUN npm install --omit=dev --no-audit --no-fund
COPY server.js ./
USER node
CMD ["node", "server.js"]

My first version had neither the pool.on("error") line nor the retry loop. When I stopped Postgres, pg emitted 57P01 terminating connection due to administrator command on an idle client, nothing listened for it, and Node exited. The restart policy brought it back, boot failed because the DB was still down, it exited again: RestartCount was 6 after 40 seconds.

Check it: start, then break the database

Run it with --wait so the command only returns once every healthcheck passes. Then stop and pause Postgres and watch what each probe says.

$ time docker compose up -d --build --wait
 Container lab-2026-10-11-healthcheck-db-1  Healthy
 Container lab-2026-10-11-healthcheck-app-1  Healthy
real    0m14.674s

$ docker compose ps
SERVICE   STATUS
app       Up 1 second (healthy)
db        Up 11 seconds (healthy)

$ docker inspect --format '{{json .State.Health}}' lab-2026-10-11-healthcheck-db-1
{"Status": "healthy", "FailingStreak": 0, "Log": [{"ExitCode": 0, "Output": "127.0.0.1:5432 - accepting connections\n"}]}

$ curl -s localhost:3000/healthz; curl -s localhost:3000/readyz
{"status":"ok"}
{"status":"ready","db":"ok"}

The inspect output now says 127.0.0.1:5432, so the check went over TCP. Now take the database away:

$ docker compose stop db
$ curl -s -w ' HTTP %{http_code}\n' localhost:3000/healthz
{"status":"ok"} HTTP 200
$ curl -s -w ' HTTP %{http_code}\n' localhost:3000/readyz
{"status":"not_ready","db":"Connection terminated due to connection timeout"} HTTP 503

# 40 seconds later
$ docker compose ps -a
SERVICE   STATUS
app       Up 43 seconds (unhealthy)
db        Exited (0) 42 seconds ago
$ docker inspect --format '{{.RestartCount}} {{.State.Health.Status}}' lab-2026-10-11-healthcheck-app-1
0 unhealthy

That 0 unhealthy is the line most people don't expect. restart: unless-stopped reacts to the process exiting. It ignores health. A hung database is a different failure from a stopped one, so pause it too:

$ docker compose start db     # app goes back to (healthy) on its own
$ docker compose pause db
$ time curl -s -w ' HTTP %{http_code}\n' localhost:3000/readyz
{"status":"not_ready","db":"Query read timeout"} HTTP 503
real    0m1.012s

One second and a 503, instead of a probe that hangs until Docker's timeout.

Should the Docker healthcheck call /healthz or /readyz?

It depends on what reacts to unhealthy in your setup. If nothing restarts unhealthy containers, point it at /readyz so depends_on and up --wait mean "ready to serve". If something does restart them, point it at /healthz, or a database blip turns into every app container restarting at once.

What reacts to `unhealthy`	Point the healthcheck at	Why
Nothing (plain Compose, `restart:` policy only)	`/readyz`	Gates `depends_on` and `up --wait` on real readiness, restarts nothing
Swarm, or an autoheal-style sidecar	`/healthz`	A DB outage shouldn't restart healthy app processes
Reverse proxy health checks (Traefik, Caddy, a load balancer)	`/readyz` from the proxy	Pull traffic from a replica that can't serve
Deploy script after `docker compose pull`	`/readyz` with `curl -f`	Fail the deploy before you switch traffic

Failure checklist

When the status and reality disagree, it's usually one of these. I reproduced 1 through 5 and 7 in the lab above.

pg_isready with no -h. It checks the socket, so it passes during first-boot init. Use -h 127.0.0.1.
curl isn't in the image. node:22.23.3-slim ships neither curl nor wget, and the check fails with exec: "curl": executable file not found in $PATH. Use node -e "fetch(...)", or busybox wget on Alpine.
One string in exec form. ["CMD", "pg_isready -U postgres"] makes Docker look for a binary literally named pg_isready -U postgres. Split the arguments, or use CMD-SHELL when you need && or pipes.
localhost resolves to ::1. In the Alpine container, localhost resolved to ::1. My test server listened on 0.0.0.0 (IPv4 only), and wget http://localhost:3000/ got Connection refused. Use 127.0.0.1.
No pool.on("error") in Node. A Postgres restart kills the process with an unhandled 'error' event, and the restart loop hides it.
/readyz with no timeout. Without connectionTimeoutMillis and query_timeout, a hung DB makes the probe wait until Docker kills it, and the health log only says Health check exceeded timeout (3s). Time out inside the app, below the healthcheck timeout.
Expecting unhealthy to restart anything. It doesn't, in plain Docker. It only feeds depends_on, --wait, docker ps and whatever tool you add.
$ in a CMD-SHELL string. Compose interpolates $VAR before the shell sees it. Write $$POSTGRES_USER if you want the container's env var.

Where this runs for me

The stack I'm building, Epure, is a two-container Compose file: one Rust binary and Postgres 16. It does exception-only error monitoring for apps that already use the official Sentry SDKs, no APM, replay or logs. Its production checklist has the same lesson in another form: /health returns ok over plain HTTP while login loops, because the session cookie is HTTPS-only. A green probe only tells you the process answered.

The lab is small enough to copy: compose.yml, server.js, package.json, the Dockerfile above, and one init script with a sleep 8 in it if you want to see the first-boot race yourself.

What does your /readyz check besides the database?

DEV Community