DEV Community

anders lee
anders lee

Posted on

Three Docker Compose health checks that say "healthy" too early (tested on Docker 29)

#docker #devops #mysql #postgres

depends_on with condition: service_healthy is only as good as the health check behind it. While testing database containers for a Docker tutorial, I found that three of the most copied health checks report healthy before the service can actually be used. Your app container starts, its first connection fails, and the logs make it look like a network problem.

Everything below was run on Docker Engine 29.9 with Compose v5.6, using mysql:8.4, postgres:18 and redis:8-alpine.

1. MySQL: mysqladmin ping -h localhost

This is the health check you'll find in many examples:

healthcheck:
  test: ["CMD-SHELL", "mysqladmin ping -h localhost"]
Enter fullscreen mode Exit fullscreen mode

On a fresh volume:

$ docker compose up -d --wait
 ✔ Container mysql-demo-db-1  Healthy     5.8s

$ docker compose exec db sh -c 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SELECT 1"'
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

$ docker compose logs db 2>&1 | grep -E "Temporary server|port: 3306"
db-1  | [Note] [Entrypoint]: Temporary server started.
Enter fullscreen mode Exit fullscreen mode

Healthy after 5.8 seconds, but root can't log in yet, and the log shows why: the container is still on its temporary server.

On first start (empty data directory), the image's entrypoint starts a temporary mysqld to create users and run the files in /docker-entrypoint-initdb.d/. That temporary server is started with networking disabled (port: 0) and only listens on the Unix socket. Two things then line up badly:

  • The MySQL client treats -h localhost as "use the socket", so the ping reaches the temporary server.
  • mysqladmin ping succeeds as soon as the server answers, even if the login itself would fail.

Fix: force TCP, which the temporary server doesn't accept.

healthcheck:
  test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -uroot -p\"$$MYSQL_ROOT_PASSWORD\" --silent"]
  interval: 10s
  retries: 10
  start_period: 30s
Enter fullscreen mode Exit fullscreen mode

With this version, the same project became healthy after about 11 seconds, which matches when the real server logged port: 3306. (The $$ stops Compose from substituting the variable itself, so the shell inside the container reads it.)

2. PostgreSQL: pg_isready without -h

Same idea, different database:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U app -d app"]
Enter fullscreen mode Exit fullscreen mode 
$ docker compose up -d --wait
 ✔ Container pg-demo-db-1  Healthy     3.2s

$ docker compose exec db psql -U app -h 127.0.0.1 -c "SELECT count(*) FROM products;"
psql: error: connection to server at "127.0.0.1", port 5432 failed: Connection refused
    Is the server running on that host and accepting TCP/IP connections?
Enter fullscreen mode Exit fullscreen mode

Healthy after 3.2 seconds, while TCP connections are still refused. The Postgres entrypoint also runs your init scripts on a temporary server that listens on the Unix socket only. Without -h, pg_isready uses that socket and reports ready.

Fix:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"]
  interval: 2s
  timeout: 3s
  retries: 30
Enter fullscreen mode Exit fullscreen mode

Now the check went healthy at 11.3 seconds, after PostgreSQL init process complete and listening on IPv4 address "0.0.0.0", port 5432.

3. Redis: redis-cli ping with a password

This one has nothing to do with startup. With requirepass set and a wrong password:

$ docker compose exec -e REDISCLI_AUTH=wrong redis redis-cli ping; echo "exit code: $?"
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
NOAUTH Authentication required.

exit code: 0
Enter fullscreen mode Exit fullscreen mode

redis-cli prints an error but exits with 0, and Docker only looks at the exit code. So a health check of redis-cli ping passes even when the password is wrong, and your app fails later with NOAUTH.

Fix: check the reply, not the exit code.

$ docker compose exec -e REDISCLI_AUTH=wrong redis sh -c "redis-cli ping | grep -q PONG"; echo "exit code: $?"
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
exit code: 1
Enter fullscreen mode Exit fullscreen mode 
redis:
  image: redis:8-alpine
  command: ["redis-server", "--requirepass", "${REDIS_PASSWORD}"]
  environment:
    REDISCLI_AUTH: ${REDIS_PASSWORD}
  healthcheck:
    test: ["CMD-SHELL", "redis-cli ping | grep -q PONG"]
    interval: 5s
    retries: 10
Enter fullscreen mode Exit fullscreen mode

(Pass the password with the REDISCLI_AUTH environment variable rather than -a, which puts it in the process list and prints a warning.)

How to check your own health checks

A health check is a command whose exit code Docker trusts blindly. Two quick tests catch most problems:

  1. Start on an empty volume (docker compose down -v, then docker compose up -d --wait) and immediately try a real connection the way your app does: over TCP, with the app's user. If --wait returns before that works, the check is too optimistic.
  2. Break the credentials on purpose and run the check by hand with echo $?. It should exit non-zero.

To see what Docker recorded for each run:

{% raw %}docker inspect --format "{{json .State.Health}}" <container>{% endraw %}
Enter fullscreen mode Exit fullscreen mode

I collected these and other Docker pitfalls, all with real command output, in a free tutorial (also available in Chinese, Japanese and Spanish): codingkit.com/docker. The MySQL, PostgreSQL and Redis chapters have the full Compose files for the examples above.

Top comments (1)

Collapse
 
abhinavs1920 profile image
Abhinav Sharma •

The temporary server detail is the trap: the ping answers over the socket while TCP is not even listening yet. Testing the actual connection the app will use, instead of just server liveness, is what makes service_healthy mean something.