Controlling Service Readiness in Docker Compose

#docker #networking

TL;DR Add dokku/wait as a service, then run wait -c service:port

If you use Docker Compose, you probably have already tried to request a service that isn't ready, despite the container is running.

Actually "running" and "ready" are two different concepts from Docker's point of view. "Running" refers to the container state. "Ready" refers to the service running inside it.

You can control the order of service startup and shutdown with the depends_on option. (...) However, for startup Compose does not wait until a container is "ready" only until it’s running.

https://docs.docker.com/compose/startup-order/

Reproducing The Problem

To illustrate the problem, let's run some SQL command against a PostgreSQL instance just after starting it as a Docker Compose.

# docker-compose.yaml

version: '3'

services:

    postgres:
        image: postgres:12
        environment:
            POSTGRES_PASSWORD: s3cr3t

Using the docker-compose.yaml file above, if we run these commands sequentially:

docker-compose up -d
docker-compose exec postgres psql -U postgres -c "SELECT NOW()"

We'll probably get this output error:

psql: error: could not connect to server: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

This error is due to the fact we try to run a SQL before the PostgreSQL is ready.

We therefore need to find a way to be sure the service is ready before making requests to it. That's make sense, but how to fix it?

Note that this problem may also happen to other services, because they follow the same pattern: container is running but the service isn't ready. Here a list of outputs for some popular services:

Elasticsearch
- Failed connect to elasticsearch:9200; Connection refused
MySQL
- ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
RabbitMQ
- Error: unable to perform an operation on node 'rabbit@a1b2c3d4e5f6'
Redis
- Could not connect to Redis at redis:6379: Connection refused

The Solution

We can solve this by (1) adding dokku/wait as a service in the docker-compose.yaml, then (2) asking it to watch the service we want.

dokku/wait is a (...) Docker utility that blocks until another container is accepting TCP connections, and errors-out if it cannot connect within a given timeout.

https://github.com/dokku/docker-wait

Step 1: Add dokku/wait as a service

Let's change the docker-compose.yaml file by adding a new wait service.

# docker-compose.yaml

version: '3'

services:

    postgres:
        image: postgres:12
        environment:
            POSTGRES_PASSWORD: s3cr3t

    wait:
        image: dokku/wait

Step 2: Ask dokku/wait to watch the target service

The new wait service we added to the docker-compose.yaml file is able to reach any service in the stack by giving its name and its exposed port (e.g. postgres:5432) thanks to the network set up by Docker Composer.

We can then ask it to block until the database get ready.

docker-compose up -d
docker-compose run wait -c postgres:5432
docker-compose exec postgres psql -U postgres -c "SELECT NOW()"

Result: SQL commands will work as expected

The wait -c postgres:5432 command will block for some seconds the psql -U postgres -c "SELECT NOW()" execution. That's why the SQL will work as expected.

Furthermore, we can wait for multiples services at once:

docker-compose run wait -c mysql:3306,redis:6379,rabbitmq:5672,elasticsearch:9200

Conclusion

There are other approaches¹ to solve this problem, but they assume we have to install some script into the host machine or into each dependent service.

Adopting dokku/wait as suggested here is less intrusive because it's not needed to install a script on our machine neither on docker services stack.

Tip: We can even apply this to an existing project via docker-compose.override.yaml without changing anything in the project itself.

¹ wait-for-it, dockerize, wait-for