How to Run Multiple Long Commands in Docker Compose

Use Case

I want to run multiple/long commands for my service(s) in my docker-compose file, like below:

---
# docker-compose.yaml

services:
  server: &server
    build:
      context: .
      dockerfile: ./compose/staging/server/Dockerfile
    image:  someproject_slug_staging_server:staging_shortsha_unixtime
    depends_on:
      - postgres
    env_file:
      - ./.envs/.staging/.server
      - ./.envs/.staging/.postgres
    command: python /app/manage.py collectstatic --no-input
    command: python /app/manage.py migrate
    command: gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app

  postgres:
    build:
      context: .
      dockerfile: ./compose/staging/postgres/Dockerfile
    image:  someproject_slug_staging_postgres:staging_shortsha_unixtime
    volumes:
      - staging_postgres_data:/var/lib/postgresql/data:Z
      - staging_postgres_data_backups:/backups:z
    env_file:
      - ./.envs/.staging/.postgres

However it won't work since it's not possible.

Solutions

Script File

One solution is that putting all your commands in a script file -let's say start.sh, put it into your image, build with it and call it directly.

As an example here is a docker compose file:

---
# staging.yaml

version: '3.7'

volumes:
  staging_postgres_data: {}
  staging_postgres_data_backups: {}
  staging_traefik: {}
  staging_redis: {}

services:
  server: &server
    build:
      context: .
      dockerfile: ./compose/staging/server/Dockerfile
    image:  someproject_slug_server:staging_shortsha_unixtime
    depends_on:
      - postgres
      - redis
    env_file:
      - ./.envs/.staging/.server
      - ./.envs/.staging/.postgres
      - ./.envs/.staging/.redis
    command: /start

  postgres:
    build:
      context: .
      dockerfile: ./compose/staging/postgres/Dockerfile
    image:  someproject_slug_postgres:staging_shortsha_unixtime
    volumes:
      - staging_postgres_data:/var/lib/postgresql/data:Z
      - staging_postgres_data_backups:/backups:z
    env_file:
      - ./.envs/.staging/.postgres

  traefik:
    build:
      context: .
      dockerfile: ./compose/staging/traefik/Dockerfile
    image:  someproject_slug_traefik:staging_shortsha_unixtime
    depends_on:
      - server
    volumes:
      - staging_traefik:/etc/traefik/acme:z
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "0.0.0.0:5555:5555"

  redis:
    image: docker.io/bitnami/redis:6.2
    env_file:
      - ./.envs/.staging/.redis
    ports:
      - "6379:6379" # WARNING: In production never bind 0.0.0.0 wildcard adress of the host
    volumes:
      - staging_redis:/bitnami/redis/data:z

  celeryworker:
    <<: *server
    command: /start-celeryworker

  celerybeat:
    <<: *server
    command: /start-celerybeat

  celeryflower:
    <<: *server
    command: /start-celeryflower

  aws-backup:
    build:
      context: .
      dockerfile: ./compose/staging/aws/Dockerfile
    env_file:
      - ./.envs/.staging/.server
    volumes:
      - staging_postgres_data_backups:/backups:z

WARNING

Don't bind your host's 0.0.0.0 wildcard adress in production environment. (Actually instead of deploying via docker-composein production you should consider to use more reliable, and robust distributed systems like Kubernetes but this is a topic for another post.)

6379:6379 means 0.0.0.0:6379:6379, so you open your redis instance to whole open world. And if you don't use AUTH or don't take required security precaution -like TLS etc. you may get some headache. So use it only if you
know the related consequences.

For more redis security topic look at this: Redis Security

And as an example here is content of start from server service:

#!/usr/env/bin bash
# start

set -o errexit
set -o pipefail
set -o nounset


python /app/manage.py collectstatic --noinput
python /app/manage.py migrate   # WARNING: Not a good idea for prod
gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app

WARNING

Migration is not a command you want to run every container start but since it's staging environment let's assume it's OK.

Multiline Scalar

Running in a script file is OK however maybe it's local or staging environment which means you need to debug and tweak these commands one by one; maybe some flags are not working etc. and you don't want to re-build the image every time.

How are we gonna do this?

The answer lines in the multiline capability of yaml syntax:

command: >
  bash -c "python /app/manage.py collectstatic --noinput
  && python /app/manage.py migrate
  && gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app"

This is called Folded Style. The indention in each line will be ignored. A line break (\n) will be inserted at the end.

Or you can use Literal Style:

command:
  - /bin/bash
  - -c
  - |
  python /app/manage.py collectstatic --noinput
  python /app/manage.py migrate
  gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app

This turns every newline within the string into a literal newline, and adds one at the end.

More on these styles:

Folded Style

Key: >
  this is my very very very
  long command

Output:

this is my very very very long command\n

Mind new line-\n (end of line-EOL)

Literal Style

Key: |
  this is my very very very
  long command

Output:

this is my very very very\nlong command\n

Mind new lines-\n, and compare to it above output.

Fore more detailed style and yaml syntax you can look at this amazing explanation in this link.

These were the solutions for running multiple or long commands in your docker-compose file.

All done!