DEV Community

Cover image for Efficent usage of Docker
Oleksandr Kashytskyi
Oleksandr Kashytskyi

Posted on Edited on

Efficent usage of Docker

#architecture #devops #docker #softwareengineering

I've seen this more than once, even on production projects. Somebody needs a test environment, so they create docker-compose.test.yml, copy the whole docker-compose.yml into it, and declare every service again from scratch.

Ctrl+C and Ctrl+V are great shortcuts, until they end up in your infrastructure. Sooner or later the copies drift apart: one file gets a Postgres upgrade, the other doesn't, and your tests happily pass against a database version you don't actually run.

Docker Compose can do much better with two features:

  1. A shared base file: every service is declared once and reused with extends.
  2. Profiles: one docker-compose.yml holds all environments, and you choose which one to start.

Add one env file per environment and a well-chosen command, and adding a new environment becomes a few lines instead of a full copy-paste.

The snippets are simplified from my side project, a flower shop backend, but the pattern works for any stack.

Contents

The full setup

Here's everything at once, then we'll take it apart piece by piece. The example is a Node.js app with PostgreSQL and MailHog (a fake SMTP server that catches outgoing emails).

.
├── docker-compose.base.yml   # shared service settings
├── docker-compose.yml        # environments: dev and test
├── .env.dev
└── .env.test
Enter fullscreen mode Exit fullscreen mode

docker-compose.base.yml

services:
  app:
    image: node:22.12.0
    ports:
      - ${PORT}:${PORT}
    volumes:
      - ./:/usr/src/app
    working_dir: /usr/src/app

  postgres:
    image: postgres:17.3
    ports:
      - ${POSTGRES_PORT}:${POSTGRES_PORT}
    environment:
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - PGPORT=${POSTGRES_PORT}

  mailhog:
    image: mailhog/mailhog
    ports:
      - ${EMAIL_SMTP_PORT}:${EMAIL_SMTP_PORT}  # SMTP
      - ${EMAIL_HTTP_PORT}:${EMAIL_HTTP_PORT}  # web UI
    environment:
      - MH_SMTP_BIND_ADDR=0.0.0.0:${EMAIL_SMTP_PORT}
      - MH_UI_BIND_ADDR=0.0.0.0:${EMAIL_HTTP_PORT}
      - MH_API_BIND_ADDR=0.0.0.0:${EMAIL_HTTP_PORT}
Enter fullscreen mode Exit fullscreen mode

docker-compose.yml

services:
  dev_app:
    extends:
      file: docker-compose.base.yml
      service: app
    profiles:
      - dev
    env_file: .env.dev
    depends_on:
      - dev_postgres
      - mailhog
    command: sh -c "npm ci && npm run start"

  test_app:
    extends:
      file: docker-compose.base.yml
      service: app
    profiles:
      - test
    env_file: .env.test
    depends_on:
      - test_postgres
      - mailhog
    command: sh -c "npm ci && npm test"

  dev_postgres:
    extends:
      file: docker-compose.base.yml
      service: postgres
    profiles:
      - dev
    volumes:
      - ./docker/pgdata:/var/lib/postgresql/data

  test_postgres:
    extends:
      file: docker-compose.base.yml
      service: postgres
    profiles:
      - test
    tmpfs:
      - /var/lib/postgresql/data

  mailhog:
    extends:
      file: docker-compose.base.yml
      service: mailhog
    profiles:
      - dev
      - test
Enter fullscreen mode Exit fullscreen mode

.env.dev

NODE_ENV=dev
PORT=3000
JWT_SECRET=dev_secret

POSTGRES_PORT=54321
POSTGRES_DB=development
POSTGRES_USER=admin
POSTGRES_PASSWORD=postgres
POSTGRES_URL=postgresql://admin:postgres@dev_postgres:54321/development

EMAIL_HOST=mailhog
EMAIL_SMTP_PORT=1025
EMAIL_HTTP_PORT=8025
Enter fullscreen mode Exit fullscreen mode

.env.test

NODE_ENV=test
PORT=3001
JWT_SECRET=test_secret

POSTGRES_PORT=54322
POSTGRES_DB=test
POSTGRES_USER=test
POSTGRES_PASSWORD=test
POSTGRES_URL=postgresql://test:test@test_postgres:54322/test

EMAIL_HOST=mailhog
EMAIL_SMTP_PORT=1025
EMAIL_HTTP_PORT=8025
Enter fullscreen mode Exit fullscreen mode

How to run it

One command per environment:

# Development
docker compose --env-file .env.dev --profile dev up

# Tests
docker compose --env-file .env.test --profile test up --exit-code-from test_app
Enter fullscreen mode Exit fullscreen mode

That's the whole interface. Now let's see why it works.

The shared base file

docker-compose.base.yml describes each service once, with only the settings that are the same in every environment: image, ports, volumes and basic configuration. There are no profiles, no names and no commands in it. It's not meant to be started on its own; it's a library of building blocks.

docker-compose.yml then builds the real services from those blocks with extends:

dev_postgres:
  extends:
    file: docker-compose.base.yml
    service: postgres
  profiles:
    - dev
Enter fullscreen mode Exit fullscreen mode

dev_postgres gets everything from the base postgres service and adds only what makes it different.

Why bother: the Postgres version, ports and settings now live in exactly one place. Upgrade the image once and every environment gets it. No more "which file is the right one?"

Tip: with extends, profiles and env files combined, it can be hard to see what Compose will actually run. docker compose config prints the final merged result with all variables filled in:

docker compose --env-file .env.dev --profile dev config
Enter fullscreen mode Exit fullscreen mode

Profiles

A profile is a label on a service. When you run docker compose --profile dev up, only services labelled dev start:

Profile Starts
dev dev_app, dev_postgres, mailhog
test test_app, test_postgres, mailhog

This is what replaces the copy-pasted docker-compose.test.yml. Both environments live in one file, side by side, and each one only adds what is actually different.

A few decisions worth explaining:

  • Postgres is per environment. dev_postgres mounts a folder, so my dev data survives restarts. test_postgres keeps its data in memory (tmpfs), so every test run starts with an empty database.
  • MailHog is shared. It's listed in both profiles, dev and test. It doesn't care which environment sends an email; it catches everything and judges nobody.

Why tmpfs and not just "no volume"? My first version simply left the volume out, and that's not enough. The postgres image declares its own volume, so Docker quietly creates one anyway. --exit-code-from stops the containers but doesn't remove them, so the next up starts the same container with the same data. A tmpfs mount is wiped every time the container stops, so the reset happens only for tests and never touches your dev data. Easy to check: insert a row in one test run, run the exact same command again without down, and the row should be gone.

One gotcha: a service without a profiles key starts with every profile. So give every service a profile, or you'll wonder why your test run also started something it shouldn't.

Env files

Each profile gets its own env file: .env.dev for dev and .env.test for test. The compose files stay the same; only the values change.

The variable names are identical in every environment. Dev and test both have PORT, JWT_SECRET, POSTGRES_URL and so on, just with different values:

Variable .env.dev .env.test
PORT 3000 3001
JWT_SECRET dev_secret test_secret
POSTGRES_PORT 54321 54322
POSTGRES_DB development test

So the structure of your config and secrets never changes, and your app never has to ask "which environment am I in?". It reads process.env and gets on with life.

A new environment is a new env file plus a small edit. Want staging? Create .env.staging, add a staging_app and staging_postgres that extend the base services, and give them a staging profile. That's it.

--env-file and env_file are two different things. They look almost the same but do different jobs:

  • --env-file .env.dev on the command line fills in the ${...} placeholders inside the compose files, like ports and passwords.
  • env_file: .env.dev inside a service passes the variables into the container, so the app can read them from process.env.

In this setup the app uses the same file for both, so everything about an environment lives in one place.

The same port inside and outside. A line like ${POSTGRES_PORT}:${POSTGRES_PORT} maps the host port to the same port inside the container. For that to work, the service has to listen on that port too, which is what PGPORT does for Postgres and the MH_*_BIND_ADDR variables do for MailHog. The result: one number per service, and dev and test never point at the same database.

Keep the real env files out of git. They hold secrets, and your future self will thank you.

The command key

command overrides what a container runs on start, and it's what lets one image do different jobs.

Both dev_app and test_app use the plain node image with the source code mounted in. No Dockerfile, no custom image to rebuild. The only difference is the command:

  • dev_app runs npm ci && npm run start and keeps the server running.
  • test_app runs npm ci && npm test, then exits.

Two small things matter here:

  • Use &&, not ;. With npm ci; npm test, the tests still run after a failed install and fail with a confusing error. && stops at the first failure.
  • --exit-code-from test_app stops the containers when the tests finish (it doesn't remove them) and returns the tests' exit code. Your terminal (or CI) gets a clear pass or fail.

Also, depends_on only controls the start order. It doesn't wait until Postgres is actually ready to accept connections. In practice npm ci is slow enough that Postgres is ready long before it finishes (finally, a use for slow installs). For a bulletproof setup, add a Postgres healthcheck and use condition: service_healthy.

Try it yourself

  1. Take any Node.js project with start and test scripts in package.json.
  2. Add the four files from The full setup, and add .env.dev and .env.test to .gitignore.
  3. Start development: 
   docker compose --env-file .env.dev --profile dev up
Enter fullscreen mode Exit fullscreen mode

Your app is on http://localhost:3000, and every email it sends shows up at http://localhost:8025.

  1. Run the tests: 
   docker compose --env-file .env.test --profile test up --exit-code-from test_app
Enter fullscreen mode Exit fullscreen mode
  1. Clean up when you're done: 
   docker compose --env-file .env.dev --profile dev --profile test down
Enter fullscreen mode Exit fullscreen mode

A full working example lives in my repo: kot-1999/flowers_shop_be.

Copy-paste is still allowed. Just not for your infrastructure. 😉

How do you handle multiple environments with Docker Compose? Let me know in the comments.

Top comments (2)

Collapse
 
launchgatecheck profile image
Launch Gate •

The shared base and the --env-file/env_file distinction are useful. I'd put a repeat-run check next to "every test run starts with a clean database": insert a sentinel row in run one, then run the exact test command again without down and assert the row is gone. --exit-code-from stops containers; it doesn't remove them, and up can reuse them and their mounted data. That separates "no volume declared in this YAML" from "fresh database on each invocation". I'd make the reset test-only rather than adding down -v to the combined dev/test cleanup command.

Collapse
 
oleksandr_kashytskyi_a630 profile image
Oleksandr Kashytskyi •

Thanks, you're right, and I've reproduced it.

Two runs of the exact test command, no down in between: the second run still had the first run's data. The postgres image declares its own VOLUME, so Docker creates an anonymous volume even though the YAML doesn't, and --exit-code-from only stops the containers.

I've updated the post: test_postgres now keeps its data on a tmpfs, which is wiped every time the container stops. The reset stays test-only, dev data is never touched, and I reworded the --exit-code-from line too.

Thanks for the careful read!