Dotenvx with Docker, the better way to manage project environment variables with secrets

Today we're going to be discussing the easiest way to integrate Dotenvx into a project that heavily utilizes Docker services. Dotenvx is an open-source environment variable management tool that specializes in handling secrets and was created by the developer of the popular tool Dotenv. Dotenvx serves as the successor to Dotenv.

Let's begin

There are two primary challenges when using Dotenvx with Docker Compose:

1. How can I get my decrypted environment variable secrets to Docker Compose, so that I can configure the Docker Compose tool itself, and retain the ability to pass along and access environment variables through the compose.yml file?

2. How can I get my decrypted environment variables inside my Docker Compose service container, so that they are available to the service process and container filesystem at runtime?

The obvious way to do it

The most common way to inject environment variables into services is to utilize Dotenvx's run argument, for example:

dotenvx run -f .env.dev -- python webserver.py

But since we're using Docker, this is what we end up with:

dotenvx run -f .env.dev -- docker compose up --build

Now we've managed to get decrypted environment variables into Docker Compose, so that we can use them inside our compose.yml file like so:

# compose.yml
services:
   postgres:
      image: postgres:latest
      environment:
         # $POSTGRES_PASSWORD is stored as an encrypted password in .env.dev
         # But dotenvx has decrypted it for us and injected it, so we can use it here
         POSTGRES_PASSWORD: $POSTGRES_PASSWORD

We can also pass decrypted env vars into our image build stage if we need to access them in our Dockerfile, like so:

# compose.yml
services:
   custom_service:
      build:
         target: custom_service
      args:
         SUPER_SECRET_FROM_ENV: $SUPER_SECRET_FROM_ENV

But how do we inject the env vars into the service container at runtime and solve the second challenge we mentioned earlier?

Headaches

Normally, we'd rely on our trusty env_file Docker Compose directive, but that won't work because we'll only be able to load our encrypted file into the container if we can point it to a file on disk. As it stands, we're decrypting our env vars on the fly and injecting them into the Compose tool.

Here's the standard way people deal with this: inserting a copy of the Dotenvx binary into the container filesystem and decrypting and injecting for a second time, into the service container's process. However, this has a lot of downsides:

1. Every single Docker service that utilizes secrets now needs its own build stage in our Dockerfile, and a custom image built from our pre-image with the addition of the Dotenvx binary.

For example, you can't just declare the Postgres service like we did in the first example above, using the postgres:latest image. We now need to download and install the Dotenvx binary into that filesystem.

1. We need to either bind mount our encrypted env file, or copy it into the image, so that it's available to the container at runtime for decryption. We can't use the env_file directive because the running service will only see encrypted values. If we want our container's version of Dotenvx to decrypt our environment, it needs to be able to point to a physical file.

Bind mounting is the better choice out of these, to avoid having to rebuild on every environment change, and to avoid having multiple sources of truth due to duplicating the env files.

1. We need to pass our private key to Docker Compose, so it can be injected into our container's runtime, ideally without leaking it into shell history logs or having to manually enter it on the command line.

Here's what it looks like:

# Dockerfile
FROM postgres:16.2-bookworm AS postgres

RUN apt-get update 
   && apt-get install -y curl
   && curl -fsS https://dotenvx.sh/ | sh

RUN apt-get remove curl && apt-get autoremove

USER postgres

# compose.yml
services:
   postgres:
      build:
         target: postgres
      environment:
         POSTGRES_USER: ${POSTGRES_USER}
         POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
         POSTGRES_DB: ${POSTGRES_DB}
      ports:
         - ${POSTGRES_PORT}:${POSTGRES_PORT}
      volumes:
         - .env.dev:/app/.env.dev:ro
         - dev_postgres:/var/lib/postgresql/data
      command: ["env", "DOTENV_PRIVATE_KEY=$KEY", "dotenvx", "run", "-f", "/app/.env.dev", "--", "/usr/local/bin/docker-entrypoint.sh", "postgres"]

$: KEY=your_private_key dotenvx run -o -f .env.dev -- docker compose up postgres

If you're paying attention you may have noticed the strange entrypoint argument. This was a result of the following: https://github.com/dotenvx/dotenvx/issues/142 - I found multiple services that had niggling issues like this.

These problems arise as a byproduct of our requirement to inject the Dotenvx binary into these image filesystems.

---

So there we are, we're finished, and it works... But I can't help but hate it.

The command directives and Dockerfile are messy and bloated, I'm having to work around issues getting binaries into images that aren't equipped for customization, I have a mandatory bind mount to my env file, and I have two simultaneously running binaries of the same program serving as the gatekeeper to my Docker Compose and container processes.

Let's reassess the approach

Firstly, a little bit on why it's even worthwhile messing with all of this in the first place.

Storing the env files in source control has the following benefits:

• It's particularly useful when there are multiple environments to configure, each with their own unique settings.
• Reproducibility is a lot higher if developers are using the same configurations, and sharing configurations promotes reproducible environments.
• It's great keeping a fine-grained history of configuration changes.
• It allows for improvements in documentation, by encouraging inline comments.
• It simplifies build processes - no need to dynamically build env files in CI or prod by pulling secrets from a secret store.

---

How to make this better

Let's figure out a way to clean up the previous workflow, to obtain the perks without the bloat.

In production, we don't need dotenvx run at all. Production servers and their containers are already hardened and have permissions to the Secrets Manager. Storing secrets within the production host environment, or on disk there, is not a security concern because we have isolation through our Docker containers. The server needs these decrypted values to function, anyway.

So all we need to do for prod is simply store the private key on the server (preferably in the secrets manager), and have a bash script copy and decrypt the .env.production file as a part of the continuous deployment flow. Then we can use the env_file directive to load the decrypted env file into our production containers, without the need to invoke the Dotenvx binary at any point.

But how can we do this for development, where things are a little different? In development, we don't want developers having to manually decrypt .env files and leave them on disk - it's a security risk. They would also have to manually decrypt the files every time they change their env files.

The solution

Here's my solution to the downsides we discovered earlier, and the new workflow that solves them.

An env file watcher that dynamically decrypts your env file on file change using inotifywatch and copies it to a secure, ephemeral, in-memory filesystem called ramfs, so that we can point our Docker env_file directive to this decrypted file, and have real-time env file updates propagate through.

#!/usr/bin/env bash

#
# ./watch.sh [env-files...]
#

function show_help() {
    cat << EOF
Usage: $0 [options] [file...]

Options:
  -h, --help                Show this help message and exit.

Description:
  This script monitors one or more environment files for changes.
  When a change is detected, it will decrypt the monitored file using dotenvx,
  and store the decrypted file in a ramfs memory-based filesystem mount.
  If no file is specified, it defaults to watching '.env.dev'.

Examples:
  1. Watch the default environment file:
     $0

  2. Watch a specific environment file:
     $0 .env.dev

  3. Watch multiple environment files:
     $0 ~/.env.dev /path/to/.env.prod

This command allows you to specify custom environment files to monitor. If no arguments are provided,
it assumes the file '.env.dev'. Multiple files can be watched by providing each as an argument separated
by spaces.
EOF
}

_mount_ramfs() {
  local mount_point=$1

  # Create a 20mb ramfs mount if we don't already have one to use
  if ! mountpoint -q "$mount_point"; then
    sudo mkdir -p "$mount_point"
    sudo mount -t ramfs -o size=20M,mode=1777 ramfs "$mount_point"
  fi
}

_watcher_cleanup() {
  local mount_point=$1

  # Cleanup background inotifywatcher jobs
  while read p; do
    kill $p 2>/dev/null || true
  done <"/tmp/env_watch_pids.txt"
  rm "/tmp/env_watch_pids.txt" 2>/dev/null || true

  echo ""
  echo "Deleting decrypted env files from memory: $mount_point"
  rm -rf "$mount_point" >/dev/null || true
  rm /tmp/env_watch.lock >/dev/null || true
}

_setup_watcher() {
  local file=$1
  local mount_point=$2

  # Initial run to get the decrypted file into the ramfs mount
  _decrypt_and_save "$file" "$mount_point" true

  # Setup watcher to decrypt on modification to env file
  inotifywait -q -m -e close_write -e delete_self -e move_self "$file" > >(while read path action; do
    if [[ "$action" == "DELETE_SELF" || "$action" == "MOVE_SELF" ]]; then
      echo "Env file deleted or moved. Terminating watcher for $file..."
      break
    fi
    _decrypt_and_save "$file" "$mount_point"
  done) &

  decrypted_file="${mount_point}/$(basename "${file}").decrypted"
  echo "Env file watcher started: $file -> $decrypted_file"
  echo $! >>/tmp/env_watch_pids.txt
}

_decrypt_and_save() {
  local encrypted_file=$1
  local mount_point=$2
  local decrypted_file="${mount_point}/$(basename "${encrypted_file}").decrypted"

  # Decrypt and convert JSON to .env format
  dotenvx get -f "$encrypted_file" | jq -r 'to_entries | .[] | "\(.key)=\(.value)"' >"$decrypted_file"
  if [ $? -eq 0 ]; then
    if [ $# -eq 2 ]; then
      echo "Detected modification in $encrypted_file, decrypting and updating $decrypted_file ..."
    fi
  else
    echo "Failed to decrypt $encrypted_file"
    return 1
  fi
}

function env_watch {
  set +e # disable exit on error to ensure cleanup doesn't get skipped
  local sub_dir="${RAMFS_SUBDIR:-dotenvx}"
  local mount_point="/mnt/ramfs/${sub_dir}"

  # Check for another instance running
  if [ -f /tmp/env_watch.lock ]; then
    echo "Another instance of env:watch is already running. If this is not the case, please check running processes and remove lockfile: /tmp/env_watch.lock"
    exit 1
  fi

  # Check if inotifywait and jq are installed
  if ! command -v inotifywait >/dev/null || ! command -v jq >/dev/null || ! command -v dotenvx >/dev/null; then
    echo "This script requires inotify-tools, jq, and dotenvx. Please install them first."
    exit 1
  fi

  # Error if no args supplied to ./run env:watch
  if [ $# -lt 1 ]; then
    echo "Warning: No env file supplied. Defaulting to .env.dev, see --help for more info."
    set -- ".env.dev"
  fi

  for env_file in $@; do
    if [[ "$env_file" == *".env.prod"* || "$env_file" == *".env.production"* ]]; then
      echo "Running on production env files is insecure. The env:watcher should only be used on dev."
      exit 1
    fi
    if [ ! -f "$env_file" ]; then
      echo "Error: '$env_file' does not exist."
      exit 1
    fi
  done

  touch /tmp/env_watch.lock

  # Make sure we don't have a stale pids file
  rm "/tmp/env_watch_pids.txt" 2>/dev/null || true

  # Set up ramfs mount and exit cleanup
  _mount_ramfs "/mnt/ramfs"
  trap "_watcher_cleanup '/mnt/ramfs/$sub_dir'" EXIT

  # Ensure subdirectory exists
  mkdir -p "$mount_point"

  # Main loop to setup watchers for each file
  pids=()
  for env_file in $@; do
    _setup_watcher "$env_file" "$mount_point"
    pids+=($!)
  done

  echo "Env file watchers running and waiting for file changes. Ctrl+C to quit..."

  # If all watchers terminate, exit app
  wait ${pids[@]}
}

# Main script logic
case "$1" in
    -h|--help)
        show_help
        ;;
    *)
        env_watch "$@" 
        ;;
esac

And now we can clean up our compose file. All we need is to add the env_file directive and remove the rest of the clutter we added earlier:

# compose.yml
services:
   postgres:
      image: postgres:latest
      env_file:
         - /mnt/ramfs/dotenvx/.env.dev.decrypted
      environment:
         # $POSTGRES_PASSWORD is automatically decrypted inside .env.dev.decrypted
         POSTGRES_PASSWORD: $POSTGRES_PASSWORD

And now we can clean up our Dockerfile, and delete all of the custom images we made when we had to inject Dotenvx binaries. We don't need them anymore.

Example run of the above bash script:

$: ./watch.sh .env.dev
Env file watcher started: .env.dev -> /mnt/ramfs/dotenvx/.env.dev.decrypted
Env file watchers running and waiting for file changes. Ctrl+C to quit...
Detected modification in .env.dev, decrypting and updating /mnt/ramfs/dotenvx/.env.dev.decrypted ...
^C
Deleting decrypted env files from memory: /mnt/ramfs/dotenvx

You can find the most recent version of the script on my Github repo: https://github.com/nullbio/dotenvx-watcher.