DEV Community

Shan Desai
Shan Desai

Posted on • Originally published at shantanoo-desai.github.io

Exploring the Marshland of Docker Compose Versions with single quotes, special characters and environment variables

#devops #docker #vagrant #ansible

Problem

Since Docker's Compose V2 General Availability announcement, a lot of things have changed which might lead to
Incompatibility in your currently running application stacks.

Compose V2: what changes?

  • instead of docker-compose the syntax will be docker compose
  • docker-compose (v1) was written in Python, docker compose (v2) is in Go
  • Compose V1 is official EOL (End Of Life) aka. Deprecated

There are lot of fine things in Compose V2 but this post isn't about it. This write-up takes you through the murky
marshland between Compose V1 and Compose V2 and how things might start you break when you decide to jump to the
cool V2 vers without doing some thorough checks in your stack application

Swiss Knife for the Exploration

Tools you will need:

  • Hashicorp Vagrant
  • Ansible
  • Different versions of Docker Compose (v1 as well as v2)

Vagrant

Instead of setting up a pain-staking Virtual Machine, we will leverage a bit simplified and inexpensive Vagrant Machine
with Ubuntu 20.04 (focal amd64) base image.

Within this image we will install the following:

  • Docker Engine 20.10.17
  • Docker Compose v1: v1.25.4 build 8d51620a and v1.29.2 build 5b3cea4c
  • Docker Compose v2: 2.6.0

Ansible

I have Ansible installed on my host machine

ansible --version
ansible [core 2.13.1]
Enter fullscreen mode Exit fullscreen mode

Ansible will install all the spicy Compose versions into the Vagrant Machine for us

Why are we doing this?

If you have containers that require passwords at rest e.g. in .env environment files to be encrypted and with
special characters like the $ in them, we are going for ride which will make you want to bookmark this write-up.

It turns out, I had a system running Compose v1 (v1.25.4) and upon upgrading to Compose v2 (2.6.0) I wasn't able
to login to my Node-RED container which had authentication setup already.

Node-RED Container

Node-RED requires that the administrator password must be encrypted in bcrypt type scheme and the password looks
something like your dog walked over your keyboard.

$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
Enter fullscreen mode Exit fullscreen mode

This looks fairly okay, but the menace is the $ characters in them.

Let's dive deep shall we!

Code

Vagrantfile

I am using a Linux Machine with Virtual Box as a provider

# -*- mode: ruby -*-
# vi: set ft=ruby :


Vagrant.configure("2") do |config|

  # Use Ubuntu 20.04 as base
  config.vm.box = "ubuntu/focal64"

  config.vm.box_check_update =false

  # Synchronize all the docker-compose related file to `/vagrant` dir in VM
  config.vm.synced_folder ".", "/vagrant"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
  end

  # Setup the VM with all the different variants of Docker Compose in them
  config.vm.provision "ansible" do |a|
    a.verbose = "v"
    a.playbook = "./docker-compose-playbook.yml"
  end
end
Enter fullscreen mode Exit fullscreen mode

Ansible Playbook: docker-compose-playbook.yml

We will let Ansible install all the required Docker Compose versions for us in the VM through the playbook.

  • Docker Compose v1.25.4 will be available in the VM as CLI docker-compose-125
  • Docker Compose v1.29.2 will be available in the VM as CLI docker-compose-129
  • Docker Compose v2.6.0 will be available in the VM as CLI docker compose 
---
- hosts: all
  become: yes
  tasks:
    - name: install prerequisites
      apt:
        name:
          - apt-transport-https
          - ca-certificates
          - curl
          - gnupg-agent
          - software-properties-common
        update_cache: yes
    - name: add apt-key
      apt_key:
        url: https://download.docker.com/linux/ubuntu/gpg
    - name: add docker repo
      apt_repository:
        repo: deb https://download.docker.com/linux/ubuntu focal stable
    - name: install docker
      apt:
        name:
          - docker-ce
          - docker-ce-cli
          - containerd.io
          - docker-compose-plugin
        update_cache: yes
    - name: add userpermissions
      shell: "usermod -aG docker vagrant"
    - name: Install docker-compose v1.29.2 from official github repo
      get_url:
        url : https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
        dest: /usr/local/bin/docker-compose-129
        mode: 'u+x,g+x'
    - name: change compose v1.29.2 user permissions
      shell: "chmod +x /usr/local/bin/docker-compose-129"
    - name: Install docker-compose v1.25.4 from official github repo
      get_url:
        url: https://github.com/docker/compose/releases/download/1.25.4/docker-compose-Linux-x86_64
        dest: /usr/local/bin/docker-compose-125
        mode: 'u+x,g+x'
    - name: change compose v1.25.4 user permissions
      shell: "chmod +x /usr/local/bin/docker-compose-125"
Enter fullscreen mode Exit fullscreen mode

Files for reproduction of incompatibility

We wont be spinning to much complex stuff so a simple compose file and the relevant .env.node-red
file with the encrypted password should suffice

docker-compose.yml 

version: '3.7'
services:
  node-red:
    image: nodered/node-red:2.2.2
    container_name: test-nodered
    env_file:
      - .env.node-red
Enter fullscreen mode Exit fullscreen mode

.env.node-red 

# Encrypting plain-text password with value `password` with bcyrpt
NODERED_ADMIN_PASSWORD=$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
Enter fullscreen mode Exit fullscreen mode

Compatibility Checks

Let's bring the Vagrant Machine up using:

vagrant up
Enter fullscreen mode Exit fullscreen mode

this will download the base ubuntu image if you previously don't have it on your host machine,
provision it with all the docker compose versions in it.

Login into the Machine:

vagrant ssh
Enter fullscreen mode Exit fullscreen mode

Once into the Machine head to the /vagrant directory, and just to be safe we check if all the
required compose versions are available or not.

$ cd /vagrant

$ docker-compose-125 --version
$ docker-compose-129 --version
$ docker compose version
Enter fullscreen mode Exit fullscreen mode

Checks: Password not encloses in Single Quotes

As the .env.node-red file currently shows that my encrypted password is not enclosed in single quotes

Let's understand what Compose thinks of the value of the environment variable.

Compose v1.25.4 (without single quotes) 

docker-compose-125 config
Enter fullscreen mode Exit fullscreen mode

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'
Enter fullscreen mode Exit fullscreen mode

NO Problem! My dollar characters are escaped by docker-compose. I can login with password password

Compose v1.29.2 (without single quotes) 

docker-compose-129 config
Enter fullscreen mode Exit fullscreen mode

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'
Enter fullscreen mode Exit fullscreen mode

NO Problem! My dollar characters are escaped by docker-compose. I can login with password password

Compose v2 without single quotes) 

docker compose config
Enter fullscreen mode Exit fullscreen mode

produces

name: node-red-app
services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: a$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
    networks:
      default: null
networks:
  default:
    name: node-red-app_default
Enter fullscreen mode Exit fullscreen mode

Oh NO! What happened here? Password looks eaten up. The YAML file looks different too!

Checks: Password enclosed in single quotes

Now let's enclose our encrypted password between single quotes, so our .env.node-red file
should look like:

NODERED_ADMIN_PASSWORD=$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
Enter fullscreen mode Exit fullscreen mode

Compose v1.25.4 (with single quotes) 

docker-compose-125 config
Enter fullscreen mode Exit fullscreen mode

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: '''$$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.'''
    image: nodered/node-red:2.2.2
version: '3.7'
Enter fullscreen mode Exit fullscreen mode

Oh NO! Why are there more single quotes than expected. This is not going to let me login!

Compose v1.29.2 (with single quotes) 

docker-compose-129 config
Enter fullscreen mode Exit fullscreen mode

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'
Enter fullscreen mode Exit fullscreen mode

NO problem! here this will work fine as the dollar characters are escaped safely by compose

Compose v2 (with single quotes) 

docker compose config
Enter fullscreen mode Exit fullscreen mode

produces

name: node-red-app
services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
    networks:
      default: null
networks:
  default:
    name: node-red-app_default
Enter fullscreen mode Exit fullscreen mode

NO Problem! Phew ! This will work just fine !

Result: Compose Compatibility Matrix

Compose Version / Values 1.25.4 1.29.2 2.6.0
with Single Quotes ❌ ✅ ✅
without Single Quotes ✅ ✅ ❌

What did we learn today, kids?

No Single Quotes:

  • Docker Compose v1.25.4 was able to resolve special characters without having enclose them in single quotes
  • Docker Compose v1.29.2 will also work just fine for values not enclosed in single quotes
  • Docker Compose v2 will try go hay-wire (try to resolve the value after the $ sign) if you do not use things in single quotes

With Single Quotes:

  • Docker Compose v1.25.4 will literally take things enclosed in single quotes as part of your environment variable and will
    break your Container's Login logic

  • Docker Compose v1.29.2 will work well

  • Docker Compose v2 will work well

In a nutshell

DON'T JUMP THE GUN, when moving from different V1 to V2 in Compose. Do some throrough research as well as experiments.

Since Compose V1 is now deprecated, when transitioning to Compose V2 check all your environment variable files and if they require
special characters to be either enclosed in single quotes or not. Estimate your downtimes and conduct isolated tests when required.

Rule of thumb from Compose V2 onwards: if it got some special char in them password, pack em between single quotes

but to each their own!

Github Gist

GitHub Gist available publicly

Top comments (0)

Read next

muhammad_ahmad_khan profile image

Integrating Multiple EKS Clusters with ArgoCD for Simplifying Kubernetes Operations

Muhammad Ahmad Khan -

art_ptushkin profile image

How to configure DNS caching timeout in a Spring Boot docker image?

Artem Ptushkin -

hirushafernando profile image

Automate Docker Image Building , Tagging And Publishing With Github Actions

Hirusha Fernando -

moozzyk profile image

How JavaScript helped me buy my dream boat

Pawel Kadluczka -