DEV Community: Shan Desai

Deep Dive with Ansible: Patching an Ansible Collection

Shan Desai — Fri, 19 Apr 2024 12:26:13 +0000

Problem

Currently, I have been developing a lot with Ansible at work for Automation logic
for servers and industrial hardware. It is also a fantastic tool to performing testing
of provisioned systems. One such test was checking whether a Grafana container was configured
correctly through a Docker Compose project. The Grafana container was behind a Traefik reverse-proxy
configured with TLS-termination for some internal self-signed certificates, where it wasn't required
to check certificate validation.

There exists a nice community.grafana Ansible Collection that can help do lookup of some
provisioned Grafana Dashboards.

The usage based on the Documentation was quite simple:

- name: Gather Information about current Grafana Dashboards
  ansible.builtin.set_fact:
    existing_grafana_dashboards: |
      {{ lookup(
        'community.grafana.grafana_dashboard',
        'grafana_url=https://{{ ansible_host }}/grafana grafana_user=admin grafana_password={{ grafana_admin_pass }} search=fooDash'
      }}
 - ansible.builtin.debug:
     var: existing_grafana_dashboards

This plugin would simply connect to each Grafana dashboard instance in my inventory and try to search for existence of dashboards
with name fooDash and store them in a local Ansible Fact for the Playbook called existing_grafana_dashboards.

This could have worked out of the box, but as it turned out since the URL was HTTPS and my instance was hosted with a self-signed
certificate, I got URI verification failures, which is quite common when working with Ansible and URIs.

Instincts told me to just add a validate_certs=false parameter to the lookup plugin. So a refactor would look something like:

- name: Gather Information about current Grafana Dashboards
  ansible.builtin.set_fact:
    existing_grafana_dashboards: |
      {{ lookup(
        'community.grafana.grafana_dashboard',
        'grafana_url=https://{{ ansible_host }}/grafana grafana_user=admin grafana_password={{ grafana_admin_pass }} search=fooDash'
        validate_certs=false
      }}
 - ansible.builtin.debug:
     var: existing_grafana_dashboards

However it turns out I still got the same SSL verification error and upon inspecting the Upstream code, the plugin did not have an
implementation for the validate_certs logic, which is quite common in Ansible's URI lookup plugin.

With a proper Issue report at the collection's repository, I tried to figure out how to patch this requirement to the lookup plugin.

Local Setup for Ansible Collections

Based on the Repository's documentation, the easiest way would be to have the collection cloned in to the COLLECTIONS_PATH on a work-machine
and have the collection reflect in a project to test the changes out.

The COLLECTIONS_PATH is an Ansible Configuration which points to a directory where all other collections exists. If not explicitly set
it can vary from where the collections exists in the system.

If Ansible is installed via pip install --user ansible then chances are the path to the collections is

~/.local/lib/python<version>/site-packages/ansible_collections

If Ansible is installed globally either via some distro package manager it might be somewhere in:

/usr/local/lib/python<version>/dist-packages/ansible_collections

TIP: Use ansible-galaxy collection list to figure out where are the collections pointed at when using ansible.
The first line shows you the path where the collections are located on the machine.

Since community.grafana already exists on the work machine, it would be wise not to clone the repository in the same path
as where all the collections already exist.

Overriding the Collections Path for Development

If we read the documentation for the COLLECTIONS_PATH again, it mentions:

if COLLECTIONS_PATHS includes {{ ANSIBLE_HOME ~ "/collections" }}, and you want to add my.collection to that directory,
it must be saved as {{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}

Here ANSIBLE_HOME would be ~/.ansible directory. So in my case, it would be the following:

~/.ansible/collections/ansible_collections/community/grafana

So here are the steps to setting up the override logic for the Ansible Collection logic locally:

Create a fork of the upstream repository on GitHub
Create the base collections directory if it doesn't exist:
```
mkdir -p ~/.ansible/collections/ansible_collections
```
Clone to fork in such a way that the contents of the repo are placed under ansible_collections/community/grafana
```
git clone <FORK_URL> ~/.ansible/collections/ansible_collections/community/grafana
```
this could be generally described for other ansible collections as follows:

   git clone <URL> ~/.ansible/collections/ansible_collections/<namespace>/<collection>

Now you can make changes to the codebase in this particular repository and test it somewhere else locally to see if the changes
work as expected.

Setting Up a Local Test Base

In order to test the changes create a simple directory where your development codebases exist e.g. ~/Development/ansible/grafana_patch

In the particular directory add the following Ansible configuration file called ansible.cfg:

[defaults]
collections_path = ~/.ansible/collections/ansible_collections

This is a local override to tell ansible, that add the collections that now exist in our ~/.ansible/collections/ansible_collections
for the local project too.

This is verified by performing:

$ pwd
/home/User/Development/ansible/grafana_patch
$ ansible-galaxy collection list
# /home/User/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.grafana 1.8.0
# /usr/local/lib/python3.8/dist-packages/ansible_collections
Collection                    Version
----------------------------- -------
### Other collections omitted for brewity

Now a simple playbook called test_dashboard_lookup.yml

- hosts: localhost
  tasks:
    - ansible.builtin.set_fact:
        local_dashboard: |
          {{ lookup(
            'community.grafana.grafana_dashboard',
            'grafana_url=https://localhost:3000 grafana_user=admin grafana_password=admin search=foo',
            validate_certs=false
          }}
    - ansible.builtin.debug:
        var: local_dashboards

can be used to verify if things are working locally with a default Grafana Container setup with self-signed certs using:

ansible-playbook test_dashboard_lookup.yml -vvvv

Result

Based on the Issue filed by me, there now exists a patch that provides the necessary feature for validation / not validating
SSL certificates.

New Tool, New Environment, New way to Patch something => Better to document!

Forward Compatibility for Mosquitto MQTT Broker with Docker Compose v2

Shan Desai — Mon, 23 Oct 2023 20:36:08 +0000

Planning

While working on a personal project Komponist, I was due to update
Mosquitto MQTT Broker due to a CVE and found some interesting changes
that will impact me in the future when it comes to configuring the Broker. This post
provides a solution to make the Broker compatible with future versions using new and
less visited concepts in Docker Compose v2, namely:

Docker Init Containers
Using the include feature in Docker Compose files

Observations

This section provides information on what currently is observed post upgrading the
Mosquitto Broker to v2.0.18.

Current Docker Compose File

This is my current docker-compose.mosquitto.yml with the following file structure
generated by Komponist:

|- docker-compose.mosquitto.yml
|-- mosquitto
    |--- acl
    |--- users
    |--- mosquitto.conf

The Compose file content is:

services:
  mosquitto:
    image: docker.io/eclipse-mosquitto:2.0.18
    container_name: komponist_mosquitto
    configs:
      - mosquitto_conf
      - mosquitto_acl
      - mosquitto_users
    command: mosquitto -c /mosquitto_conf
    security_opt:
      - "no-new-privileges=true"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

configs:
  mosquitto_conf:
    file: ./mosquitto/mosquitto.conf
  mosquitto_acl:
    file: ./mosquitto/acl
  mosquitto_users:
    file: ./mosquitto/users

Bringing the container up using:

docker compose -f docker-compose.mosquitto.yml up

The logs will throw some warnings such as the following:

Warning: File /mosquitto_users has world readable permissions. Future versions will refuse to load this file.

Warning: File /mosquitto_users owner is not mosquitto. Future versions will refuse to load this file.
...
...

So for future versions the files need to have a specific user and file permissions
to make the current configuration files valid.

Caveats

In systems where I do not have superuser privileges to create a new user mosquitto whose
UID/GID should be 1883 and changing file permissions is not an option, the only way to tackle
such a solution is through Docker Init Containers.

Docker Init Containers are like sidecar containers in Kubernetes or a patching container where
it initializes a state (here specific configuration file) and then starts the core container.

This feature has existed for a while, but it is not well documented in the Compose Specifications
with some practical examples. In essence, we will use the depends_on service dependency logic
between a generic alpine container that will:

change the user for the core configuration files
change the file permissions for the core configuration files
pass on the core files to the eclipse-mosquitto broker using shared volumes

Once this container is mounted with the files and is run successfully, the adapted configuration
files will be passed on to the main docker container for the Mosquitto MQTT Broker.

Solution

We keep the file structure same as previously mentioned but we create a new Compose file
called docker-compose.mosquitto-init.yml file as follows:

services:
  mosquitto_init:
    image: alpine:3.18
    container_name: mosquitto_init
    command: sh -c 'chmod 0400 -R /config/ && chown 1883:1883 -R /config/'
    volumes:
      - ./mosquitto/mosquitto.conf:/config/mosquitto_conf
      - ./mosquitto/users:/config/mosquitto_users
      - ./mosquitto/acl:/config/mosquitto_acl

We are mounting the configuration files into the init container under the /config directory
and changing the file persmissions and ownership for the files in this init container.

NOTE: we will have to refactor out the Docker Configs section previously configured in
original docker-compose.mosquitto.yml file into the docker-compose.mosquitto-init.yml file

We now refactor the mosquitto.conf file a bit to accept the new location for the files
as follows:

# MQTT Port Listener
listener    1883
protocol    mqtt

# Authentication
allow_anonymous     false
password_file       /config/mosquitto_users

# Authorization
acl_file    /config/mosquitto_acl
# Logging Configuration
log_timestamp true
log_type all

Note the /config as prefix to the ACL and password file.

Final refactoring takes place in the docker-compose.mosquitto.yml file as follows:

include:
  - docker-compose.mosquitto-init.yml

services:
  mosquitto:
    image: docker.io/eclipse-mosquitto:2.0.18
    container_name: komponist_mosquitto
    entrypoint: mosquitto -c /config/mosquitto_conf
    depends_on:
      mosquitto_init:
        condition: service_completed_successfully
    security_opt:
      - "no-new-privileges=true"
    volumes_from:
      - mosquitto_init
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

Upon bringing the stack up again:

docker compose -f docker-compose.mosquitto.yml up

the Warnings should be gone and your config files that were valid prior to
version 2.0.18 will still be compatible for the upcoming versions (unless the development
of mosquitto demands something more in the future).

In a Nutshell, the main mosquitto service is now dependent on the mosquitto-init service
via the depends_on spec and core files are exchanged from the init container to the main container
via volumes_from spec.

NOTE: this approach will change the file permissions / owner for the core configuration files
on the host machine since they are volume mounted.

At the time of writing this post, this is the only approach I have tested out.

If you have read this post and have better approaches in mind, connect with me on LinkedIn and share
your solutions or improve upon the method. I would love to hear your feedback.

GitHub Gist

the Gist of the code can be found here

Generating Mosquitto MQTT Broker Credentials with Ansible

Shan Desai — Wed, 16 Aug 2023 11:37:23 +0000

Challenge

This is a development note from a personal, open-source project called Komponist.
The aim of the project is to provide IoT / IIoT Developers a platform that can be used locally as well as in production using commonly used containers
in the Edge computing space such as:

Time-series Databases (InfluxDBv1/v2, QuestDB, TimescaleDB)
Data processing tools (Node-RED, Telegraf)
Dashboard visualization / monitoring (Grafana)
Message Broker (Eclipse-Mosquitto) There are standard tools / software used quite ubiquitously with containerization technology like Docker / Compose v2.

One major challenge that developers face is to maintain a lot of files that are needed to bring these tools / software up on initialization.
These files mostly contain user credentials, initialization scripts and configurations needed to start these containers correctly and according to user specifications.
Additionally, each container of these tools / software are very distinct and may rely on various setups, markup languages (YAML, TOML, INI) etc.
and there is rarely any coherence available between one tool against the other.

Komponist solves this issue by add a logical configuration generation tool over the tools and relies only on two core files
that describes the complete stack along side the respective credentials and configurations.
It uses the easy to use ansible configuration management tool to generate the respective docker-compose files, credentials file, YAML and TOML files for the software
and prepares the stack within a couple of minutes either locally or within a group of IoT edge devices.

This writeup focuses more on the Mosquitto MQTT Broker. The writeup describes how to leverage Ansible and Jinja2 powered Templating engine to generate Mosquitto Broker's
credentials.

NOTE: This writeup requires some basic to intermediate knowledge of Ansible as a tool.

Mosquitto Broker's Credentials Generation

`mosquitto_passwd` CLI native

According to the mosquitto broker's authentication methods documentation
it is possible to generate password files when one either has the mosquitto_passwd CLI installed locally / remotely on the IoT devices.
For example, the following plain-text passwd.txt file:

user1:superSecretPass1
user2:superSecretPass2

will encrypt the plain-text passwords into hashes that are acceptable by the mosquitto broker via the mosquitto-passwd CLI using:

mosquitto_passwd -U /path/to/passwd.txt

However, it relies on the existence of mosquitto_passwd tool to be available locally.

`mosquitto_passwd` CLI via Docker Container

The same tool can be used via a Docker container where the plain-text password passwd.txt file can be encrypted via spinning a docker container
locally and mounting the plain-text file to a dedicated path into it. An example with the same passwd.txt can be:

docker run --rm -v /path/to/passwd.txt:/mosquitto/config/users \
eclipse-mosquitto:2.0.15 \
mosquitto_passwd -U /mosquitto/config/users

Since the file is mounted into the container, any changes within the container will be reflected on the passwd.txt on the host i.e., the plain-text passwords will be encrypted accordingly.

Both these methods, require dependency on either the mosquitto_passwd CLI or the eclipse-mosquitto Docker Image to exist locally beforehand.

Independent Credential Generation for Broker

Upon observing the code base for eclipse/mosquitto it turns out the password hash generation relies on two methods:

SHA512 + additional salting and base64 encryption logic
SHA512 type PBKDF2 encryption logic

The details of the these two methods are beyond the scope of the write up but the implementations for them are quite common and can be found in every programming language.

Since Komponist relies on Ansible and Jinja2 templating engine which are directly reliant on Python3.x as a programming language, the implementation is rather trivial thanks to passlib python package.
passlib provides the implementation of PBKDF2_SHA512 logic with all the necessary encoding logic needed for the hashing of plain-text passwords that are acceptable by the Broker.

Templates for Password File

As previously seen a passwd.txt an acceptable structure is:

user1:password1
user2:password2
userN:passwordN

A users.j2 template file can be created as follows:

{%- for user in mosquitto.users %}
{{ user.username }}:{{ user.password }}
{% endfor %}

where the mosquitto.users is an array of dictionaries:

mosquitto:
  users:
    - username: user1
      password: password1
    - username: user2
      password: password2

To test this logic we can create an Ansible Playbook called mosquitto_users.yml as follows:

- name: Mosquitto Users (plain-text) file generation
  hosts: localhost
  gather_facts: false
  vars:
    mosquitto:
      users:
        - username: testuser1
          password: password1
        - username: testuser2
          password: password2

  tasks:
    - name: generate the Mosquitto users file
      ansible.builtin.template:
        src: users.j2
        dest: users

Execute the playbook using:

$ tree
├── mosquitto_users.yml
└── users.j2
$ ansible-playbook mosquitto_users.yml

upon execution of the playbook a users file will be generated that will fill the values of the credentials accordingly.

Custom Jinja2 Filter for Hash Generation

The template file users.j2 currently fulfills the criteria of structure required for the Broker.
We now create a custom Jinja2 filter that will consume the plain-text passwords and generate the acceptable hashes for the Broker.

A Jinja2 filter in Ansible is rather a very simple python script that Ansible executes to generate the respective logic.
As previously mentioned, we know that passlib library provides a PBKDF2_SHA512_
implementation and upon inspecting the code base for eclipse mosquitto we can also determine how many rounds(iterations) are required and what the salt length should be.
The values are as follows:

Iterations are 101
salt length is 12 bytes (random bytes) Based on the passlib API this information can be passed into a function call rather easily.

Let's call the filter the same as mosquitto_passwd . Here is the implementation:

Create a folder called filter_plugins and create mosquitto_passwd.py under it
install the passlib package using pip install --user passlib
Add the following code into the mosquitto_passwd.py

from ansible.errors import AnsibleError

def mosquitto_passwd(passwd):
    try:
        import passlib.hash
    except Exception as e:
        raise AnsibleError('mosquitto_passlib custom filter requires the passlib pip package installed')

    SALT_SIZE = 12
    ITERATIONS = 101

    digest = passlib.hash.pbkdf2_sha512.using(salt_size=SALT_SIZE, rounds=ITERATIONS) \
                                        .hash(passwd) \
                                        .replace("pbkdf2-sha512", "7") \
                                        .replace(".", "+")

    return digest + "=="


class FilterModule(object):
    def filters(self):
        return {
            'mosquitto_passwd': mosquitto_passwd,
        }

Few things to note in the implementation:

Hash digest generated by passlib.hash.pbkdf2_sha512 is the of the following structure: $pbkdf2-sha256$6400$0ZrzXitFSGltTQnBWOsdAw$Y11AchqV4b0sUisdZd0Xr97KWoymNE0LNNrnEgY4H9M
Mosquitto Broker requires that the first part of the digest either be 7 for PBKDF2_SHA512 or 6 for the SHA-512. This is done by the .replace("pbkdf2_sha512", "7") in the code
passlib PBKDF2_SHA512 implementation has a shortened base64 format which omits padding and whitespaces, which will not be acceptable by the Broker. In order to make the digest compatible we replace the instances of . with + in the digest (via .replace(".", "+"))
we add the == characters at the end of the digest to make up for the shortened base64 encoding implementation when it comes to the overall length of the digest

The final step should is to add the custom mosquitto_passwd filter into our users.j2 file as follows:

{%- for user in mosquitto.users %}
{{ user.username }}:{{ user.password | mosquitto_passwd }}
{% endfor %}

the template during generation will pass the user.password to the mosquitto_passwd filter and provide the hashed value as opposed to the plain-text values in a users file.

Results

For the following structure:

$ tree .
.
├── filter_plugins
│   └── mosquitto_passwd.py
├── mosquitto_users.yml
└── users.j2

executing the playbook mosquitto_users.yml:

ansible-playbook mosquitto_users.yml

will generate a users file in the directory. Upon viewing the contents of the file:

cat users
testuser1:$7$101$9h4jJKT0HiMEgDAm$3ynyYne/t4KJyjDqOyCVzF0G0Oa71nu0K9iaa1NNfzCNc5c71iVnUVqV+zxDjGkaDy6gSAWAyk6KmqgTaE1IDQ==
testuser2:$7$101$HwNAKGVsjbE2JkRI$Nk+xnkjOTyllT7hD4N5kd9kGarMAQJMJVgWraa0VxmKZgyRyfUjG0+kUCPE1ZuLaGPrbeq0H80Pl6tHd+Qm2Lw==

(password hashes may vary, since salt lengths are randomly generated)

Verifying with the Broker

The best way to confirm that the hashes are acceptable is to generate a mosquitto.conf file with the following contents:

# MQTT Port Listener
listener    1883
protocol    mqtt

# Authentication
allow_anonymous     false
password_file       /mosquitto_users

and then using the following docker-compose.yml file:

services:
  mosquitto:
    image: docker.io/eclipse-mosquitto:2.0.15
    container_name: test_broker
    configs:
      - mosquitto_conf
      - mosquitto_users
    entrypoint: mosquitto -c /mosquitto_conf
    ports:
      - 1883:1883
configs:
  mosquitto_conf:
    file: ./mosquitto.conf
  mosquitto_users:
    file: ./users

upon performing:

docker compose up

You will be able to see the logs of the broker without any erroneous information:

test_broker  | 1692181531: mosquitto version 2.0.15 starting
test_broker  | 1692181531: Config loaded from /mosquitto_conf.
test_broker  | 1692181531: Opening ipv4 listen socket on port 1883.
test_broker  | 1692181531: Opening ipv6 listen socket on port 1883.
test_broker  | 1692181531: mosquitto version 2.0.15 running

Use an MQTT client and try to connect to the broker <IP_ADDRESS>:1883 with either of the credentials values mentioned in mosquitto.users and it should work.

Now you have a functioning configuration management tool for your eclipse Mosquitto Broker using a popular DevOps / Configuration Management Tool (Ansible)
which provides you a guarantee of authentication without having any Mosquitto related software installed beforehand on your IoT Devices / Servers / Industrial Hardware.

Potential Extensions

You can now generate Jinja2 based template files for the Access-Control Lists (ACLs) or other related configurations with having to maintain multiple versions of them rather let Ansible generate them for you.

Similar, logic has been developed for Komponist
where a user can define how many users and what ACL permissions each user has on topics by just maintaining a single credentials file for the Mosquitto Broker as well as other containers such as InfluxDB, node-RED etc.

Telegraf Deployment Strategies with Docker Compose

Shan Desai — Fri, 21 Jul 2023 07:53:45 +0000

InfluxData's Telegraf

Telegraf is widely used as a metric aggregation tool thanks to the diverse amount of plugins it provides which interface with a multitude of systems without having to write complex software logic. With the advent of the Low-Code / No-Code paradigm in system operations; Telegraf sees itself as a front-runner.

However, with Low-Code/No-Code tools it might get complicated to maintain a lot of additional yet necessary information like credentials to other subsystems. Although standard practice dictates the use of Environment Variables within Telegraf - with version 1.27 and beyond Secret Stores will come in handy when passing credentials into Telegraf plugins without having to pass environment variables explicitly, especially when using a containerization tool like Docker.

We will go through some strategies of deploying a Telegraf Docker Container using Docker Compose v2 tool in this post that may help intermediate to advanced users decide upon how to organize their stack configurations appropriately

Using Docker Secrets

Docker Compose v2 specifications provide a useful Secrets feature which may also be used for standalone Compose Application Stacks and not just in Docker Swarm mode. With Docker Secrets the environment variables that contain credentials for other subsystems are mounted into the Telegraf Container as files. These secret files are read through the Docker Secret Store plugin and passed to the respective plugins in a relatively safe manner. By using the Docker Secret Store Plugin, one can also avoid credentials that were previously visible via environment variables, to be now hidden behind runtime secret files within the container.
Standard Method with Environment Variables
As an example, it is possible to pass the credentials to a plugin via the environment variable placeholder in a telegraf configuration file where the credentials for a plugin exist in a .env file (e.g. MQTT input Plugin)

MQTT_USERNAME=telegraf
MQTT_PASSWORD=superSecurePasswordMQTT

A telegraf configuration file for the MQTT Input plugin can be as follows:

[[inputs.mqtt_consumer]]

  servers = [ "tcp://<broker_ip>:1883" ]
  topics = [ "test1/#", "test2/+/topic" ]
  username = "${MQTT_USERNAME}"
  password = "${MQTT_PASSWORD}"

A Docker Compose File for the example can be as follows:

services:
  telegraf:
    image: docker.io/telegraf:latest
   container_name: telegraf
   environment:
     - MQTT_USERNAME=${MQTT_USERNAME}
     - MQTT_PASSWORD=${MQTT_PASSWORD}
   volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro

When the container is brought up, the values of the environment variables are interpolated by telegraf in order to connect to the MQTT Broker. This is with an assumption that the .env file and the docker-compose.yml file are in the same directory. This works well however a simple inspection of the running container may yield the credential values via the environment variable by a command such as:

docker compose exec telegraf env

docker inspect -f “{{ .Config.Env }}” <telegraf_container_name / telegraf_container_id>

With Docker Secrets and Telegraf Docker Secret Store Plugin
With Docker Secrets we can change the Compose file to pick the environment variables up from the .env file and let Docker Compose mount these values as files under the /run/secrets directory in the telegraf container as follows:

services: 
  image: docker.io/telegraf:latest
  container_name: telegraf
  secrets:
    - source: mqtt_username
      mode: 0444
   - source: mqtt_password
     mode: 0444
  volumes: 
./telegraf.conf:/etc/telegraf/telegraf.conf:ro

secrets: 
  mqtt_username: 
    environment: MQTT_USERNAME
  mqtt_password: 
    environment: MQTT_PASSWORD

Docker Compose will mount the values of MQTT_USERNAME and MQTT_PASSWORD into the /run/secrets/mqtt_username and /run/secrets/mqtt_password file within the container at runtime. We also set the file permissions to world-readable (0444) so that all users within the container may be able to read the values

Similarly we adapt the telegraf configuration file with the Docker Secret Store plugin and the MQTT credentials as follows:

[[ inputs.mqtt_consumer ]]
  servers = [ “tcp://<broker_ip>:1883” ]
  topics = [ "test1/#", "test2/+/topic" ]
  username = “@{custom_secretstore:mqtt_username}”
  password = “@{custom_secretstore:mqtt_password}”

[[ secretstores.docker ]]
  id = “custom_secretstore”

Bringing the container up with docker compose up will let Telegraf connect to the MQTT Broker and subscribe to the required topics.

A benefit with this method is that the previously visible environment variables within the docker container are now safely mounted into the container at runtime and upon inspection via

docker compose exec telegraf env

docker inspect -f “{{ .Config.Env }}” <telegraf_container_name/ telegraf_container_id>

The environment variables won’t be visible and provide a bit more safe usage of our credentials.

Splitting Telegraf Configuration Files

There may be situations where a Telegraf Configuration file starts to grow in size and the number of plugin configurations would be better maintained if they were to be split into individual files. Telegraf is able to consume multiple configuration plugin files and is able to process them one-by-one to run the telegraf agent as if these files were merged together (it is not a simple merge however, there is some more effort that the source code has to undertake).

It is possible to mount a directory with split configuration files to the telegraf Compose service as follows:

telegraf/
| - inputs.<plugin_name#1>.conf
| - inputs.<plugin_name#2>.conf
| - outputs.<plugin_name#1>.conf
| - outputs.<plugin_name#2>.conf
| - processors.<plugin_name#1>.conf
| - processors.<plugin_name#2>.conf
| - secretstores.<plugin_name>.conf

The respective Compose file is as follows:

services:
  telegraf:
    image: telegraf:latest
    container_name: telegraf
    volumes:
      - ./telegraf:/etc/telegraf/telegraf.d/

We notify telegraf that the configuration is to be picked up from a directory i.e. /etc/telegraf/telegraf.d/ and not a standalone configuration file here.

NOTE: It is recommended to use the order parameter in processor plugins to let telegraf know in which order the data coming in from input plugins needs to be manipulated.

Of course, the split configuration files can be used together with Docker Secrets.

Docker Config for Telegraf Configuration File

There is also a possibility where a telegraf configuration file can be mounted via Docker Configs feature instead of volume mounts.
According to the Compose specification document for Docker Configs:

Configs allow services to adapt their behaviour without the need to rebuild a Docker image.

Mounting Telegraf Configurations files via Docker Config is very seldomly used in deployments with stand alone Compose applications, however they may be beneficial when working with custom Docker images for Telegraf.

A common Compose file for Telegraf

services:
  image: telegraf:latest
  container_name: telegraf
  Volumes:
./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro

Can be transformed to let Compose mount the telegraf configuration file via Docker Config as follows:

services:
  image: telegraf:latest
  container_name: telegraf
  command: --config /telegraf_conf
  configs:
    - telegraf_conf

configs:
  telegraf_conf:
    file: ./telegraf/telegraf.conf

By default, the configuration is mounted in the root of the container with the name of the docker config mapped to the file of the configuration file. When using such a configuration it is essential to mention within the command parameter where should telegraf load the configuration file from i.e. command: –-config /telegraf_conf

Inference

This guide should provide a baseline for how Engineers can plan to design their Telegraf Configuration file(s) with Docker Compose v2 as a deployment tool. This is not an exhaustive guide by any means, but it explores some new features in Telegraf (Docker Secret Store) and some less familiar Docker Compose v2 specifications that may not be often used in deployments.

Resources / Links

How Reproducibility + Documentation in Software goes deep: Practical Scenario

Shan Desai — Sun, 11 Jun 2023 18:58:53 +0000

Problem

I have been working on a personal project which I recently open-sourced called Komponist.
In a nutshell, the project is heavily reliant on some of the most widely used tools today in
the software landscape i.e., Ansible, Docker, Docker Compose.

Everything was smooth-sailing with a new feature planned when suddenly I started getting a
strange error when bringing my container stack up using:

docker compose --project-directory=deploy up -d

The error was rather mysterious and not much verbose. It screamed something like:

Error response from daemon: Could not find the file / in container <container_id>

This seemed rather odd, the same thing that worked a couple of days back now is rendered
buggy! As usual common searches didn't lead me anywhere. What was rather weird was the fact that
I never was mounting any root directory anywhere (/). The good thing about this error was
I had only two realms to look into, either Docker Compose v2 tool or the Docker Engine

Hackin' Around

Since this was a sudden bug that I faced with, I suspected initially something going wrong with
the Compose Files (docker-compose.yml) file which my tool generates. However, this felt rather
counter-intuitive because the tool itself also takes up the initiative to validate all my Compose Files.
Had there been any error, the validation logic would have failed and hence would not have generated me
a production-ready file in the first place.

Through some previous recollections, I was conscious about User ID within the container causing problems
especially when trying to mount secrets into Docker Containers. In essence, my tool mounts Secrets (credentials)
into the container via Docker Compose's specification which copies the values under the /run/secrets directory
within a container.

This feature, although great had some small caveats. As security best-practice most containers have a user within
them, which does NOT have enough privileges to actually mount the secrets into the /run/secrets directory.
The only exception being container with root user (which is generally not the best idea in the first place).

A common fix is to add user parameter to each service with the current Host's User ID, e.g., 1000 which
will run the container as the same user ID as that of the host's user. This was already part of the current system
in my project.

I started hacking around thinking this particular value might be the main culprit so I tried adding
various combinations like user: "1000:1000" or user: "<my_host-machine_name>" but it did not work.

A last dumpster dive into some GitHub Issues on the Moby Project which is essentially the Docker Engine
landed me onto Issue 34142 for Moby Project which is open since 2017 (yes you read that right!).

At wit's end, I thought if the user on my host has stopped working, I might as well give the user within the image
a shot i.e., if I have a Grafana container, upon running:

$ docker run -it grafana/grafana-oss:latest whoami
grafana

so updating my user parameter to user: "grafana" suddenly brought got everything working!

Temporary fix for the Project! Hurrah!

Reporting the Behaviour

I generally am very thorough in reporting bugs / issues to Open-Source Projects. Essentially I start
with mentioning my Host, all the software versions that are affected (here, Docker Engine / Docker Compose v2)
and provide a Minimum Example that can be reproduced by anyone. Since the fix was pertaining to Docker Compose v2's
user property I reported a bug on Github's docker/compose repo.

The Maintainers had a look into it, but then came the nightmare answer of:

Tried to reproduce it, I got something else !

Dread of "It works on my machine, can't reproduce it"

After a few back and forth with the maintainer and even trying out certain examples, it felt like I had messed up
my Docker Engine with some configuration. It wasn't the case, since my configuration file was the least bit
complicated and it was close to running an out-of-the-box Docker Engine.

It seemed like this was going to be one of those GitHub Issues that would be left open and gets lost in the
Oblivion of Open-Source Projects. Everything seemed hay-wired.

One Last Effort!

Whilst the back and forth between my Bug report and the Maintainer's non-reproducible errors started to reduce,
I reverted back to the an older Docker Engine Version to see if a sudden update started to cause this error on my end.

For Context, the Issue was about using Docker Engine v24.x with Docker Compose v2.18.1. I downgraded the
Docker Engine to the last v23.x.x and would you know it! Things started to work again.

I was able to pinpoint out that things were going wrong, since the leap of faith from Docker Engine v23 to v24.
I mentioned this difference in the Issue's Thread

So I was convinced that something is breaking between the Docker Engine versions but I wasn't sure whether there was
some discrepancy in Docker Compose v2 too.

This led me to a previous Compatibility Anaylyses that I had done about a year back called
Exploring the Marshland: Docker Compose Versions
where I was able to isolate different versions of the software to determine which versions were working with what
specific parameters. The tool I had used was Vagrant.

Isolate, Compare, Determine!

I took the initiative of creating two Isolated Vagrant Boxes with the following software:

Vagrant Box	Docker Engine Version	Docker Compose Version(s)
Box 1	`v23.0.6`	`v2.18.1` `v2.17.3` `v2.16.0`
Box 2	`v24.0.2`	`v2.18.1` `v2.17.3` `v2.16.0`

Documenting the Discrepancy!

I decided that I would rather add GIFs of the reproduction of the errors as opposed to adding a lot of
verbosity to an already complicated problem and create a repo that would reduce the time for the Bug reproduction.
This does come with an assumption that the Maintainer would also have Vagrant or something similar to test at their
end.

I was able to reproduce the same bugs in a rather isolated environment using Vagrant disproving my theory that I had
messed my Docker Engine up and perhaps the Maintainer should try the stuff out again at their end.

The Repo can be Found on GitHub

Inference

Upon providing an environment to reproduce this error as well as some GIF documentation, it turned out that this was
actually an error even the Maintainer can reproduce. This error is related to the Docker Engine v24 and not Docker Compose v2
and it is now being tracked with fixes already on the way.
See Moby/Moby Issue #45719.

This might look like a lot of work, which in hindsight might be, but it is worth being comfortable with tools where a decent
amount of reproducibility via Isolation can provide a strong foundation to look into software a bit more deeper!

Vagrant is meant to be Developer-Friendly tool which indeed helped me overcome the dreadful situation of

It worked on my machine, why isn't it working on yours?!

It also helped collaborate on a problem which stemmed not in Docker Compose v2 but went all the way down to Docker Engine
itself. The maintainer was able to produce the same error with Canonical's multipass which also
reached the same goal through a different route.

So in a nutshell, a few hours of going in the deep end of error reproducibility may feel like time wasted but it might
just solve a problem that is long-lasting (till something new pops up!)

Create a minimal OS using Docker Containers and Hashicorp Packer

Shan Desai — Sun, 18 Sep 2022 11:02:55 +0000

Why do this?

I was mightly impressed by Ivan Velichko's Blogpost on using Docker Containers
to create bootable Disk images. He even has a GitHub Repository that still gets a lot of
attention from developers who have tried and tested things out.

I am already using Hashicorp Packer at work and for personal projects and I wanted to test
this idea out by wrapping it a single Packer Template file. This reduces the level of maintaining
a lot of small scripts, Dockerfiles and configurations and the user can simply trigger a couple of
commands to get a minimalist OS at the end of the process.

Also just sheer curiousity from side. Why not try things out!

What else is there in the market?

Don't get me wrong there are some projects already out there doing things similarly, and a lot of products
leverage these tools to ship out custom, minimalist OS images for Edge Computing Devices / Cloud Platforms.

LF-Edge EVE project leverages Linuxkit to create custom OSs for Edge Devices which in turn leverages Containers as Lego Blocks

Albeit these projects do much more than just create a Lego Tower of Docker Containers to create a bootable image,
but if it peeks your curiosity you should check them out.

Presenting: PockerISO

PockerISO is just Packer + Docker to create bootable images. The difference here is that Packer does the
heavy-lifting of doing the following things:

pulling the base images
bringing them up
executing the respective shell scripts within the container
saving/exporting the current filesystem state into a tarball
creating an isolated container to execute the steps to create the bootable image without disturbing the host machine fs

Packer is the Maestro and Docker is the Orchestra that plays the symphony, and at the end the end-user cheers with
a nice minimalist OS image

How does PockerISO work?

Filesystem Generation

I strongly recommend reading Ivan Velichko's blogpost mentioned previously to get some greate understanding on what needs
to be done. He does some insanely cool visualizations too!

In a nutshell, containers DO NOT have a kernel (they leverage the kernel of the host machine), and they DO NOT have
system manager, containers always run with Process ID 1.

So in order to make an operating system, we will install these two things in the container base image to atleast create a
filesystem tarball which we can use to create an image.

Once the container installs the kernel, generates the respective initramfs for us and installs systemd via APTITUDE
package manager, we tell Packer to finish of this process by creating a tarball of the container's filesystem.

Bootable Image creation

Once the tarball is created, we extract it on the host for the simple reason that extracting the tarball within a docker
container generally leads to a lot of Failures when done through via Packer.

We then spin up a container of the same base image copy the necessary files and folders into it. The main thing to note here
is this container is brought up with privileged mode. Remember we will still need the host device to use Operating System
Loop devices, to mount filesystems and create the final base image.

Within this phase we do the following via a dedicated shell script:

Create an empty image file using dd of approximately 1GB
Make Partitions for the disk image
Create a filesystem on the image with ext4 format
Copy our filesystem with the kernel, systemd and initramfs
Install a bootloader and create a boot partition on the image
Mount / Unmount the dedicated image

The final result will be an .img as well as .qcow2 image that can be tinkered with using QEMU.

As a use you only need to execute a simple make command:

make ubuntu # or `make debian`

and at the end of the magical automation rainbow you will have a minimalist OS you can play with.

What base images does PockerISO offer?

At this point the following:

Ubuntu: 20.04, 22.04
Debian: bullseye, bookworm

Potential Updates

I am planning to add Alpine Images with it's own Packer Template
I wish to make this repository compatible also for linux/arm64 images since Docker lets you cross-compile images via BuildKit. The only caveat is that for devices like the Raspberry Pi, the bootloader logic might be a tad bit tedious to figure out.

If you would like to give some feedback, criticisms, suggestions on improving the project or this write-up leave me a message on
LinkedIn and I will happily revert back.

Customized Ubuntu Images using Packer + QEMU + Cloud-Init & UEFI bootloading

Shan Desai — Sun, 21 Aug 2022 15:49:00 +0000

Custom Ubuntu Images

NOTE: This post requires more than intermediate knowledge of the tools being used.

In current times, you might want to spin up a customized, well-known Distribution like Ubuntu, Debian, CentOS etc.
without having to write large Shell Scripts. This post sheds light on how to create images using tools like
Hashicorp's Packer and QEMU (Quick Emulator).

Cloud-Init

Cloud-Init is a standardized way to configure your Images without having to write shell scripts. It is a set of YAML
files that tells the image what needs to be done on the first-boot of the OS. We will use Cloud-Init to create the following:

Create 2 Users (admin and user1)
Set the Bootloader Sequence when trying to boot the Image using UEFI

Packer

Hashicorp Packer provides a nice wrapper / abstraction over the QEMU in order to boot the image and use it to set it up on first-boot.
Instead of writing really long commands in order to boot up the image using QEMU, Packer provided a nice Configuration Template in a more
readable fashion.

QEMU

QEMU is one of the most renowned emulator. We will use it to actually boot the ISO image, which Packer will download for us and use it
customize our Ubuntu Image. We will use the X86_64 architecture.

Ubuntu Live-Server

You can use either the Cloud Images or Live-Server from Ubuntu depending on your use. We will be using Ubuntu's Live-Server.

Implementation

NOTE: The configurations used here will work for Ubuntu 20.04 LTS (Focal Fossa) as well as Ubuntu 22.04 (Jammy Jellyfish)

Pre-Requisites

Make sure to install:

qemu (Optionally kvm)
packer

Cloud-Init: `user-data` File

The user-data file is where we will be add configuration that is needed for first boot. As a clarification, Ubuntu Live Server uses a tool
called Autoinstall / Subiquity Installer wherein Cloud-Init configuration is a subset.

As previously mentioned, we will use this file to setup:

Create Two Users: admin and user1
Setup the bootloader logic in order to quickly boot the ISO after the first-boot

#cloud-config
autoinstall:
  version: 1
  locale: en_US
  keyboard:
    layout: us
  ssh:
    install-server: true
    allow-pw: true
  packages:
    - qemu-guest-agent
  late-commands:
    - "sudo apt update && sudo apt install -y efibootmgr"
    - "sudo efibootmgr"
    - "sudo efibootmgr -o 0007,0001,0000,0002,0003,0004,0005,0006"
  user-data:
    preserve_hostname: false
    hostname: packerubuntu
    package_upgrade: true
    timezone: Europe/Berlin
    chpasswd:
      expire: true
      list:
        - user1:packerubuntu
    users:
      - name: admin
        passwd: $6$xyz$74AlwKA3Z5n2L6ujMzm/zQXHCluA4SRc2mBfO2/O5uUc2yM2n2tnbBMi/IVRLJuKwfjrLZjAT7agVfiK7arSy/
        groups: [adm, cdrom, dip, plugdev, lxd, sudo]
        lock-passwd: false
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
      - name: user1
        plain-txt-passwd: packerubuntu
        lock-passwd: false
        shell: /bin/bash

The above mentioned file does the following:

the autoinstall is Ubuntu's AutoInstall / Subiquity configuration section which will set:
- Keyboard locale to en_US and layout to US
- Install SSH Server and allow Password login (will be used by Packer)
- packages will install some necessary packages on first-boot. Here we install qemu-guest-agent to help out with login
- late-commands will be triggered at the end of the installation. Here we install the Bootloader Manager (efibootmgr). We also define the sequence that tells the Boot Manager how it should setup the boot sequence. This will tell the manager where the OS is and when should it be loaded
- user-data is the actual section where the Cloud-Init configuration takes place.
The cloud-init configuration above should do the following:
- change the hostname to packerubuntu
- set the timezone Europe/Berlin
- Force change the password for the user called user1 through chpasswd
- Describe which users need to be created:
  - admin should be create with the password packerubuntu (the encrypted password is created using openssl passwd -6 -salt xyz)
  - admin should be granted sudo access without requirements for password
  - user1 should be created with packerubuntu

Packer File

Packer configuration will set all the necessary values to download the ISO Image from Ubuntu Artifacts Repository, give the boot command
options when the ISO is first booted, tell Ubuntu that it will do an automatic installation rather than anticipating the user to intervene.

I am using the Hashicorp Language to define my Packer template file but one can also JSON to setup the configuration. The file is called
ubuntu.pkr.hcl.


variable "vm_template_name" {
  type    = string
  default = "ubuntu-22.04"
}

variable "ubuntu_iso_file" {
  type    = string
  default = "ubuntu-22.04.1-live-server-amd64.iso"
}

source "qemu" "custom_image" {

  # Boot Commands when Loading the ISO file with OVMF.fd file (Tianocore) / GrubV2
  boot_command = [
    "<spacebar><wait><spacebar><wait><spacebar><wait><spacebar><wait><spacebar><wait>",
    "e<wait>",
    "<down><down><down><end>",
    " autoinstall ds=nocloud-net\\;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
    "<f10>"
  ]
  boot_wait = "5s"

  http_directory = "http"
  iso_url   = "https://releases.ubuntu.com/22.04.1/${var.ubuntu_iso_file}"
  iso_checksum = "file://https://releases.ubuntu.com/22.04.1/SHA256SUMS"
  memory = 4096

  ssh_password = "packerubuntu"
  ssh_username = "admin"
  ssh_timeout = "20m"
  shutdown_command = "echo 'packerubuntu' | sudo -S shutdown -P now"

  headless = false # to see the process, In CI systems set to true
  accelerator = "kvm" # set to none if no kvm installed
  format = "qcow2"
  disk_size = "30G"
  cpus = 6

  qemuargs = [ # Depending on underlying machine the file may have different location
    ["-bios", "/usr/share/OVMF/OVMF_CODE.fd"]
  ] 
  vm_name = "${var.vm_template_name}"
}

build {
  sources = [ "source.qemu.custom_image" ]
  provisioner "shell" {
    inline = [ "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for Cloud-Init...'; sleep 1; done" ]
  }
}

The Template file tells Packer where to find and download the ISO file. The boot_command is quite important because it tells
Packer how to navigate through the initial Boot Loaders Interface in order to enter the Grub Settings where we mention:

  autoinstall ds=cloud-net\\;s=http://{{ .HTTPIP}}:{{ .HTTPPort }}/

This will tell ubuntu to install the Live server without manual intervention and obtain the cloud-init configuration files
from an HTTP File Server (setup by Packer itself).

NOTE: please check beforehand where the dedicated OVMF.fd file is located on your build machine. In Majaro Linux it is located
at /usr/share/OVMF/OVMF_CODE.fd

The headless = false is very useful because Packer will open a VNC Viewer window that will completely show all the necessary
Boot loader UI + Setups + Logs. If using this file in a CI Pipeline, please set the value to true

The build section of the template file will just check whether the Cloud-Init Steps have been finished or not.

Run the following command:

packer build -force ubuntu.pkr.hcl

This will do all the magic for you! At the end you will have qcow2 image which you can use to load it on your bare-metal machine
or use qemu commands to simply boot it up and test it.

Voilà ! You now have a custom Ubuntu Image which you can load on your devices, servers etc. and do not have to painstakingly configure
anything by hand!

Don't let this stop you here, Packer let's you provision your Images with all necessary software packages using your favorite tools like
Ansible, Chef, Puppet. This implies you can even fine tune your images further to make your images exactly the way you want it to be.

Repository

The code for this post can be found on GitHub.

Get in touch via LinkedIn, Email if you have queries, suggestions or criticism about this post!

Resources

Julien Brochet's Blog Post on Using Packer + Proxmox for Ubuntu 22.04

Dogukan Cagatay's QEMU VM Template Packer Repo
Pupeteers.net Blog on Ubuntu 20.04 qemu images with Packer

Exploring the Marshland of Docker Compose Versions with single quotes, special characters and environment variables

Shan Desai — Fri, 15 Jul 2022 16:06:41 +0000

Problem

Since Docker's Compose V2 General Availability announcement, a lot of things have changed which might lead to
Incompatibility in your currently running application stacks.

Compose V2: what changes?

instead of docker-compose the syntax will be docker compose
docker-compose (v1) was written in Python, docker compose (v2) is in Go
Compose V1 is official EOL (End Of Life) aka. Deprecated

There are lot of fine things in Compose V2 but this post isn't about it. This write-up takes you through the murky
marshland between Compose V1 and Compose V2 and how things might start you break when you decide to jump to the
cool V2 vers without doing some thorough checks in your stack application

Swiss Knife for the Exploration

Tools you will need:

Hashicorp Vagrant
Ansible
Different versions of Docker Compose (v1 as well as v2)

Vagrant

Instead of setting up a pain-staking Virtual Machine, we will leverage a bit simplified and inexpensive Vagrant Machine
with Ubuntu 20.04 (focal amd64) base image.

Within this image we will install the following:

Docker Engine 20.10.17
Docker Compose v1: v1.25.4 build 8d51620a and v1.29.2 build 5b3cea4c
Docker Compose v2: 2.6.0

Ansible

I have Ansible installed on my host machine

ansible --version
ansible [core 2.13.1]

Ansible will install all the spicy Compose versions into the Vagrant Machine for us

Why are we doing this?

If you have containers that require passwords at rest e.g. in .env environment files to be encrypted and with
special characters like the $ in them, we are going for ride which will make you want to bookmark this write-up.

It turns out, I had a system running Compose v1 (v1.25.4) and upon upgrading to Compose v2 (2.6.0) I wasn't able
to login to my Node-RED container which had authentication setup already.

Node-RED Container

Node-RED requires that the administrator password must be encrypted in bcrypt type scheme and the password looks
something like your dog walked over your keyboard.

$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.

This looks fairly okay, but the menace is the $ characters in them.

Let's dive deep shall we!

Code

`Vagrantfile`

I am using a Linux Machine with Virtual Box as a provider

# -*- mode: ruby -*-
# vi: set ft=ruby :


Vagrant.configure("2") do |config|

  # Use Ubuntu 20.04 as base
  config.vm.box = "ubuntu/focal64"

  config.vm.box_check_update =false

  # Synchronize all the docker-compose related file to `/vagrant` dir in VM
  config.vm.synced_folder ".", "/vagrant"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
  end

  # Setup the VM with all the different variants of Docker Compose in them
  config.vm.provision "ansible" do |a|
    a.verbose = "v"
    a.playbook = "./docker-compose-playbook.yml"
  end
end

Ansible Playbook: `docker-compose-playbook.yml`

We will let Ansible install all the required Docker Compose versions for us in the VM through the playbook.

Docker Compose v1.25.4 will be available in the VM as CLI docker-compose-125
Docker Compose v1.29.2 will be available in the VM as CLI docker-compose-129
Docker Compose v2.6.0 will be available in the VM as CLI docker compose

---
- hosts: all
  become: yes
  tasks:
    - name: install prerequisites
      apt:
        name:
          - apt-transport-https
          - ca-certificates
          - curl
          - gnupg-agent
          - software-properties-common
        update_cache: yes
    - name: add apt-key
      apt_key:
        url: https://download.docker.com/linux/ubuntu/gpg
    - name: add docker repo
      apt_repository:
        repo: deb https://download.docker.com/linux/ubuntu focal stable
    - name: install docker
      apt:
        name:
          - docker-ce
          - docker-ce-cli
          - containerd.io
          - docker-compose-plugin
        update_cache: yes
    - name: add userpermissions
      shell: "usermod -aG docker vagrant"
    - name: Install docker-compose v1.29.2 from official github repo
      get_url:
        url : https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
        dest: /usr/local/bin/docker-compose-129
        mode: 'u+x,g+x'
    - name: change compose v1.29.2 user permissions
      shell: "chmod +x /usr/local/bin/docker-compose-129"
    - name: Install docker-compose v1.25.4 from official github repo
      get_url:
        url: https://github.com/docker/compose/releases/download/1.25.4/docker-compose-Linux-x86_64
        dest: /usr/local/bin/docker-compose-125
        mode: 'u+x,g+x'
    - name: change compose v1.25.4 user permissions
      shell: "chmod +x /usr/local/bin/docker-compose-125"

Files for reproduction of incompatibility

We wont be spinning to much complex stuff so a simple compose file and the relevant .env.node-red
file with the encrypted password should suffice

`docker-compose.yml`

version: '3.7'
services:
  node-red:
    image: nodered/node-red:2.2.2
    container_name: test-nodered
    env_file:
      - .env.node-red

`.env.node-red`

# Encrypting plain-text password with value `password` with bcyrpt
NODERED_ADMIN_PASSWORD=$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.

Compatibility Checks

Let's bring the Vagrant Machine up using:

vagrant up

this will download the base ubuntu image if you previously don't have it on your host machine,
provision it with all the docker compose versions in it.

vagrant ssh

Once into the Machine head to the /vagrant directory, and just to be safe we check if all the
required compose versions are available or not.

$ cd /vagrant

$ docker-compose-125 --version
$ docker-compose-129 --version
$ docker compose version

Checks: Password not encloses in Single Quotes

As the .env.node-red file currently shows that my encrypted password is not enclosed in single quotes

Let's understand what Compose thinks of the value of the environment variable.

Compose v1.25.4 (without single quotes)

docker-compose-125 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'

NO Problem! My dollar characters are escaped by docker-compose. I can login with password password

Compose v1.29.2 (without single quotes)

docker-compose-129 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'

NO Problem! My dollar characters are escaped by docker-compose. I can login with password password

Compose v2 without single quotes)

docker compose config

produces

name: node-red-app
services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: a$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
    networks:
      default: null
networks:
  default:
    name: node-red-app_default

Oh NO! What happened here? Password looks eaten up. The YAML file looks different too!

Checks: Password enclosed in single quotes

Now let's enclose our encrypted password between single quotes, so our .env.node-red file
should look like:

NODERED_ADMIN_PASSWORD=$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.

Compose v1.25.4 (with single quotes)

docker-compose-125 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: '''$$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.'''
    image: nodered/node-red:2.2.2
version: '3.7'

Oh NO! Why are there more single quotes than expected. This is not going to let me login!

Compose v1.29.2 (with single quotes)

docker-compose-129 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'

NO problem! here this will work fine as the dollar characters are escaped safely by compose

Compose v2 (with single quotes)

docker compose config

produces

name: node-red-app
services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
    networks:
      default: null
networks:
  default:
    name: node-red-app_default

NO Problem! Phew ! This will work just fine !

Result: Compose Compatibility Matrix

Compose Version / Values	`1.25.4`	`1.29.2`	`2.6.0`
with Single Quotes	❌	✅	✅
without Single Quotes	✅	✅	❌

What did we learn today, kids?

No Single Quotes:

Docker Compose v1.25.4 was able to resolve special characters without having enclose them in single quotes
Docker Compose v1.29.2 will also work just fine for values not enclosed in single quotes
Docker Compose v2 will try go hay-wire (try to resolve the value after the $ sign) if you do not use things in single quotes

With Single Quotes:

Docker Compose v1.25.4 will literally take things enclosed in single quotes as part of your environment variable and will
break your Container's Login logic
Docker Compose v1.29.2 will work well
Docker Compose v2 will work well

In a nutshell

DON'T JUMP THE GUN, when moving from different V1 to V2 in Compose. Do some throrough research as well as experiments.

Since Compose V1 is now deprecated, when transitioning to Compose V2 check all your environment variable files and if they require
special characters to be either enclosed in single quotes or not. Estimate your downtimes and conduct isolated tests when required.

Rule of thumb from Compose V2 onwards: if it got some special char in them password, pack em between single quotes

but to each their own!

Github Gist

GitHub Gist available publicly

Multi-Image Docker Images: Using COPY with Images directly from registries

Shan Desai — Fri, 24 Jun 2022 20:10:18 +0000

Problem

I was on the lookout for a solution to a problem in Docker that needed to do the following:

Create a Docker Image for a bunch of compiled ASP.NET .dll files and try reducing the
container footprint

Sounds simple, right? But it turned out to be a bit more messy than I imagined. To top it off,
I have NEVER even programmed in .NET so I was feeling like a visually impaired person made to
navigate his way through a dark room with furniture scattered all over!

Thousand Leagues under the Container Sea!

Keeping the whole Maritime theme alive with Docker (and Kubernetes), I jumped into the sea of
Docker Hub with millions of containers and found out that Microsoft hosts all the .NET related
container images as .NET by Microsoft registry.

Gentle reminder that I have never even thought of .NET before in my life, so just trying to figure out
whether I need an one or more of the following images:

SDK
ASP.NET Core Runtime
.NET Runtime
.NET Runtime Dependencies
.NET Monitor Tool
.NET Samples

was a side-mission in its own right.

Side-Mission

A quick deep-dive into Microsoft's Docker .NET Docs made me think I only need the following docker image:
mcr.microsoft.com/dotnet/aspnet:6.0

So packing everything in a simple Dockerfile


FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal


WORKDIR /DotNetVoyage

COPY ./appsettings.json /DotNetVoyage/Server-Files/

# Copy every '.dll' file I was handed 
COPY ./* /DotNetVoyage/Server-Files/
CMD [ "dotnet", "Server-Files/Server.dll" ]

Building it locally and accessing the App initially on the dedicated ports seemed to work just fine and I thought
I was getting off work a bit early today. I jinxed myself.

Upon some API testing of the image I started seems some logs that were strange to my non .NETian brain.

The specified framework can be found at:
It was not possible to find any compatible framework version
The framework 'Microsoft.AspNetCore.App', version '5.0.0' (x64) was not found.
 - The following frameworks were found:
    6.0.6 at [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
You can resolve the problem by installing the specified framework and/or SDK.

I was head scratching as to how I can make my runtime which is 6.0.0 potentially have an SDK version that must
have 5.0.0 SDK on it. It got worse, I found out I needed the now defunct and not supported 2.1 version of the SDK
too!

Build everything from scratch?

Unaware if there was a way to introduce the different SDKs into the already slimmed container image, I had lost hope
and decided to build an image with maybe debian or ubuntu or alpine.

However, upon asking the Slack Community and rummaging through some StackExchange queries, I traced out a plan that was
pretty familiar to me: Using Multi-Stage Docker Builds.

Charting towards a Destination

I use Multi-Stage Docker builds almost everyday for work and so I decided to design my container image as follows:

Stage 1: pull SDK 2.1
  1.1: find out which directory do the SDK Files persist

Stage 2: pull SDK 5.1
  2.1: find out which directory do the SDK Files persist

Stage Prod: Pull the 6.0.0 Runtime
  Prod.1: copy all the SDK Files (from Stage 1 and Stage 2) to the same directory here in Prod Stage
  Prod.2: copy all the DLL files from host and run the dedicated DLL

Where art thou SDKs?

A quick pull:

docker pull mcr.microsoft.com/dotnet/sdk:2.1

and check within the container:

docker run -it --rm --name=sdk21-check mcr.microsoft.com/dotnet/sdk:2.1 dotnet --list-sdks

resulted in:

2.1.818 [/usr/share/dotnet/sdk]

Gotcha! the directory for the SDKs is /usr/share/dotnet/

Multi-Image Image

what I mean by this is whether Docker is smart enough to directly pull an image either using COPY or ADD
instructions without me having to create Stage 1 and Stage 2 using syntax such as:

FROM mcr.microsoft.com/dotnet/sdk:2.1 as base21

FROM mcr.microsoft.com/dotnet/sdk:5.0 as base50

It turns out that COPY is extremely flexible! with the COPY --from instruction in the Dockerfile
is completely capable of pull the image from a registry and the dedicated directories can be copied easily!

Solution

Final Checks for Dockerfile

FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal

# Add those missing SDKs here
## Syntax: COPY --from=<registry/image:version> image_directory ProdStage__dest_directory
COPY --from=mcr.microsoft.com/dotnet/sdk:2.1 /usr/share/dotnet /usr/share/dotnet/
COPY --from=mcr.microsoft.com/dotnet/sdk:5.1 /usr/share/dotnet /usr/share/dotnet

# List all the SDKs / Runtimes available in the container
CMD ["dotnet", "--list-sdks", "dotnet", "--list-runtimes"]

build the image and run it and the output will be similar to:

Microsoft.AspNetCore.All 2.1.30 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.30 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.30 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Voilá ! Runtimes v6.0, v5.0, v2.1 all in the same container !!

Final Dockerfile

FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal

# Add those missing SDKs here
## Syntax: COPY --from=<registry/image:version> image_directory ProdStage__dest_directory
COPY --from=mcr.microsoft.com/dotnet/sdk:2.1 /usr/share/dotnet /usr/share/dotnet/
COPY --from=mcr.microsoft.com/dotnet/sdk:5.1 /usr/share/dotnet /usr/share/dotnet

WORKDIR /DotNetVoyage

COPY ./appsettings.json /DotNetVoyage/Server-Files/

# Copy every '.dll' file I was handed 
COPY ./* /DotNetVoyage/Server-Files/
CMD [ "dotnet", "Server-Files/Server.dll" ]

Dedicated StackOverflow Query I created and answered

Just another day on the decks with Docker !!

Get in touch if you feel like dropping on suggestions, feedbacks or criticism!

Use htpasswd to secure your node-RED Container

Shan Desai — Thu, 09 Jun 2022 21:10:01 +0000

Securing Node-RED

node-RED containers can be made secure when using Admin Authorization features
using its settings.js file and mounting it to the container. Sounds quite simple,
however unlike other containers that let your configure their environments via environment
variables whereby the stored credentials are in plaintext, node-RED passwords typically are
stored in encrypted format i.e., bcrypt.

Automation: Encrypted Passwords

There is a possibility of obtaining the hashed (encrypted) password using an ephemeral container
using:

docker run -it --rm --entrypoint /bin/bash nodered/node-red:2.2.2-12-minima

One in the container use:

bash-5.0$ node-red-admin hash-pw

which will prompt you to enter the password and the resultant hash password will be available for you
to use.

This is however, a bit more of manual work and in case where you wish to perform more easy, automated
setups we might need to find a different way.

Automation Script via `htpasswd`

Apache provide a great CLI tool called htpasswd that can encrypt our plaintext passwords
into the required bcrypt hashed format which the node-RED UI and Admin Auth backend will decrypt.

For most of the distributions the package name is apache2-utils so for Ubuntu/Debian you can install
htpasswd using:

sudo apt-get install apache2-utils

Jumping right into it!

You can go through the man pages of htpasswd to find how it might suit for your needs. I am cutting
to the chase as to how I used it.

In order to generate my hashed password that will be compatible with node-RED's decryption backend, I had
to find the total Computation Cost (rounds) needed, which I found out the value was 8 based on some
code diving using node-red-admin repository.

htpasswd -nb -B -C 8 admin testpassword

will produce the following result:

admin:$2y$08$JTOJDi/whZxz8qwsdwCkfedhVOhh00QiKHICdHLImGS8XM2v.fng2

Creating an Environment Variable Template file for node-RED Docker Container

we will make the automation a bit more easier for us, whereby we can use envsubst commandline to fill
and environment variable called NODERED_AUTH_ADMIN in a file called node-red.env. But in order leverage
the power of envsubst we will create a template file called node-red.tpl.env which has the following
content:

cat node-red.tpl.env
NODERED_AUTH_ADMIN='${NODERED_AUTH_ADMIN}'

We create simple bash script which will do the following:

Accept a plaintext password as an argument to the script
use this plaintext password and hash it with htpasswd
export the variable which has the result of htpasswd
use the exported variable to replace the node-red.tpl.env file to node-red.env with the value
unset variable

The script is as follows:

cat nodeREDPwgen.sh
#!/bin/env bash

function nodeREDPW {
    # 1st argument to this script is plaintext password
    local NODERED_AUTH_ADMIN=$(htpasswd -nb -B -C 8 admin "$1")
    export NODERED_AUTH_ADMIN
    touch node-red.env
    envsubst '${NODERED_AUTH_ADMIN}' < node-red.tpl.env > node-red.env
    unset NODERED_AUTH_ADMIN 
}

nodeREDPW $1

Example Usage:

chmod +x nodeREDPwgen.sh

./nodeREDPWgen.sh testpasswd

will produce a node-red.env file after successful execution with the following content:

NODERED_AUTH_ADMIN='admin:$2y$08$/8NVmF2gJ3JDrmv0tX2ud.P7S9gqvB0ds2cRByhnZcRdZOWhTnPN.'

Creating a `settings.js` file for Node-RED Container

copy the settings.js file from the node-red repository and adapt the adminAuth object
as follows:


  adminAuth: {
    type: "credentials",
    users: [{
      username: process.env.NODERED_AUTH_ADMIN.slice(0, process.env.NODERED_AUTH_ADMIN.indexOf(':')),
      password: process.env.NODERED_AUTH_ADMIN.slice(process.env.NODERED_AUTH_ADMIN.indexOf(':')+1)
    }]
  },
}

Creating a `docker-compose.yml` file

services:
  node-red:
    image: nodered/node-red:2.2.2-12-minimal
    container_name: secure_nodered
    env_file:
      - ./node-red.env
    ports:
      - "1880:1880"
    volumes:
      - ./settings.js:/data/settings.js

bring the container up:

docker compose up -d

The UI is available at http://localhost:1880

Problem

If you try to login in by typing testpassword in the Admin UI, you will get Login Failed! But Why?

Investigations

It turns out that the password generated by htpasswd produces a variant of bcrypt $2y where as,
if we were to generate the password using node-red-admin hash-pw and added testpassword it would
generate a completely different looking password with the variant $2b.

Example: $2b$08$sfPlvPNDVVz/duAbAC2bbuNEVuvrwmU01x7plOAQIem6H187Xxq9G

Solution

It turns out that the $2y variant of bcrypt has compatibiltity and we could simply try and replace
$2y to $2b in our settings.js as follows:

module.exports = {
  // .. removed for brewity
  adminAuth: {
    type: "credentials",
    users: [{
      username: process.env.NODERED_AUTH_ADMIN.slice(0, process.env.NODERED_AUTH_ADMIN.indexOf(':')),
      password: process.env.NODERED_AUTH_ADMIN.slice(process.env.NODERED_AUTH_ADMIN.indexOf(':')+1).replace("$2y$", "$2b$"),
      permissions: "*"
    }]
  },
  // removed for brewity
}

notice the .replace("$2y", "$2b") at the end of password key.

Let's spin the container up again using docker compose up -d and try to login with testpassword et Voila!!

We are able to login into Node-RED!

I found that Nick O'Leary (founder of Node-RED and the great guy that he is..) already did some research on this,
I just had to bring this all together.

If you have some suggestions, critiques than please get in touch with me and I would be happy to help.

Resources

GitHub Gist of the files for the impatient ones!

Set the hostname of a Docker Container same as that of your host machine

Shan Desai — Wed, 08 Jun 2022 19:02:30 +0000

Setting Container Hostname to be same as your Host Machine

Sounds trivial, but it turns out you need to do some tweaks in order to set the hostname
to be the same as that of Host Machine.

Scenario

I had a requirement at work, where I needed to set the hostname of a specific container
to be that of the Machine it was supposed to run on.

The Linux Distribution under consideration was Ubuntu 20.04 LTS.

Ubuntu's mysterious `HOST` and `HOSTNAME` environment variables

If you are on an Ubuntu machine right now, try echo $HOS and press the TAB key to let the bash completion fill out. High chances that you will see HOST and HOSTNAME
as available Environment Variables already available to your bash shell's session.

Fairly Simple, you could simply use either one of them in your Compose file as an environment variable to the hostname key and should work out of the box! Not quite!

Investigation via a simple Example

Take the following docker-compose.yml file

services:
  test:
    image: alpine:latest
    container_name: hostname-tester
    hostname: alpine-${HOST}
    command:
      - hostname

The following test service should be able to retrieve the HOST variable value and set it as the hostname of the alpine container. The command should be able to print something like alpine-my-ubuntu as an example.

Let's see what happens when we run docker compose up

WARN[0000] The "HOST" variable is not set. Defaulting to a blank string. 
[+] Running 1/1
 ⠿ Container hostname-tester  Recreated                                    0.1s
Attaching to hostname-tester
hostname-tester  | alpine-
hostname-tester exited with code 0

As you see the HOST variable although available to the shell is not available within the
Compose file.

Most of the shell environment variables are visible via using printenv on the host machine, so a quick search for HOST or HOSTNAME reveals that these specific environment variables are not in the env of the shell

printenv | grep -i "host"

Solution

The most standard way to make a variable available for a shell session is by exporting it.

Let's export HOST using:

export HOST

Let's try bringing the container back up again and see if it picks up the variable values

❯ export HOST
❯ docker compose up
[+] Running 1/0
 ⠿ Container hostname-tester  Recreated                                    0.0s
Attaching to hostname-tester
hostname-tester  | alpine-shan-pc
hostname-tester exited with code 0

for my machine (Manjaro Linux (Rolling)) I could now set the hostname of the container to be the same as that of the Host Machine.

Sure enough searching through printenv again you will find the export HOST variable.

If you have on-prem servers or cloud images where you might need such a configuration a solution would be to add export HOST to your user's ~/.bashrc or ~/.zshrc (if using ZSH)
and the value will be available when bringing the corresponding Compose Stack up.

Hope this information helps people trying to find a similar solutions when it comes to Docker and Docker Compose based environments.

No Setup Development: Productivity Experience with Docker

Shan Desai — Sun, 08 May 2022 09:13:47 +0000

Why I stopped worrying about setting up environments!

If Stanley Kubrick were a Software Engineer, he would have named this post

Dr. Nosetup: How I Stopped Worrying About Setting Up and Love the Development

(I'll see myself out with that pun!)

I tried contributing to an open-source project without actually setting up the
complete programming language tools, and it felt like worth documenting.

Problem: So much to download, and setup before getting to work

I tried sending a feature to the node-red GitHub Repository with a new TOML configuration node.

However, I didn't want to taint (pardon me for using the word) my personal laptop by installing
node.js and npm.

One particular reason being, is that I have less time now to continue with Web Development stuff,
and node.js isn't my preferred language anyways. I want my host laptop to be as minimal as possible.

But I wanted to send the feature patch upstream because I was in the zone.

Solution: Docker encapsulated Environment

Since I have been heavily using docker for a while now, I asked myself

What do I need to send a patch upstream?

A: Only relevant files

Does docker provide me a node.js environment?

A: Yes, certainly it does. Not just node.js but for all possible programming languages

How do I avoid doing manual copy-paste labour for files between the container and my laptop?

A: Volume-Mounts. Any changes within the containe get reflected back to the host laptop and vice-versa

Setting it Up!

All I really needed was docker on my machine and we are ready to go!

Steps:

clone the repository to dedicated directory on my host laptop
Visit Docker Hub and find the node-js Image repository
Find the Long-Term-Support (LTS) version image tag. In my case it was 16.15.0

So we almost have everything we want!

Caveats

Remember that Docker Containers are their own ephemeral worlds all together.

If the containers are designed to with root users, your files may change ownership, or might have
different owners. You could check this using ls -la in your directory.

I really want to avoid such scenarios, such ownership issues might affect your filesystem as well as
the upstream code. But no issues, docker CLI provides way to control the user and group settings
before bringing the container up.

It is also worth mentioning that, the container environments also produce files that should be not be
reflected in your commits upstream. In the case of node-red the package-lock.json is a file created
within the container that will be mapped to the host machine.

It might be wise to keep such files into .gitignore as well as .dockerignore files within the development
repository to avoid accidently committing them upstream or bringing them within the container.

Docker CLI

$ # assuming your are in the development repository
$ docker run -it --name=node-red-TOML \
     -u $(id -u):$(id -g) \
     -v $(pwd):/usr/src/app \
     -p 1880:1880 \
     node:16.15.0 \
     /bin/bash

The -u parameter maps your current user id and group to the container, avoiding any root ownership
conflicts.

The -v parameter is the volume mount that will map the codebase to the /usr/src/app directory in the
container.

There you have it ! A node-js environment without having to download and setup the tool on the host!

You can now code everything with ease with your Editor of your choice with the container running.

Any changes, either on the host or within the container will be reflected to your editor.

Just make sure to run the execution commands in the container.

Benefits

This worked out well for me! I was able to get the codebase up and running in no time, without having to
worry about incompatibility issues.

Changes made in my editor (new files, refactored files) are available in the container to use and execute.

Running the commands within the container makes it easier to know what happens and all of this is ephemeral
so I don't have to do a lot of cleanup afterwards.

Just remove the container and commit the code!

On a side-note, the feature patch upstream wasn't required by the core team :(, but I could use the
same development environment pattern to create a node-red-contrib node. So nothing does to waste!

Hope this helps, get in touch if you would like to provide some suggestions, criticisms!

DEV Community: Shan Desai

Deep Dive with Ansible: Patching an Ansible Collection

Problem

Local Setup for Ansible Collections

Overriding the Collections Path for Development

Setting Up a Local Test Base

Result

Forward Compatibility for Mosquitto MQTT Broker with Docker Compose v2

Planning

Observations

Current Docker Compose File

Caveats

Solution

GitHub Gist

Generating Mosquitto MQTT Broker Credentials with Ansible

Challenge

Mosquitto Broker's Credentials Generation

mosquitto_passwd CLI native

mosquitto_passwd CLI via Docker Container

Independent Credential Generation for Broker

Templates for Password File

Custom Jinja2 Filter for Hash Generation

Results

Verifying with the Broker

Potential Extensions

Telegraf Deployment Strategies with Docker Compose

InfluxData's Telegraf

Using Docker Secrets

Splitting Telegraf Configuration Files

Docker Config for Telegraf Configuration File

Inference

How Reproducibility + Documentation in Software goes deep: Practical Scenario

Problem

Hackin' Around

Reporting the Behaviour

Dread of "It works on my machine, can't reproduce it"

One Last Effort!

Isolate, Compare, Determine!

Documenting the Discrepancy!

Inference

Create a minimal OS using Docker Containers and Hashicorp Packer

Why do this?

What else is there in the market?

Presenting: PockerISO

How does PockerISO work?

Filesystem Generation

Bootable Image creation

What base images does PockerISO offer?

Potential Updates

Customized Ubuntu Images using Packer + QEMU + Cloud-Init & UEFI bootloading

Custom Ubuntu Images

Cloud-Init

Packer

QEMU

Ubuntu Live-Server

Implementation

Pre-Requisites

Cloud-Init: user-data File

Packer File

Repository

Resources

Exploring the Marshland of Docker Compose Versions with single quotes, special characters and environment variables

Problem

Compose V2: what changes?

Swiss Knife for the Exploration

Vagrant

Ansible

Why are we doing this?

Node-RED Container

Code

Vagrantfile

Ansible Playbook: docker-compose-playbook.yml

Files for reproduction of incompatibility

docker-compose.yml

.env.node-red

Compatibility Checks

Checks: Password not encloses in Single Quotes

Compose v1.25.4 (without single quotes)

Compose v1.29.2 (without single quotes)

Compose v2 without single quotes)

`mosquitto_passwd` CLI native

`mosquitto_passwd` CLI via Docker Container

Cloud-Init: `user-data` File

`Vagrantfile`

Ansible Playbook: `docker-compose-playbook.yml`

`docker-compose.yml`

`.env.node-red`

Automation Script via `htpasswd`

Creating a `settings.js` file for Node-RED Container

Creating a `docker-compose.yml` file

Ubuntu's mysterious `HOST` and `HOSTNAME` environment variables