Generating Mosquitto MQTT Broker Credentials with Ansible

Challenge

This is a development note from a personal, open-source project called Komponist.
The aim of the project is to provide IoT / IIoT Developers a platform that can be used locally as well as in production using commonly used containers
in the Edge computing space such as:

• Time-series Databases (InfluxDBv1/v2, QuestDB, TimescaleDB)
• Data processing tools (Node-RED, Telegraf)
• Dashboard visualization / monitoring (Grafana)
• Message Broker (Eclipse-Mosquitto) There are standard tools / software used quite ubiquitously with containerization technology like Docker / Compose v2.

One major challenge that developers face is to maintain a lot of files that are needed to bring these tools / software up on initialization.
These files mostly contain user credentials, initialization scripts and configurations needed to start these containers correctly and according to user specifications.
Additionally, each container of these tools / software are very distinct and may rely on various setups, markup languages (YAML, TOML, INI) etc.
and there is rarely any coherence available between one tool against the other.

Komponist solves this issue by add a logical configuration generation tool over the tools and relies only on two core files
that describes the complete stack along side the respective credentials and configurations.
It uses the easy to use ansible configuration management tool to generate the respective docker-compose files, credentials file, YAML and TOML files for the software
and prepares the stack within a couple of minutes either locally or within a group of IoT edge devices.

This writeup focuses more on the Mosquitto MQTT Broker. The writeup describes how to leverage Ansible and Jinja2 powered Templating engine to generate Mosquitto Broker's
credentials.

NOTE: This writeup requires some basic to intermediate knowledge of Ansible as a tool.

Mosquitto Broker's Credentials Generation

mosquitto_passwd CLI native

According to the mosquitto broker's authentication methods documentation
it is possible to generate password files when one either has the mosquitto_passwd CLI installed locally / remotely on the IoT devices.
For example, the following plain-text passwd.txt file:

user1:superSecretPass1
user2:superSecretPass2

will encrypt the plain-text passwords into hashes that are acceptable by the mosquitto broker via the mosquitto-passwd CLI using:

mosquitto_passwd -U /path/to/passwd.txt

However, it relies on the existence of mosquitto_passwd tool to be available locally.

mosquitto_passwd CLI via Docker Container

The same tool can be used via a Docker container where the plain-text password passwd.txt file can be encrypted via spinning a docker container
locally and mounting the plain-text file to a dedicated path into it. An example with the same passwd.txt can be:

docker run --rm -v /path/to/passwd.txt:/mosquitto/config/users \
eclipse-mosquitto:2.0.15 \
mosquitto_passwd -U /mosquitto/config/users

Since the file is mounted into the container, any changes within the container will be reflected on the passwd.txt on the host i.e., the plain-text passwords will be encrypted accordingly.

Both these methods, require dependency on either the mosquitto_passwd CLI or the eclipse-mosquitto Docker Image to exist locally beforehand.

Independent Credential Generation for Broker

Upon observing the code base for eclipse/mosquitto it turns out the password hash generation relies on two methods:

1. SHA512 + additional salting and base64 encryption logic
2. SHA512 type PBKDF2 encryption logic

The details of the these two methods are beyond the scope of the write up but the implementations for them are quite common and can be found in every programming language.

Since Komponist relies on Ansible and Jinja2 templating engine which are directly reliant on Python3.x as a programming language, the implementation is rather trivial thanks to passlib python package.
passlib provides the implementation of PBKDF2_SHA512 logic with all the necessary encoding logic needed for the hashing of plain-text passwords that are acceptable by the Broker.

Templates for Password File

As previously seen a passwd.txt an acceptable structure is:

user1:password1
user2:password2
userN:passwordN

A users.j2 template file can be created as follows:

{%- for user in mosquitto.users %}
{{ user.username }}:{{ user.password }}
{% endfor %}

where the mosquitto.users is an array of dictionaries:

mosquitto:
  users:
    - username: user1
      password: password1
    - username: user2
      password: password2

To test this logic we can create an Ansible Playbook called mosquitto_users.yml as follows:

- name: Mosquitto Users (plain-text) file generation
  hosts: localhost
  gather_facts: false
  vars:
    mosquitto:
      users:
        - username: testuser1
          password: password1
        - username: testuser2
          password: password2

  tasks:
    - name: generate the Mosquitto users file
      ansible.builtin.template:
        src: users.j2
        dest: users

Execute the playbook using:

$ tree
├── mosquitto_users.yml
└── users.j2
$ ansible-playbook mosquitto_users.yml

upon execution of the playbook a users file will be generated that will fill the values of the credentials accordingly.

Custom Jinja2 Filter for Hash Generation

The template file users.j2 currently fulfills the criteria of structure required for the Broker.
We now create a custom Jinja2 filter that will consume the plain-text passwords and generate the acceptable hashes for the Broker.

A Jinja2 filter in Ansible is rather a very simple python script that Ansible executes to generate the respective logic.
As previously mentioned, we know that passlib library provides a PBKDF2_SHA512_
implementation and upon inspecting the code base for eclipse mosquitto we can also determine how many rounds(iterations) are required and what the salt length should be.
The values are as follows:

1. Iterations are 101
2. salt length is 12 bytes (random bytes) Based on the passlib API this information can be passed into a function call rather easily.

Let's call the filter the same as mosquitto_passwd . Here is the implementation:

1. Create a folder called filter_plugins and create mosquitto_passwd.py under it
2. install the passlib package using pip install --user passlib
3. Add the following code into the mosquitto_passwd.py

from ansible.errors import AnsibleError

def mosquitto_passwd(passwd):
    try:
        import passlib.hash
    except Exception as e:
        raise AnsibleError('mosquitto_passlib custom filter requires the passlib pip package installed')

    SALT_SIZE = 12
    ITERATIONS = 101

    digest = passlib.hash.pbkdf2_sha512.using(salt_size=SALT_SIZE, rounds=ITERATIONS) \
                                        .hash(passwd) \
                                        .replace("pbkdf2-sha512", "7") \
                                        .replace(".", "+")

    return digest + "=="


class FilterModule(object):
    def filters(self):
        return {
            'mosquitto_passwd': mosquitto_passwd,
        }   

Few things to note in the implementation:

• Hash digest generated by passlib.hash.pbkdf2_sha512 is the of the following structure: $pbkdf2-sha256$6400$0ZrzXitFSGltTQnBWOsdAw$Y11AchqV4b0sUisdZd0Xr97KWoymNE0LNNrnEgY4H9M
• Mosquitto Broker requires that the first part of the digest either be 7 for PBKDF2_SHA512 or 6 for the SHA-512. This is done by the .replace("pbkdf2_sha512", "7") in the code
• passlib PBKDF2_SHA512 implementation has a shortened base64 format which omits padding and whitespaces, which will not be acceptable by the Broker. In order to make the digest compatible we replace the instances of . with + in the digest (via .replace(".", "+"))
• we add the == characters at the end of the digest to make up for the shortened base64 encoding implementation when it comes to the overall length of the digest

The final step should is to add the custom mosquitto_passwd filter into our users.j2 file as follows:

{%- for user in mosquitto.users %}
{{ user.username }}:{{ user.password | mosquitto_passwd }}
{% endfor %}

the template during generation will pass the user.password to the mosquitto_passwd filter and provide the hashed value as opposed to the plain-text values in a users file.

Results

For the following structure:

$ tree .
.
├── filter_plugins
│   └── mosquitto_passwd.py
├── mosquitto_users.yml
└── users.j2

executing the playbook mosquitto_users.yml:

ansible-playbook mosquitto_users.yml

will generate a users file in the directory. Upon viewing the contents of the file:

cat users
testuser1:$7$101$9h4jJKT0HiMEgDAm$3ynyYne/t4KJyjDqOyCVzF0G0Oa71nu0K9iaa1NNfzCNc5c71iVnUVqV+zxDjGkaDy6gSAWAyk6KmqgTaE1IDQ==
testuser2:$7$101$HwNAKGVsjbE2JkRI$Nk+xnkjOTyllT7hD4N5kd9kGarMAQJMJVgWraa0VxmKZgyRyfUjG0+kUCPE1ZuLaGPrbeq0H80Pl6tHd+Qm2Lw==

(password hashes may vary, since salt lengths are randomly generated)

Verifying with the Broker

The best way to confirm that the hashes are acceptable is to generate a mosquitto.conf file with the following contents:

# MQTT Port Listener
listener    1883
protocol    mqtt

# Authentication
allow_anonymous     false
password_file       /mosquitto_users

and then using the following docker-compose.yml file:

services:
  mosquitto:
    image: docker.io/eclipse-mosquitto:2.0.15
    container_name: test_broker
    configs:
      - mosquitto_conf
      - mosquitto_users
    entrypoint: mosquitto -c /mosquitto_conf
    ports:
      - 1883:1883
configs:
  mosquitto_conf:
    file: ./mosquitto.conf
  mosquitto_users:
    file: ./users

upon performing:

docker compose up

You will be able to see the logs of the broker without any erroneous information:

test_broker  | 1692181531: mosquitto version 2.0.15 starting
test_broker  | 1692181531: Config loaded from /mosquitto_conf.
test_broker  | 1692181531: Opening ipv4 listen socket on port 1883.
test_broker  | 1692181531: Opening ipv6 listen socket on port 1883.
test_broker  | 1692181531: mosquitto version 2.0.15 running

Use an MQTT client and try to connect to the broker <IP_ADDRESS>:1883 with either of the credentials values mentioned in mosquitto.users and it should work.

Now you have a functioning configuration management tool for your eclipse Mosquitto Broker using a popular DevOps / Configuration Management Tool (Ansible)
which provides you a guarantee of authentication without having any Mosquitto related software installed beforehand on your IoT Devices / Servers / Industrial Hardware.

Potential Extensions

You can now generate Jinja2 based template files for the Access-Control Lists (ACLs) or other related configurations with having to maintain multiple versions of them rather let Ansible generate them for you.

Similar, logic has been developed for Komponist
where a user can define how many users and what ACL permissions each user has on topics by just maintaining a single credentials file for the Mosquitto Broker as well as other containers such as InfluxDB, node-RED etc.