DEV Community: Henk van den Brink

The circle of life: Embracing technical debt

Henk van den Brink — Tue, 08 Apr 2025 14:22:59 +0000

Embracing technical debt: A journey through life

It is good to have technical debt, when managed wisely.

Consider how our lives evolve, accumulating “debt” much like the mess we allow to pile up in our homes. At every stage, we make choices that balance the need for maintenance and support against our available resources.

Life’s phases: a living analogy

Imagine your life as a series of evolving chapters:

Student days

When you’re living in a dorm or a shared student flat, keeping your space immaculate isn’t a top priority. The “mess” or technical debt is allowed to accumulate because there’s minimal pressure to change.

Young adulthood & Relationships

Entering a relationship changes everything. Suddenly, clearing the clutter becomes a sign of care and commitment. You start to pay attention to the little details because someone else is now part of your life.

Family & Parenthood

As kids arrive, maintaining a pristine home becomes a luxury. With so many pressing priorities, the technical debt builds up even faster. Every spare moment isn’t dedicated to cleaning, but that’s okay; it’s part of life.

Rebalancing in later years

As your children grow and you reclaim some time, you begin cleaning up again. However, your priorities now include other passions and pursuits. Not every room needs daily attention, just enough to keep things in balance.

Retirement

In your later years, tackling household chores becomes more challenging. At this point, support systems like professional cleaners or caregivers come into play. The cost of maintenance increases, and you lean on external help to keep the balance.

Getting the right support

At each phase, you choose your level of help based on three key factors:

Necessity: How critical is the task?
Budget: What can you afford?
Lifestyle: What suits your way of being?

From parental interventions during your student years to hiring full-time support as you age, you’re always balancing effort (and costs) with the quality of life.

Striking a balance

Life is a series of decisions, whether you’re weighing the cost of a new gadget or deciding whether to move houses. These choices are part of every aspect of your existence. Some people opt to outsource many tasks by hiring help, while others shoulder responsibilities on their own, sometimes at the cost of increased stress. Ultimately, balance is the secret ingredient to maintaining both your life and your technical debt.

Parallels in software development

The lifecycle of a software product mirrors our lives chapters:

Product launch

Like starting out as a student, a new product begins with minimal structure, technical debt might be low, or even accumulating unnoticed.

Growth phase

As new features are added and demand increases, the balance between innovation and maintenance becomes critical. Decisions about addressing technical debt come into play much like cleaning up your living space when entering a new personal phase.

Maturity & Maintenance

Just as family life brings its own challenges, a mature product faces high technical debt. You must decide whether to invest time in refactoring or risk losing market ground.

End-of-Life

Similar to retirement, maintaining old software may require substantial external support; yet, investing too much might not be feasible when the product’s time has passed.

In both life and software, technical debt isn’t inherently bad, it’s all about making the right choices at the right time.

              +-------------------------------------+
              |        The Circle of Life           |
              +-------------------------------------+
                         /               \
         +-------------------------+   +-------------------------+
         |    Personal Life        |   |  Software Development   |
         | (Managing Technical Debt)|   | (Handling Technical Debt)|
         +-------------------------+   +-------------------------+
         |                         |   |                         |
         |  [Student Days]         |   | [Product Launch]        |
         |  (Minimal debt)         |   | (Initial, manageable)   |
         |                         |   |                         |
         |         ↓               |   |         ↓               |
         |  [Young Adulthood]      |   | [Growth Phase]          |
         |  (Cleaning up for love) |   | (Debt begins to rise)   |
         |                         |   |                         |
         |         ↓               |   |         ↓               |
         |  [Family & Parenthood]  |   | [Maturity]              |
         |  (Debt accumulates)     |   | (High technical debt)   |
         |                         |   |                         |
         |         ↓               |   |         ↓               |
         |  [Retirement]           |   | [End of Life]           |
         | (Reliance on support)   |   | (Balancing maintenance) |
         +-------------------------+   +-------------------------+
                         \               /
                           +-----------+
                           |  Balance  |
                           | is the key|
                           +-----------+

Final thoughts

Yes, it is good to have technical debt, but only to a certain degree. Just as every stage in your life requires you to weigh costs vs. benefits, so does every phase of a software project. Finding and maintaining the right balance is unique to every individual and every project.

In both life and code, embracing a little debt can pave the way for growth. Yet, always remain mindful that unaddressed debt can accumulate and ultimately affect your overall stability.

Risks vs Rewards.

Additional food for thought:

Perhaps you might also explore how agile methodologies in software are shaped by these balancing decisions, or reflect on personal experiences where strategic ignorance paved the way for (unexpected) innovation. The interplay between calculated risk and measured maintenance is what creates resilience, both for our homes and in our code.

What are your thoughts on embracing debt (technical or otherwise) as a tool for progress?

Backup MotionEye to OneDrive

Henk van den Brink — Thu, 03 Aug 2023 06:02:44 +0000

Problem

MotionEye doesn't have an integration for OneDrive, so you can't backup files as easy as when using Google Drive.

But since I have 1TB of storage on OneDrive, I do want to use it :-)

Solution

So the solution is some scripting and rclone

There are a few steps that needs to be done.

Setup new remote in rclone
Create script that does the syncing
Create log directory
Update MotionEye to use the script
Done.

1. Setup rclone

Either login as the user that is used by MotionEye (in my case motion)
Or execute it as root and later copy the rclone conf (and chown to right user)

Please follow the instruction from here to setup RClone OneDrive remote.
It is fairly easy, if your server does not have a browser you do need a second machine that does have a browser to finish the setup.

2. Create script for sync

Go to /etc/libmotioneye/
create a new file: backupToOneDrive.sh
ie nano backupToOneDrive.sh

!/bin/bash
requiredUser="motion" #Edit to suit username of the RCLone config
remoteFolderName="onedrive:backups/motioneye" #Edit to suit your OneDrive remote folder
remoteFolder=$1
if [ $remoteFolder = "Achtertuin" ]; then #Edit to suit your camera name in MotionEye
localFolder="Camera1" #Edit to suit you local camera folder
fi
if [ $remoteFolder = "Bijkeuken" ]; then #Edit to suit your camera name in MotionEye
localFolder="Camera2" #Edit to suit you local camera folder
fi

#Add more cameras here as required#
if [ $remoteFolder = "" ]; then
echo "ERROR, ROOT Folder" $rootFolder "NOT FOUND" > /var/log/RClone/error.txt
exit
else
rm -f /var/log/RClone/error.txt
fi
DIR="/etc/motioneye" # get cur dir of this script
progName=$(basename -- "$0")
cd $DIR
if pidof -o %PPID -x "$progName"; then
echo "WARNING - $progName Already Running. Only One Allowed." >> /var/log/RClone/upload.txt
echo "WARNING - $progName Already Running. Only One Allowed."
else

localCamFolder="/var/lib/motioneye/"$localFolder
remoteCamFolderName=$remoteFolderName"/"$remoteFolder
#echo $localCamFolder "to" $remoteCamFolderName
if [ ! -d "$localCamFolder" ] ; then # Check if Local sync Folder Exists
echo $localCamFolder "Folder NOT Found " $num
exit 1
fi
nowDay=$(date +"%d")
nowSuffix="th"
if [ $nowDay -eq 1 ]; then
nowSuffix=$nowDay"st"
fi
if [ $nowDay -eq 21 ]; then
nowSuffix=$nowDay"st"
fi
if [ $nowDay -eq 21 ]; then
nowSuffix=$nowDay"st"
fi
nowDate=$(date +"%A,%d$nowSuffix %B %Y")
logDate=$(date +"%d-%m-%Y")
nowTime1=$(date +"%T")
#echo "Upload Started on" $nowDate "at" $nowTime1
#echo $num "- Sending" $localCamFolder "to" $remoteCamFolderName
sudo -u $requiredUser rclone move -v --log-file /var/log/RClone/RCloneLog.txt $localCamFolder $remoteCamFolderName
nowTime2=$(date +"%T")
#echo "Upload Started on" $nowDate "at" $nowTime1 "- FINISHED AT" $nowTime2
echo $logDate "Started" $nowTime1 "Finished" $nowTime2 >> /var/log/RClone/uploads.txt
echo $localCamFolder "to" $remoteCamFolderName >> /var/log/RClone/uploads.txt
fi

echo "Finished uploading files"
exit 0

note: This base of the script comes from this ticket at GitHub. So big thanks to him for creating it.

The script is altered a bit, since the original didn't work for me on Debian at least.
And it would not sync 'beyond the day'.

Let's walk through the script a bit.

In essence what it does is, it moves all the movies recorded by MotionEye from local to remote folder.
In order to do that, there are some settings that needs to be set.

There are a few important settings and options that needs to be set.

requiredUser
remoteFolderName

requiredUser
This is the user that will execute the rclone command.
So this must be the user that is used to start the motioneye process.
note: it needs to have sudo right, more on that later

remoteFolderName
You can define the folder name that must be used as root in OneDrive.

And the camera section:
MotionEye saves the movies in a folder called Camera*ID*
If you upload that, it can be hard to know which Camera is Camera1
So there is some transformation done to map the ID directory to a nice name.

Currently there are two, but you can remove or add a new one.
Just copy the if until fi

if [ $remoteFolder = "Garden" ]; then #Edit to suit your camera name in MotionEye
localFolder="Camera1" #Edit to suit you local camera folder
fi
if [ $remoteFolder = "Kitchen" ]; then #Edit to suit your camera name in MotionEye
localFolder="Camera2" #Edit to suit you local camera folder
fi

And that is it. Script is ready, let's move on.

I am running MotionEye inside a LXC container in Proxmox and want to limit the storage, so I move the files to OneDrive to free up space.
If you don't want to move, replace it with copy in the rclone command.

3. Create log dir

Now create the log directory for rclone:

mkdir `/var/log/RClone`

Make sure that the execution user has rights, either login as that user or chown the file.

4. Update MotionEye to use the script.

Go to your MotionEye instance.

Open the settings of a Camera.
Go to the File Storage tab
Click Run a command
And use your newly created script:

sh /etc/motioneye/backupToOneDrive.sh Garden

The last part of the command is the name that maps to the camera ID.
So if you specified Camera1 in the script to be the Garden, use that.

5. Done!

Done. Now it will move all your recordings to your OneDrive

This is an initial setup and will update this over time, as the camera mapping part can be a bit more elegant (Use the name specified in MotionEye)
And probably having this script on GitHub with version control would help.

But for now, good luck!

Cascading build triggers in Azure DevOps

Henk van den Brink — Wed, 02 Aug 2023 06:41:45 +0000

Introduction

Say you want to combine the documentation of all your projects into one project. Whenever one of your projects is build it should trigger a new build of the documentation project so that one can also be updated.

In simple words: trigger downstream builds when your project is build.

The situation

We have a project: apple
And a project: fruit-basket that depends on apple
Whenever apple is build a build of fruit-basket must be triggered.

Apple

This project just does his thing.
Build your Apple project based on your needs, nothing special.
Maybe make sure to tag the sources if you want to make sure u use the same version later on.

If the documentation that you want to show needs to be build and you don't want to do that in fruit-basket make sure to upload the build-artefact.

Fruit-basket

This project needs a trigger that when apple is build it also executed the build.

You can do this by adding a pipeline resource.
Open your azure-pipelines.yaml (based on the name you gave it, or create a new one)

Then add the following:

resources:
  pipelines:
    - pipeline: appleBuild # Identifier
      source: 'apple-build' # This needs to be the name of the pipeline, including spaces if needed!
      trigger: # When do we want it to be triggered?
        branches:
          include: # When these branches are build
          - releases/*
          - main
          - master
        stages:
          - Publish # But only after this stages from apple build

Now whenever a master build is executed on Apple and the Publish is succeeded it will trigger the fruit-basket build :)

And inside that build you can then combine the Apple documentation and others to create one documentation.

Using apple

There are two ways to do this.

Checkout the source code of apple
Download the build artefact of the apple build

Checkout the source code.

For this you need to do two things.

Add the apple sources as repository
Checkout the apple sources at each job

At the resources part you need to add a repositories option:

resources: # Already there
  repositories:
    - repository: apple # Identifier to be used in this build
      type: git
      name: FruitCompany/apple # Name of the Azure DevOps repo
      ref: refs/heads/master # Branch to checkout
# As we are going to checkout multiple repositories, we also need to define that we want to use our own sources (apple-basket, ie: self)
    - repository: self
      type: git
      name: self

Now in order to actually use it, at the beginning of each job you need to checkout the sources.

Example stage:

stages:
  - stage: generate_documentation
    jobs:
      - job:
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - checkout: self # Checkout apple-basket

          - checkout: apple
            path: 'apple'
            fetchDepth: 1

         # Here you can then do whatever you want with the two sources :)

Download build artefact

Inside a job/step you need to add the following:
(For instance instead of the apple checkout, or after when you want to use pear build artefact.

         - task: DownloadPipelineArtifact@2
            inputs:
              source: specific # We want to use a specific pipeline
              project: FruitCompany # Projectname in Azure DevOps
              pipeline: apple-build # Pipeline 
              artifact: documentation-dist # Name of the uploaded build artefact

With patterns you can limit what it should download from the uploaded artefact

              patterns: |
                *.md

More information on the DownloadPipelineArtifact task can be found here

Other triggers

Maybe you want fruit-basket to build whenever code is pushed to apple?

You can do that by using the repositories resource option.
(Don't forget to 'checkout' the source code in each stage)

And there are a few other options that can trigger downstream build:

containers # Whenever a new container is pushed to Azure Container Registry
packages # Whenever a new package is released.
webhooks # You can create whatever you want with this one.

Full description of all the possibilities and options here

Multi NPM registries usage made easy (Azure DevOps)

Henk van den Brink — Mon, 31 Jul 2023 09:06:36 +0000

The problem

I use Azure DevOps for development and have several projects in it.

And I am using the feeds possibility from Azure DevOps.
This means that I need to change the NPMRC each time I switch projects so that it uses the correct registry.

Or you put that in a .npmrc in the repository but then you need to still handle the authentication somehow...

What?

I thought there must be an easier way.

Inspired by the npmrc project I created my own solution, more tailored for Azure DevOps.

But you should still be able to use it as a npmrc switcher, just like the other package but with some added functionality.
Like an interactive list.

To break it down a bit what it does:

It will create an NPMRC with the provided feed-name as registry AND create a PAT that is allowed to pull data from the artifact feed.
And provide options to automatically renew the PAT.

So how to use it?

Easiest

Only one parameter is required

feedName (the name of the Artifact feed as created in Azure DevOps)

Then there are a few defaults:

name of the NPMRC (defaults to the directory name)

Next to the name of the NPMRC and the feedname we also need the details of the Azure DevOps Organization and Project.

These two values can be retrieved from the repository.url if that is set in the package.json

azOrganization (first part of Azure DevOps url)
azProject (second part of Azure DevOps url)

Example Azure DevOps url: https://dev.azure.com/henkvandenbrink/kitchensink

azOrganization = henkvandenbrink

azProject = kitchensink

feedName = As specified

Full example:

snmprc create feed-name

This will generate an NPMRC with the specified feed-name as registry, the azure details are retrieved from package.json and current directory name is used as name.

More details options

When you are not able to use the defaults or you just want to have more control, you can specify all the parameters as well.

snpmrc create feed-name henkvandenbrink kitchensink directory-name

This will generate an NPMRC with the specified feed-name as registry, the azure details that are provided and directory-name is used as name.

For all other options, more information and details please go to the simple-aznpmrcs npmjs package where there is a full readme available.

Feature requested are welcome and/or pull requests.

Just another step into automating those manual steps :)

How to test your Ansible playbook with Vagrant

Henk van den Brink — Mon, 20 Sep 2021 19:59:21 +0000

How it all started

Infrastructure as Code is getting more and more attention and for good reasons.
Previously infrastructures where created and maintained by hand, which works well until...

So when I needed to create a new database cluster I looked into a setup that would be reproducible and it should be testable before it was deployed on the actual servers.

For stateless things we can used docker but that wasn't a great fit for the database servers as we need to have persistent storage and scaling and/or changes where not expected that often.

I went for a golden oldie: Ansible
While I had worked with it before, it was mainly putting some roles together run the script and validate, manually, that it worked.

This time around I didn't want to 'test' the provisioning manually. As the setup was a bit larger and the overhead (commit, deploy) was bigger.

So I thought there should be a smarter and better way.

Start of the journey

As I am mainly a dev engineer, I didn't had thorough expertise on Ansible. (still don't) So Google it was to get the journey started. Looking through all the things, I mostly saw tests for Ansible Roles. Or ways to test the playbook on the target systems, but I wanted to test the playbook before deployment during the build and/or locally.

Turned out it was not that hard to create, but getting all the information was.
Because of the combination of Ansible and Vagrant and probably my lack of knowledge on those topics 😃~

The project setup

First off, what are we actually going to provision?
We will be provisioning a 3 server cluster of a database server, RethinkDB
You probably didn't heard from it before, and you can forget about it after reading this.

There will be some specific commands for rethinkDB, but most of it will be about the testing setup and ways you should test.

Project layout

A pretty normal layout for Ansible projects.
The part we will focus on is the test folder.

Test

There are a few levels of testing that can be done for the Ansible playbook.

You can have Ansible roles unit tests, which will validate if the roles 'works'.
And you can have tests for the playbook.

In my case the roles I created where specific for my situation so I didn't, wanted to, create unit tests as that would have been a bit overkill.

Instead I focussed on a few other things:

The role itself should validate if the work it did was actually executed and working as intended.

Making sure roles are doing what they should

So for instance the install-rethinkdb role should validate that it did install the package provided.

To do so, we get the version that is installed by executing the rethinkdb -v command and save the output. In another step we validate that the output is what we expect.
This way we are verifying that the installation of the (new) version went successful as the version is being used.


    - name: Install rethinkdb
      yum:
        name: "/tmp/rethinkdb-{{ rethinkdb_version }}.x86_64.rpm"
        state: present
        allow_downgrade: yes
        disable_gpg_check: yes        

    - name: Get installed rethinkdb version
      command: rethinkdb -v
      register: rethinkdb_version_installed

    - name: Validate rethinkdb version used is the same as wanted version
      assert:
        that:
          - "'{{ rethinkdb_version }}' in rethinkdb_version_installed.stdout"

In another role I updated the configuration and needed to restart the database in order to get the configuration working.

For this we restart the service via systemd and wait for couple of seconds.
We then get the service_facts and validate that the service is in a running state.

- name: Make sure rethinkdb service is restarted
    systemd:
      state: restarted
      name: rethinkdb.service
      enabled: yes
      daemon_reload: yes

  - name: Pause for 10 seconds to start the service
    pause:
      seconds: 10

  - name: Get service statuses
    service_facts:
    register: service_state_rethinkdb

  - name: Validate status of rethinkdb service
    assert:
      that:
        - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb.service'].state"

The take way here is that you should look at your roles as parts that will have a single purpose and that you should also validate that they do what you expect. Role tests and such are great but verifying it during runtime is a must, as that is what matters in the end

Testing the playbook in a controlled (local/build) environment

For testing the playbook we need servers to executed the playbook on.
In comes, if you looked at the layout you could have guessed it, Vagrant

So here is the Vagrantfile that I used, I added some comments in there to help you understand what is going on.

In short, it will provision 3 servers that are in an own private network and execute the (test) playbook on those servers. Effectively creating a local database cluster.

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Copy certificate files to destination they are expected.
FileUtils.cp %w(tests/files/ca.pem tests/files/cert.pem tests/files/key.pem), 'roles/ansible-role-rethinkdb-configure/files/'
FileUtils.cp %w(tests/files/vagrant1.local.net.crt tests/files/vagrant1.local.net.key), 'files/'
FileUtils.cp %w(tests/files/vagrant2.local.net.crt tests/files/vagrant2.local.net.key), 'files/'
FileUtils.cp %w(tests/files/vagrant3.local.net.crt tests/files/vagrant3.local.net.key), 'files/'

Vagrant.configure("2") do |config|

  # If you have issues with SSL certificates add this
  config.vm.box_download_insecure = true
  # We want to use redhat7 as the target servers are also redhat 7
  config.vm.box = "generic/rhel7"

  # To save overhead and time we use linked_clones (https://www.vagrantup.com/docs/providers/virtualbox/configuration#linked-clones)
  config.vm.provider "virtualbox" do |v|
    v.linked_clone = true
  end

  # We want to provision 3 servers for our cluster
  N = 3
  (1..N).each do |machine_id|
    # Give the server a unique name
    config.vm.define "vagrant#{machine_id}" do |machine|
      # same for the hostname of the VM
      machine.vm.hostname = "vagrant#{machine_id}"
      # Here we will be setting up the private network for the cluster.
      # In this network the cluster can communicate to each other.
      # We also want to expose (forward) some ports that are used inside the VM to the host (so that we can connect to the dashboard)
      # In this example I only added it for server1.
      if machine_id === 1
        # Private network instruction for vagrant
        machine.vm.network "private_network", ip: "192.168.77.#{20+machine_id}", virtualbox__intnet: "network", name: "network"
        # Forward all the ports below to the host. auto_correct is set to true to chose an available port if the one specified is taken
        machine.vm.network "forwarded_port", guest: 10082, host: 10082, auto_correct: true
        machine.vm.network "forwarded_port", guest: 10081, host: 10081, auto_correct: true
        machine.vm.network "forwarded_port", guest: 8443, host: 8443, auto_correct: true
        machine.vm.network "forwarded_port", guest: 9088, host: 9088, auto_correct: true
        machine.vm.network "forwarded_port", guest: 9089, host: 9089, auto_correct: true
      else
        # The other servers (2,3) should be in the private network
        machine.vm.network "private_network", ip: "192.168.77.#{20+machine_id}", virtualbox__intnet: "network"
      end

      # Only execute when all the machines are up and ready.
      if machine_id == N
        machine.vm.provision :ansible do |ansible|
          # Disable default limit to connect to all the machines
          ansible.limit = "all"
          # Playbook that should be executed, in this case the test playbook
          ansible.playbook = "tests/test.yml"
          # Please log it all
          ansible.verbose = true
          # We need to provied a few extra vars to the playbook
          ansible.extra_vars = {
            RETHINKDB_DATABASE_ADMIN_PASSWORD: '',
            NGINX_DASHBOARD_PASSWORD: 'pasword'
          }
          # These groups are the same as you normally specify in the inventorie file
          ansible.groups = {
            leader: ["vagrant1"],
            followers: ["vagrant2", "vagrant3"],
          }
        end
      end
    end
  end

end

The 'test' playbook looks like this:
I am using a specific playbook for tests as I needed to add a few things that are normally there if you get a server from your provider. At least that was the case in my project.

---

- name: Install rethinkdb 
  hosts: all # Executed this on all 3 servers
  become: yes # become root

  # pre_tasks are there to create the rethinkdb user. 
  # These will be executed before the rest.
  pre_tasks:
    - name: Add group "rethinkdb01"
      group:
        name: rethinkdb01
        state: present

    - name: Add user "ansi_rethinkdb01"
      user:
        name: ansi_rethinkdb01
        groups: rethinkdb01
        shell: /sbin/nologin
        append: yes
        comment: "vagrant nologin User"
        state: present

    - name: Add user "rethinkdb01"
      user:
        name: ansi_rethinkdb01
        groups: rethinkdb01
        shell: /sbin/nologin
        append: yes
        comment: "vagrant nologin User"
        state: present

    - name: Allow 'rethinkdb01' group to have passwordless sudo
      lineinfile:
        dest: /etc/sudoers
        state: present
        regexp: '^%rethinkdb01'
        line: '%rethinkdb01 ALL=(ALL) NOPASSWD: ALL'
        validate: 'visudo -cf %s'

    - name: Add sudoers users to rethinkdb01 group
      user: name=ansi_rethinkdb01 groups=rethinkdb01 append=yes state=present createhome=yes

    - name: Add sudoers users to rethinkdb01 group
      user: name=rethinkdb01 groups=rethinkdb01 append=yes state=present createhome=yes

# As the firewall was interrupting with the private network I disabled it
    - name: Stop and disable firewalld.
      service:
        name: firewalld
        state: stopped
        enabled: False

 # Rethinkdb can be memory happy so we needed to increase the virtual memory on the real servers. Here to test if it works.
    - name: Increase virtual memory
      shell: echo "vm.max_map_count=262144" >> /etc/sysctl.d/rethinkdb.conf && sudo sysctl --system

# We need to have  machine certificates for cluster security.
    - name: Copy certificate files
      copy:
        src: "{{ item.src }}"
        dest: "{{ item.dest }}"
        group: "{{ target_group }}"
        owner: "{{ target_user }}"
        force: yes
      loop:
          - {src: '{{ ansible_hostname }}.local.net.crt', dest: '/etc/ssl/certs/{{ ansible_hostname }}.local.net.crt'}
          - {src: '{{ ansible_hostname }}.local.net.key', dest: '/etc/ssl/certs/{{ ansible_hostname }}.local.net.key'}

# The roles that will be executed.
  roles:
    - name: ../roles/ansible-role-install-rethinkdb-offline
    - name: ../roles/ansible-role-rethinkdb-configure
    - name: ../roles/ansible-role-install-node-exporter-offline

# This is the real 'testing' part of the playbook
# As described before each role should make sure it does what it should do.
# But in order to make sure the provisioning works as a whole we will be testing some things here again

  post_tasks:
    - name: Get virtual memory config
      shell: sysctl vm.max_map_count
      register: rethinkdb_virtual_memory

    - name: Validate virtual memory config is correct
      assert:
        that:
          - "'vm.max_map_count = 262144' in rethinkdb_virtual_memory.stdout"

    - name: Get service status
      service_facts:
      register: service_state_rethinkdb

    - name: Validate status of rethinkdb service
      assert:
        that:
          - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb.service'].state"

# Call the node_exporter locally and save the output
    - name: Check if node exporter is working
      uri:
        url: https://localhost:10081/metrics
        return_content: yes
        validate_certs: no
      register: node_exporter_response

# Validate that there is some output.  
    - name: Validate node exporter running
      assert:
        that:
          - "'go_goroutines' in node_exporter_response.content"

- name: install horizon and proxy
  hosts: leader # Only execute this on the servers in the `leader` group
  become: yes

  roles:
    - name: ../roles/ansible-role-rethinkdb-horizon
    - name: ../roles/ansible-role-rethinkdb-exporter
    - name: ../roles/ansible-role-rethinkdb-nginx

  post_tasks:
    - name: Get service status
      service_facts:
      register: service_state_rethinkdb

    - name: Validate status of rethinkdb service
      assert:
        that:
          - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb.service'].state"

    - name: Validate status of rethinkdb-exporter service
      assert:
        that:
          - "'running' in service_state_rethinkdb.ansible_facts.services['rethinkdb-exporter.service'].state"
      run_once: true    

# The rethinkdb rethinkdb_exporter should provided metrics, and we want to check the results so save it to a variable.
    - name: Check if all nodes are connected
      uri:
        url: https://localhost:10082/metrics
        return_content: yes
        validate_certs: no
      register: exporter_response
      run_once: true

# Validate that there are three servers connected to the cluster according to the rethinkdb_exporter
    - name: Validate number of nodes running
      assert:
        that:
          - "'rethinkdb_cluster_servers_total {{ rethinkdb_number_of_nodes }}' in exporter_response.content"
      run_once: true

...

In the post_taks you can add as many assertions as you want/need. Make sure that you at least test the basics, so that the thing you installed is up and working.

Execute it

Well we only need to run one command to test the whole thing:

 vagrant up

Or if you want to execute it on Azure DevOps:

stages:
  - stage: Verifying
    pool:
      vmImage: 'macOS-latest'
    jobs:
      - job: Test_the_setup
        condition: or(eq('${{parameters.testRethinkDBSetup}}', true), eq(variables.isMain, false))
        steps: 
          - script: brew install ansible
            displayName: 'Install ansible'
          - script: |
              cd rethinkdb
              vagrant up    
            displayName: 'Run rethinkdb ansible scripts'

This will start the Vagrantfile and in effect create the VM's and start the playbook execution.

vagrant1: Calculating and comparing box checksum...
==> vagrant1: Successfully added box 'generic/rhel7' (v3.2.10) for 'virtualbox'!
==> vagrant1: Preparing master VM for linked clones...
    vagrant1: This is a one time operation. Once the master VM is prepared,
    vagrant1: it will be used as a base for linked clones, making the creation
    vagrant1: of new VMs take milliseconds on a modern system.
==> vagrant1: Importing base box 'generic/rhel7'...

---
PLAY [Install rethinkdb] *******************************************************

TASK [Gathering Facts] *********************************************************
ok: [vagrant2]
ok: [vagrant3]
ok: [vagrant1]

TASK [Add group "rethinkdb01"] *************************************************
changed: [vagrant2] => {"changed": true, "gid": 1001, "name": "rethinkdb01", "state": "present", "system": false}
changed: [vagrant1] => {"changed": true, "gid": 1001, "name": "rethinkdb01", "state": "present", "system": false}
changed: [vagrant3] => {"changed": true, "gid": 1001, "name": "rethinkdb01", "state": "present", "system": false}
---

PLAY RECAP *********************************************************************
vagrant1                   : ok=79   changed=48   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vagrant2                   : ok=44   changed=27   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vagrant3                   : ok=44   changed=27   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

With this setup you are sure that the provisioned servers are working as intended and that if you make a change it is validated. Even after the creator left 😄

We now have a local running database cluster that is tested and, as a bonus, you can use it for development.

If you have any question regarding testing Ansible playbooks and or setting up a local Vagrant (cluster) environment, please let me know! (Not sure if I can answer them but will do my best 😰)

Also any other remarks/improvements are always welcome.

Build versioning made easy in Azure DevOps

Henk van den Brink — Fri, 17 Sep 2021 20:41:15 +0000

Introduction

Versioning is a very important part of creating, deploying and maintaining code. Having a understandable and readable version is required.

Here are a few example of versioning systems:

And probably a few others are out there.

I have only used the one that most people use: SemVer

The problem

I moved to Azure DevOps for building and releasing software.
After getting used to it there was one big thing that bugged me, the versioning of the builds and releases...

By default Azure DevOps uses $(Date:yyyyMMdd).$(Rev:r) as the build and release number.

And that is something where you cannot do anything with...
Yes it is unique but there is absolutely no way for you to find out which build/release pipeline contained the version you wanted to look at.

The other issue I had was the fact that when building a PR the version that was used was the one from master/main (if branched from there) and most likely this version was already released. So creating an artefact for test deployment for instance would fail, it should be a unique version.

The solution helper

So there must be a way to get the behaviour I wanted, after searching and searching I could not find an existing solution that was easy to use and wasn't polluting the templates.

So I created a little NPM module to help me: simple-versioner

What it does

This helper reads the version from the file that contains the version, defaults to package.json

There are two cases:

Stable release (ie: master, or other stable branch provided
It will use the version as is. IE 1.0.1
Build is for a non stable branch and/or a Pull Request
The versioning specified will be postfixed with the branch name and the commit sha.
IE: 1.0.1-refs-heads-branch-8a9fee0b

This way the version used in a Pull Request will always be unique, so that you can use the artefact.
And in the build pipeline overview you can easily find the build of your version:

Extra bonus: It validates it the version already exists or not (git tag with that version exists) if it exists it will fail and therefor stopping the build

How you can use it

So hopefully I got your exited to also use it or at least give it a try.

Well as the aim was to have a simple plugin, you only need to add two lines to your azure yml. (make sure to add it as one of the first tasks in your pipeline)

- bash: 'npx simple-versioner'
  displayName: 'Get and set correct build version'

That's it.
It will update the Build.Buildnumber parameter in Azure and the provided versioning file with the (new) correct version.

Done.

Possible options

There are a few options available to customise the execution of the plugin. Please have a look at simple-versioner to see the latest possible options.

For what can you use it?

Per default it expect a package.json that's because I created the plugin for that purpose.
But you can use it on any JSON file that has a version attribute.

So for instance the vss-extension.json which is used for Azure DevOps tasks.

Other files and or structure can easily be added, have a look at the code and PR's are welcome :) simple-versioner