DEV Community: Joseph Heyburn

Alerting on NHS Coronavirus Vaccine Updates With Huginn

Joseph Heyburn — Thu, 03 Jun 2021 00:00:00 +0000

UPDATE 2021-06-06

A few days after publishing this I came across a bug where agent jobs would be stuck in pending state. I've since fixed this and documented some additional changes I've made at the end of the post.

The UK’s coronavirus vaccine strategy has been to target those most vulnerable first, and then trickle down towards the healthier population. Since that age is creeping down toward my age group, I wanted to see if I could alert myself when I would be eligible for the vax.

My local GP would send out an SMS text message informing me when I’m eligible, however I’ve heard that this text can come days after you’re eligible. Knowing that the latest guidance is maintained on the NHS Coronavirus Vaccine site, I can use Huginn to alert me when the page updates with the latest eligibility.

tl;dr

Huginn is an automation tool with a number of different agents
It can be configured to monitor a property (or properties) on a web page and trigger an action
That action can take the form of an email alert
This can be used to monitor the latest age group eligible for a vaccine

Deploying Huggin

Huginn is a self-hosted automation kit that allows you to create agents and workflows in response to events, sort of like your own IFTTT (If This Then That).

Being self-hosted it can be deployed out in a number of ways. I already have Portainer (a GUI for Docker) running in a virtual machine - so to deploy it out I can follow the instructions for Docker container deployment. I created a docker-compose file so that it can be easily replicated for yourselves in Docker, or even as a Portainer Stack.

You’ll notice there are some environment variables for SMTP here; the values for these will differ for your SMTP setup. I talk more about how I set this up with my Gmail account later in the post.

Also big thanks to zblesk for their blog post on Huginn which helped me iron out some of the environment variables!

version: "3"
services:
  huginn:
    command:
      - /scripts/init
    container_name: huginn
    environment:
      - SMTP_PORT=587
      - SMTP_SERVER=<SMTP_SERVER>
      - SMTP_PASSWORD=<SMTP_PASSWORD>
      - SMTP_USER_NAME=<SMTP_USER_NAME>
      - SMTP_ENABLE_STARTTLS_AUTO=true
      - SMTP_AUTHENTICATION=plain
      - SMTP_DOMAIN=<SMTP_DOMAIN>
      - DATABASE_POOL=30
      - TIMEZONE=London
      - IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS=false
    image: huginn/huginn:latest
    ports:
      - 3000:3000/tcp
    restart: unless-stopped
    user: "1000"
    volumes:
      - mysql:/var/lib/mysql
    working_dir: /app

volumes:
  mysql:
    driver: local

Be sure to change TIMEZONE to your timezone, I found that having an incorrectly set timezone caused Huginn jobs to be backed up in a pending state. I tried Europe/London first but that caused the process to crash on boot; so I ultimately got it working with just London.

The config above will expose Huginn on the Docker host at port 3000. I like to give my services a nice domain name to access them at using Caddy, which I’ve written about before. Here’s a condensed version of what my Caddy config file looks like for Huginn.

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "192.168.2.15:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "huginn.joannet.casa"
                  ]
                }
              ],
              "terminal": true
            },
            // ... others removed for brevity
          ],
          "tls_connection_policies": [
            {
              "match": {
                "sni": [
                  "huginn.joannet.casa",
                  // ... others removed for brevity
                ]
              }
            },
            {}
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuer": {
              "challenges": {
                "dns": {
                  "provider": {
                    "api_token": "CLOUDFLARE_API_TOKEN",
                    "name": "cloudflare"
                  }
                }
              },
              "module": "acme"
            },
            "subjects": [
              "huginn.joannet.casa",
              // ... others removed for brevity
            ]
          }
        ]
      }
    }
  }
}

I’ll need to also add in a DNS entry in PiHole to route HTTP calls on my network for https://huginn.joannet.casa to the IP address of my Caddy server.

Deployed and ready for set up!

Setting up agents

The workflow we need to set up here is pretty simple whereby we only need two agents; a Website Agent and an Email Agent. The website agent will perform the scraping of the NHS website on a regular basis, and if the component on the web page has changed, then it will invoke it’s downstream notifier - the Email Agent.

Website Agent ------ invokes ------> Email Agent

Before we do that we’ll need to create an account for it - this is all local to your deployment and is not external. The invitation code you’ll need to enter is try-huginn.

When you get through to the main page, I suggest disabling or removing the agents that are there already to prevent some problems later on. If you set IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS=false then you should not see any there.

Website agent

From the Huginn home page, create a new agent, where the type will be Website Agent.

There will be a bunch of fields that appear, the only ones you need to fill in are:

Name
- e.g. NHSScrape
Schedule
- for me, checking once an hour is good enough
Options

The other fields serve a purpose that’s beyond the scope of this post.

Within Options comes the configuration used to define the website agent. The documentation for the agent config appears on the right hand side, so you can read through that for reference.

In order for us to determine how to configure this we first need to decide what part of the web page we want to be alerted on in event of a update. Refer to the screenshot below of the NHS Coronavirus Vaccine page and you’ll see a number of bullet points listed out of the vaccine criteria. The text we want to be alerted on is “people aged 30 and over”.

The website agent supports an xpath syntax as a config option, which is an expression syntax used to retrieve objects from XML documents - including HTML. But how do we find out what the xpath for this text field is?

Enter SelectorGadget. It’s a bookmarklet tool that helps with just that - I recommend taking a look at the short tutorial on how to use it.

Since I know I only want to select the first bullet point in this particular list, I start off by first clicking that property which turns that node green.

Clicking on the property I want to target highlights it green

You can see the xpath for this is a <li> node, which returns 76 results on the page we’re scraping. These are represented by the boxes that are highlighted yellow.

What I want to do is whittle it down by filtering out all the other <li> nodes I’m not interested in, so that I only have 1 result left. As I’ve already made my first selection, any subsequent clicks will now filter those out. So now it’s just a case of playing whack-a-mole until all yellow highlighted fields are gone.

Clicking the second bullet point indicated that I don’t care about any other <li> elements in this range.

<li> properties also make up headings at the top of the page - so these appear highlighted in yellow too.

Clicking the Home element filters that out.

Scrolling down further on the page, there’s another <li> range which needs to be filtered.

Boom - filtering that means we only have 1 element being targeted.

We now only have 1 selection, so now we can click the XPath button in the tool to retrieve the config we need.

//ul[(((count(preceding-sibling::*) + 1) = 6) and parent::*)]//li[(((count(preceding-sibling::*) + 1) = 1) and parent::*)]

N.B. since writing this it looks like the xpath changed - I’ve updated the config below with the same. I retrieved it using the same method as described above.

As a future task it’d be great to see if we could be alerted on when the working status of a job fails - but it seems that feature is missing.

Going back to the agent config, this xpath syntax then goes into the xpath key.

{
  "expected_update_period_in_days": "2",
  "url": "https://www.nhs.uk/conditions/coronavirus-covid-19/coronavirus-vaccination/coronavirus-vaccine/",
  "type": "html",
  "mode": "on_change",
  "extract": {
    "title": {
      "xpath": "//ul[(((count(preceding-sibling::*) + 1) = 4) and parent::*)]//li[(((count(preceding-sibling::*) + 1) = 1) and parent::*)]",
      "value": "normalize-space(.)"
    }
  }
}

Once it’s configured then you can click on Dry Run at the bottom to see the text it extracts.

Email agent

Now that the website agent is set up we need to set up our alert destination; this takes the form of an email agent. At minimum, we only need to configure these fields:

Name
- e.g. NHSEmail
Sources
- select the website agent you created beforehand
Options

Note that an agent such as email can have multiple sources - so if you wanted to be alerted on multiple web pages then you only need one email agent.

The options field is not as complex as the website agent - mine is configured with this:

{
  "subject": "NHS Coronavirus Page update",
  "headline": "Vaccine age updated",
  "expected_receive_period_in_days": "2"
}

The real complexity lies in configuring Huginn to send email…

Configure Huginn for sending email over Gmail SMTP

I have a Gmail account which opens up SMTP access to allows applications to send email programatically, where Huginn has support for this. You’ll notice earlier when we deployed Huginn that I had a number of environment variables configured for SMTP. It took a bit of trial-and-error and searching through GitHub issues to get it right, but that config works for me.

Since my Gmail account is set up with 2FA, I cannot use my actual Gmail password in the SMTP_PASSWORD field. Instead what I have to do is set up an application-specific password that only Huginn is configured for. This restriction known in Google as less secure app access.

As mentioned, we’ll need to set up an app password. For this I navigated to the app password page for my account and selected Other (Custom name) from the app dropdown, to then enter the name of the app. The name doesn’t matter here - it’s for your reference.

When you click on Generate it will display the password assigned. It is this value that goes into the SMTP_PASSWORD env var for Huginn to pick up.

As a recap, the environment variables that I needed to set in order to get emails to be sent were:

SMTP_DOMAIN=gmail.com
SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
SMTP_AUTHENTICATION=plain
SMTP_USER_NAME=$EMAIL_ADDRESS
SMTP_PASSWORD=$APP_PASSWORD
SMTP_ENABLE_STARTTLS_AUTO=true
DATABASE_POOL=30
- While not directly related to SMTP, it helps ensure there are enough threads to process database requests

Note if you did not start up Huginn with these environment variables already set, then you’ll need to restart the service so that it can pick them up.

Testing the flow

We can test the whole flow by making a modification to the website agent we created. Currently the mode we have it set to is on_change which is the desired end mode, and will only trigger its receivers if there has been a change in the property being selected. If we change the mode to be all then it will always invoke the receivers.

Couple this with setting a frequent schedule (i.e. every 5m) then we should be receiving an email with the current value every 5 minutes.

Once the flow is tested, we can set the mode on the website agent back to on_change

Now we just play the waiting time until getting vaxx’ed up as Marc Rebillet says… 💉

Fixing agent jobs stuck in pending state

Using the docker compose file above caused an issue for me where Huginn would get jobs stuck in a pending state - where rebooting the container was the only way to unblock them... no good for an alerting app!

Huginn by default includes a mysql daemon as the datastore if none is provided in the environment variables. I decided to have mysql running in a separate container to see if that fixed it... and it did!

My new docker compose file looks like this:

services:
  huginn:
    command:
      - /scripts/init
    container_name: huginn_huginn
    environment:
      - SMTP_PORT=587
      - SMTP_SERVER=<SMTP_SERVER>
      - SMTP_PASSWORD=<SMTP_PASSWORD>
      - SMTP_USER_NAME=<SMTP_USER_NAME>
      - SMTP_ENABLE_STARTTLS_AUTO=true
      - SMTP_AUTHENTICATION=plain
      - SMTP_DOMAIN=<SMTP_DOMAIN>
      - TIMEZONE=London
      - DATABASE_POOL=30
      - DATABASE_NAME=huginn
      - DATABASE_USERNAME=huginn
      - DATABASE_PASSWORD=<MYSQL_PASSWORD>
      - DATABASE_HOST=huginn_mysql
      - DATABASE_PORT=3306
      - START_MYSQL=false
      - DATABASE_ENCODING=utf8mb4
      - IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS=false
      - DOMAIN=huginn.joannet.casa
      - INVITATION_CODE=<INVITATION_CODE>
    image: huginn/huginn:latest
    ports:
      - 3000:3000/tcp
    restart: unless-stopped
    user: "1000"
    working_dir: /app
    depends_on:
      - mysql
  mysql:
    image: mysql
    container_name: huginn_mysql
    restart: always
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=<MYSQL_ROOT_PASSWORD>
      - MYSQL_DATABASE=huginn
      - MYSQL_USER=huginn
      - MYSQL_PASSWORD=<MYSQL_PASSWORD>
    volumes:
      - mysql:/var/lib/mysql

volumes:
  mysql:
    driver: local

The mysql container is pretty standard so I won't cover that here. There are some env vars I had to add to the huginn container:

DATABASE_HOST - the database hostname to connect to, we can use the container name here
DATABASE_PORT - the port to which to connect to the database
DATABASE_NAME - the name of the database to use
DATABASE_USERNAME - who we should connect to the database as
DATABASE_PASSWORD - authentication for the user
DATABASE_ENCODING - a requirement when using a newer version of MySQL as defined in the documentation
START_MYSQL - whether to use a local mysql daemon or not

Some additional env vars I added unrelated to the new database:

IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS - I don't care able the agents that are added by default
INVITATION_CODE - Lock down Huginn by requiring this code for new user sign ups
DOMAIN - the endpoint that Huginn is available at

I've also added in a depends_on on the database container to assist with orchestration. On first boot however it takes some time for mysql to initialise the database, so Huginn may fail as the database is not yet ready to be connected to. Once the initialisation is done then reboot the Huginn container and it should be able to bootstrap the database fine.

Backup to Backblaze B2 using restic and rclone

Joseph Heyburn — Tue, 04 May 2021 00:00:00 +0000

Over the last UK lockdown I spent some time making sure I had backups of my music collection after I had organised them using beets. I used this as a good opportunity to ensure I had other backups in place for some other critical files at home too.

Backup tools

Restic is a backup snapshot tool that given a set of directories, it will split the data into chunks and de-duplicate these chunks to a specified backup repository. You can specify your backup policy to a repository, where it will manage what daily, weekly, or monthly snapshots to keep.

Rclone is a cloud file transfer tool that allows you to synchronise, copy, etc., any data across a huge library of cloud backends.

Both of these are open source tools that are free to use. Getting set up on them is beyond the scope of this post.

Backups should be automated without us having to think of them (at least until you need to restore!), so for this I’ll be using systemd unit services with timers.

Restic is storing the snapshots in repositories on my local NFS server. I’m then using Backblaze B2 to store these in the cloud. At the time of writing they charge $0.005 GB/month for storage, and $0.01 per GB downloaded. So it costs more to download from B2, but since this is a disaster recovery backup I don’t anticipate downloading all that much. Being able to store it cheap over a long term is most important.

There are a load of other storage services too; here’s a list below taken from Backblaze’s website. The other one I was contending with was Wasabi; they’re only marginally more expensive than B2 for storage ($0.0059 GB/month), but they don’t charge for downloads.

Cloud storage cost comparisons

Backup contents and policy

I have restic set up to backup to two respositories:

small-files
- PiHole configuration
- UniFi backups
- Logitech Media Server (LMS) backups
media
- all music files
- beets databases

Reason being why these are split into two repositories is so that I can define a separate backup policy for each of them.

For example, small-files contains… small files. You can see from the list above that these are just configuration files or even backups of files themselves. Since config files are important, I’d like the option to go back to a particular setting from weeks or maybe months ago. So the snapshots I want to keep for this repository is 30 daily snapshots, 5 weekly snapshots, 12 monthly snapshots, and 3 yearly snapshots. It might be a bit overkill, but I can always reduce that number and restic will remove them.

Whereas for media, these files tend to be much larger which will be reflected in the repository size. Especially since music files don’t tend to change over time - all I’m looking for is a way of reverting back to a state I had recently in case I did some reorganising with beets. Based on this I only need to keep 30 daily snapshots.

Also in the media repository are the beets databases. I’ll expand on how I have this set up in a future post, but for now think of it as a database containing the metadata for your music collection. It’s useful to include this with my music files so that I can reflect on the state of my collection at a particular point in time.

Backup procedure

I have a 2TB USB hard drive connected to a Pi (dee) which is configured to be the NFS server for my network, along with PiHole for DNS, and a UniFi controller too. The drive holds my music collection and the restic local repositories.

LMS sits on a different machine to dee so we’ll need to retrieve the files first before we perform a snapshot.

Once restic has taken snapshots and stored them on the USB drive, we’ll need to upload the backups to B2 with rclone.

N.B. the configuration to automate restic with systemd was largely inspired from a post on Fedora Magazine.

You can view the code for all these scripts and systemd unit files at my GitHub repository.

Retrieving backups from remote servers

I’ll have the restic backup script call another script, pull-backups.sh to retrieve remote files that I want to back up.

For now I only need to retrieve LMS backups from PiCorePlayer - for which there is a handy command to help with that: pcp bu. Don’t get too invested in the file path locations, these are specific to PiCorePlayer.

# pull-backups.sh

#!/usr/bin/env bash

# Invoke and pull backups from remote servers and place them on USB

function pcp() {

    local fpath="/mnt/mmcblk0p2/tce"
    local fname="mydata"
    local ext="tgz"
    local now=$(date -u +"%Y-%m-%dT%H%M%S")
    local dstPath="/mnt/usb/Backup/lms/*.${ext}"

    echo "removing previous backups..."
    rm -rf $dstPath

    local sourceFile="${fpath}/${fname}.${ext}"
    local dstFile="/mnt/usb/Backup/lms/${fname}_${now}.${ext}"

    echo "starting pcp backup"
    ssh -i /home/jdheyburn/.ssh/pcp tc@pcp.joannet.casa -C 'pcp bu'

    echo "copying $sourceFile on remote to $dstFile"
    scp -i /home/jdheyburn/.ssh/pcp "tc@pcp.joannet.casa:${sourceFile}" $dstFile
}

function main() {
    echo "entered $0"
    pcp
}

main $@

This takes the backup created on the remote at /mnt/mmcblk0p2/tce/mydata.tgz and places it locally at /mnt/usb/Backup/lms/mydata_DATETIME.tgz. This lms directory can then be used as a backup path for restic.

Should I need to retrieve backups from any other servers, I’ll write a new function here and append it to main.

Restic backup

# restic-all.sh

#!/usr/bin/env bash

# Master script for backing up anything to do with restic

function do_restic() {
    local mode=$1
    local target=$2

    if [$mode == "backup"]; then
        echo "Backing up $target"
        restic backup --verbose --tag systemd.timer $BACKUP_EXCLUDES $BACKUP_PATHS
        echo "Forgetting old $target"
        restic forget --verbose --tag systemd.timer --group-by "paths,tags" --keep-daily $RETENTION_DAYS --keep-weekly $RETENTION_WEEKS --keep-monthly $RETENTION_MONTHS --keep-yearly $RETENTION_YEARS
    elif [$mode == "prune"]; then
        echo "pruning $target"
        restic --verbose prune
    fi
}

function small_files() {
    export BACKUP_PATHS="/var/lib/unifi/backup/autobackup /etc/pihole /mnt/usb/Backup/lms"
    export BACKUP_EXCLUDES=""
    export RETENTION_DAYS="7"
    export RETENTION_WEEKS="5"
    export RETENTION_MONTHS="12"
    export RETENTION_YEARS="3"
    export RESTIC_REPOSITORY="/mnt/usb/Backup/restic/small-files"
    export RESTIC_PASSWORD_FILE="/home/restic/.resticpw"

    local mode=$1

    do_restic $mode "small files"
}

function media() {
    export BACKUP_PATHS="/mnt/usb/Backup/media/beets-db /mnt/usb/Backup/media/lossless /mnt/usb/Backup/media/music /mnt/usb/Backup/media/vinyl"
    export BACKUP_EXCLUDES=""
    export RETENTION_DAYS="30"
    export RETENTION_WEEKS="0"
    export RETENTION_MONTHS="0"
    export RETENTION_YEARS="0"
    export RESTIC_REPOSITORY="/mnt/usb/Backup/restic/media"
    export RESTIC_PASSWORD_FILE="/home/restic/.resticmediapw"

    local mode=$1

    do_restic $mode "media"
}

function main() {

    mode=$1

    if [$mode == "backup"]; then
        /home/jdheyburn/dotfiles/restic/pull-backups.sh
    fi

    small_files $mode
    media $mode
}

main $@

This script is a bit messy, but it gets the job done. It takes in an argument which can be either backup or prune, depending on what task needs to run. If the mode is backup then it will invoke the pull-backups.sh prior to snapshotting so that it has the latest files to backup. I cover the prune function later.

Both restic repositories will be kept under /mnt/usb/Backup/restic.

Once the script is defined we can have systemd invoke it.

# /etc/systemd/system/restic-all.service

[Unit]
Description=Restic backup everything service
OnFailure=unit-status-mail@%n.service

[Service]
Type=oneshot
ExecStart=/home/jdheyburn/dotfiles/restic/restic-all.sh backup

[Install]
WantedBy=multi-user.target

You can see that we’re passing in the mode as an argument at ExecStart.

We can perform a test to make sure everything is defined correctly by running systemctl start restic-all.service, and viewing the logs back at journalctl -u restic-all.service -f.

In order to then have this systemd unit invoked on a regular occurrence, we need to define a timer for this unit.

# /etc/systemd/system/restic-all.timer

[Unit]
Description=Backup with restic daily

[Timer]
OnCalendar=*-*-* 2:00:00
Persistent=true

[Install]
WantedBy=timers.target

This will invoke the service at 2am every day once we enable it with systemctl enable restic-all.timer.

We can have a look at the resulting snapshots with the below command.

$ restic -r /mnt/usb/Backup/restic/small-files/ snapshots
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
ID Time Host Tags Paths
-----------------------------------------------------------------------------------------------
d93573f4 2020-06-30 02:00:01 dee systemd.timer /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

96ecf97e 2020-07-31 02:00:35 dee systemd.timer /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

d2b0a78e 2020-08-31 02:00:21 dee systemd.timer /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

... # removed for brevity

ec6fd77e 2021-05-03 02:00:34 dee systemd.timer /etc/pihole
                                                          /mnt/usb/Backup/lms
                                                          /var/lib/unifi/backup/autobackup

722fcbbf 2021-05-04 02:00:34 dee systemd.timer /etc/pihole
                                                          /mnt/usb/Backup/lms
                                                          /var/lib/unifi/backup/autobackup
-----------------------------------------------------------------------------------------------
39 snapshots

Syncing to B2 with rclone

Restic has now created snapshots and stored them in their respective repositories on the USB drive - but this isn’t really a safe place to keep backups since the USB drive could crap out at any moment. We can use rclone to offload the repositories to B2.

For this I’m going to use the same pattern as before for restic; a shell script invoked by systemd.

# rclone-all.sh

#!/usr/bin/env bash

# Master script for backing up all rclone stuff to various clouds

RCLONE_CONFIG=/home/jdheyburn/.config/rclone/rclone.conf

function main() {

    echo "rcloning beets-db -> gdrive:media/beets-db"
    rclone -v sync /mnt/usb/Backup/media/beets-db gdrive:media/beets-db --config=${RCLONE_CONFIG}

    echo "rcloning music -> gdrive:media/music"
    rclone -v sync /mnt/usb/Backup/media/music gdrive:media/music --config=${RCLONE_CONFIG}

    echo "rcloning lossless -> gdrive:media/lossless"
    rclone -v sync /mnt/usb/Backup/media/lossless gdrive:media/lossless --config=${RCLONE_CONFIG}

    echo "rcloning vinyl -> gdrive:media/vinyl"
    rclone -v sync /mnt/usb/Backup/media/vinyl gdrive:media/vinyl --config=${RCLONE_CONFIG}

    echo "rcloning restic -> b2:restic"
    rclone -v sync /mnt/usb/Backup/restic b2:iifu8Noi-backups/restic/ --config=${RCLONE_CONFIG}

    echo "Done rcloning"
}

main $@

In addition to the restic repository directory, I’m also backing up the beets databases and music files. These are going to my Google Drive storage in case I want to hook it up to some other apps that can pull from there.

Like restic-all.sh above, this script is also invoked by systemd.

# /etc/systemd/system/rclone-all.service

# This should be invoked after restic has done doing the daily backup

[Unit]
Description=Rclone backup everything service
After=restic-all.service
OnFailure=unit-status-mail@%n.service

[Service]
Type=oneshot
ExecStart=/home/jdheyburn/dotfiles/restic/rclone-all.sh

[Install]
WantedBy=restic-all.service

We can test it with systemctl start rclone-all.service.

Since we want to have rclone invoked after restic has done its thing, we need to specify After=restic-all.service and WantedBy=restic-all.service. Note that this’ll run even if restic-all.service failed - but that’s not really an issue for me since rclone is uploading more than just restic respositories.

We don’t need to specify a systemd .timer file for this unit file since we’re using the completion of restic-all.service as our invocation point, so we can start that service in order to test it’ll run afterward.

systemctl start restic-all.service

Handling service failures

N.B. big thanks to Laeffe for their excellent guide on this.

Automated backups are very useful for setting and forgetting, but when something goes wrong and backups haven’t occurred, we need to be made aware of it.

We can configure each of the systemd files to invoke a script on failure which can then send us an email when this happens.

You’ll notice that each of the systemd unit files have OnFailure=unit-status-mail@%n.service defined in them. This is an additional unit file where if the scripts fail, this service will be invoked.

# /etc/systemd/system/unit-status-mail@.service

[Unit]
Description=Unit Status Mailer Service
After=network.target

[Service]
Type=simple
ExecStart=/home/jdheyburn/dotfiles/restic/systemd/unit-status-mail.sh %I "Hostname: %H" "Machine ID: %m" "Boot ID: %b"

The @ symbol in the filename is a special case within systemd that enables special functionality. In our case when we invoked it at OnFailure=unit-status-mail@%n.service the calling unit file will pass its name via %n to allow the receiving unit file to access it at %I. So when the restic-all service fails, this value ends up in unit-status-mail@.service at %I.

This %I variable is then being passed as an argument to the script unit-status-mail.sh. The contents of the script are below:

# unit-status-mail.sh

#!/bin/bash

# From https://northernlightlabs.se/2014-07-05/systemd-status-mail-on-unit-failure.html

MAILTO="ADD_EMAIL_HERE"
MAILFROM="unit-status-mailer"
UNIT=$1

EXTRA=""
for e in "${@:2}"; do
  EXTRA+="$e"$'\n'
done

UNITSTATUS=$(systemctl status $UNIT)

sendmail $MAILTO <<EOF
From:$MAILFROM
To:$MAILTO
Subject:Status mail for unit: $UNIT

Status report for unit: $UNIT
$EXTRA

$UNITSTATUS
EOF

echo -e "Status mail sent to: $MAILTO for unit: $UNIT"

The name of the calling unit file is passed to UNIT where the status of it is received, and the output is sent in an email to MAILTO.

You’ll need to make sure you have sendmail installed on the machine to do this.

sudo apt install sendmail -y

For me, the emails landed in the spam folder - but once you configure a rule on your email client to always forward them to your inbox, you’ll always be notified if there’s an error.

The resulting email

Removing aged restic snapshots

One last area to look at with restic is pruning, this is where restic will remove old data that has been “forgotten”.

Back in restic-all.sh we are calling restic forget after each backup - this tells restic to clean up any old snapshots not required by our defined backup policy. The script has been written to accommodate for both backup and prune functionality, so in order to execute that portion of the script we need to set up another systemd unit file to invoke it.

# /etc/systemd/system/restic-prune.service

[Unit]
Description=Restic backup service (data pruning)
OnFailure=unit-status-mail@%n.service

[Service]
Type=oneshot
ExecStart=/home/jdheyburn/dotfiles/restic/restic-all.sh prune

Since it is resource intensive to perform prune against your repository, its best to run this at a different interval to your backups. From the below unit file OnCalendar=*-*-1 10:00:00 corresponds to the first day of the month at 10am.

[Unit]
Description=Prune data from the restic repository monthly

[Timer]
OnCalendar=*-*-1 10:00:00
Persistent=true

[Install]
WantedBy=timers.target

Restoring a backup

Now the most important part - how to restore a restic snapshot. Let’s start with how to restore from a local repository.

Restore from local restic repository

Firstly we need to find the snapshot ID that we want to restore to - in this example I want the latest snapshot.

$ restic -r /mnt/usb/Backup/restic/small-files/ snapshots --last
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
ID Time Host Tags Paths
-----------------------------------------------------------------------------------------------
f7ef9c33 2021-04-17 02:00:51 dee systemd.timer /etc/pihole
                                                          /mnt/usb/Backup/media/beets-db
                                                          /var/lib/unifi/backup/autobackup

e2a73c00 2021-04-21 02:00:21 dee systemd.timer /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

722fcbbf 2021-05-04 02:00:34 dee systemd.timer /etc/pihole
                                                          /mnt/usb/Backup/lms
                                                          /var/lib/unifi/backup/autobackup
-----------------------------------------------------------------------------------------------
3 snapshots

So the ID is 722fcbbf, let’s browse the contents to see if the file we want is in there.

$ restic -r /mnt/usb/Backup/restic/small-files/ ls 722fcbbf
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
snapshot 722fcbbf of [/var/lib/unifi/backup/autobackup /etc/pihole /mnt/usb/Backup/lms] filtered by [] at 2021-05-04 02:00:34.383403601 +0100 BST):
/etc
/etc/pihole
/etc/pihole/GitHubVersions
/etc/pihole/adlists.list
# ... removed for brevity

Let’s say we want to restore /etc/pihole/adlists.list, we can use the --include argument to specify just that. If we didn’t use a filter argument then restic would default to restoring the entire contents of the snapshot.

$ restic -r /mnt/usb/Backup/restic/small-files/ restore 722fcbbf --target /tmp/restic-restore --include /etc/pihole/adlists.list
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
restoring <Snapshot 722fcbbf of [/var/lib/unifi/backup/autobackup /etc/pihole /mnt/usb/Backup/lms] at 2021-05-04 02:00:34.383403601 +0100 BST by root@dee> to /tmp/restic-restore

$ tree /tmp/restic-restore/
/tmp/restic-restore/
└── etc
    └── pihole
        └── adlists.list

Restore from B2 restic repository

We’re storing the repositories on B2 too, and restic has B2 integration built into it. So in the scenario where the NFS server had died and we needed to restore a snapshot stored on B2, we can hit it directly without having to use rclone to pull down the entire repository for us to restore from. This’ll be a cheaper approach as restic is only pulling down the files it needs to restore from, lowering B2 download costs.

We just need to configure some variables to permit restic to hit B2. If you’re using rclone you can use the same ID and key here.

export B2_ACCOUNT_ID="ACCCOUNT_ID"
export B2_ACCOUNT_KEY="ACCCOUNT_KEY"
export RESTIC_PASSWORD_FILE="/path/to/passwordfile
restic -r b2:BUCKET_NAME:restic/small-files snapshots --last

From here you can then use the same commands as in the local repository to traverse the snapshots and restore.

Conclusion

The process above is probably more complex than what it needs to be; having a separate process for backing up and restoring. Since I want to back up to the NFS locally first followed by B2, this approach made the most sense as it allows me to minimise download costs from B2 through targeting the local repository first.

What’s most important is that backups are being made, and that I can restore from them.

How to automate zero downtime maintenance with AWS SSM & ALBs

Joseph Heyburn — Tue, 19 Jan 2021 00:00:00 +0000

Welcome to the last post in this series where we've been exploring SSM Documents, so far we've covered:

How Command Documents can help to execute commands on EC2 Instances
Automating these Command Documents through Maintenance Windows
Safely chaining Command Documents through Automation Documents, and aborting for any failures

This post will now look into how we can use Automation Documents to perform maintenance on EC2 instances without impacting user experience.

tl;dr

With the introduction of load balancers to front your services, you can control which instances should be receiving traffic
This enables you to proactively remove instances from rotation so that you can perform maintenance on the backends to minimalise user disruption
SSM automation documents can enable us to execute pre-maintenance steps such as removing an instance from a load balancer, as well as adding them back after
- See the document produced in this post highlighting this, and where I explain how it works, and how to deploy it using Terraform

Prerequisites

If you're just joining in from this post then I recommend reading through the previous posts to gain an understanding of how we got here; or if you know what you're looking for the tl;dr provides a summary.

As always, the code for this post can be found on GitHub.

A basic understanding and knowledge of Application Load Balancers (ALB), and its components (e.g. target groups) is required.

Introducing load balancers

Load balancers are a key component in software architecture that distribute traffic and requests across backend services in a variety of algorithms, such as:

Round-robin
- every backend serves the same number of requests
- the most commonly used algorithm
Weighted round-robin
- backends receive a fixed percentage of incoming requests
- useful if some backends are more beefy than others
- also used in canary deployments
Least outstanding requests
- the backend which is currently processing the least number of requests is forwarded the request

There are multiple benefits to having a load balancer sit in front of your services:

Protecting your infrastructure; user requests are proxied via the load balancer
Distribute traffic and requests however you like
Perform healthchecks on backends and don't forward traffic to unhealthy nodes
Drain and remove backends to permit for rolling upgrades

While there are several different software-based load balancers out there such as nginx and HAProxy, AWS has its own managed load balancer service known as an Elastic Load Balancer (ELB).

Adding web services to our demo environment

In order for us to get the benefit of load balancers to front the EC2 instances in the architecture this series left off from last time, we will need to have a service running on our instances.

Let's simulate a real web service by running a simple Hello World application across each of the instances. We can utilise EC2s user data to start a basic service up for us by giving it a script to run on instance provision.

This is the architecture we'll be building out in this section, with an ALB fronting our EC2 instances

Let's use Go to create a web service for us since it is easy to get set up quickly - we'll have the server return a simple Hello, World! message when a request hits it. Let's also have it return the name of the instance that was hit - this will be used later.

#!/bin/bash
cat <<'EOF' > /home/ec2-user/main.go
package main

import (
        "fmt"
        "net/http"
        "os"
)

func main() {
        http.HandleFunc("/", HelloServer)
        http.ListenAndServe(":8080", nil)
}

func HelloServer(w http.ResponseWriter, r *http.Request) {
        hostname, _ := os.Hostname()
        fmt.Fprintf(w, "Hello, World! From %v\n", hostname)
}
EOF
yum install golang -y
(crontab -l 2>/dev/null; echo "@reboot nohup go run /home/ec2-user/main.go") | crontab -
export GOCACHE=/tmp/go-cache
nohup go run /home/ec2-user/main.go

So on instance creation this will:

Create a new file called main.go and populate it with the lines between the EOF delimiters
Install Go
Create a crontab entry to run the service on subsequent boots
Run the Go application in the background immediately

While user data is for useful provisioning services, I don't advise storing the source code of your application in there like I've done - this is just a hacky way to get something up and running.

We've been using Terraform to provision our nodes. So we need to save this script in a file (scripts/hello_world_user_data.sh), and then pass it into the user_data attribute of our EC2 module.

module "hello_world_ec2" {
  source = "terraform-aws-modules/ec2-instance/aws"
  version = "~> 2.0"

  # ... removed for brevity

  user_data = file("scripts/hello_world_user_data.sh")
}

Terraform will recreate any EC2 nodes with a change in user_data contents, so when you invoke terraform apply all instances will be recreated.

N.B. the AMI I picked up for my EC2 instances has an issue with SSM Agent. Ensure you execute the AWS-UpdateSSMAgent across your instances after they have provisioned, or you can use an SSM Association document to do that for you as shown here.

After they have all successfully deployed, you should be able to curl the public IP address of each instance from your machine to verify your setup is correct. If you are getting timeouts then make sure your instances have a security group rule permitting traffic from your IP address through port 8080.

$ curl http://54.229.209.60:8080
Hello, World! From ip-172-31-39-169.eu-west-1.compute.internal

$ curl http://3.250.160.209:8080
Hello, World! From ip-172-31-21-197.eu-west-1.compute.internal

$ curl http://34.254.238.146:8080
Hello, World! From ip-172-31-8-52.eu-west-1.compute.internal

Fronting instances with a load balancer

Now that we have a web service hosted on our instances, let's now add a load balancer in front of it. This load balancer will now become the point of entry for our application instead of hitting the EC2 instances directly.

For this I am using a Terraform ALB module for provisioning all the components in the load balancer, and expanding on them is beyond the scope of this post.

You can navigate to the AWS ALB documentation to find out about the underlying components the module creates for us.

Security groups

Before we can provision the load balancer, we need to specify the security group (SG) and the rules that should be applied to it. You can view this on GitHub.

Since our application is written to serve request on port 8080, we need to permit both the new aws_security_group.hello_world_alb SG and the existing aws_security_group.vm_base SG to communicate between each other.

resource "aws_security_group" "hello_world_alb" {
  name = "HelloWorldALB"
  vpc_id = data.aws_vpc.default.id
}

resource "aws_security_group_rule" "alb_egress_ec2" {
  security_group_id = aws_security_group.hello_world_alb.id
  type = "egress"
  from_port = 8080
  to_port = 8080
  protocol = "tcp"
  source_security_group_id = aws_security_group.vm_base.id
}

resource "aws_security_group_rule" "ec2_ingress_alb" {
  security_group_id = aws_security_group.vm_base.id
  type = "ingress"
  from_port = 8080
  to_port = 8080
  protocol = "tcp"
  source_security_group_id = aws_security_group.hello_world_alb.id
}

Now we need to open up the ALB to allow traffic to hit it. In our case it will just be us hitting it, but this will change depending on who the consumer of the service is. If it is to serve traffic from the Internet then cidr_blocks would be ["0.0.0.0/0"].

The ALB will be hosting the traffic on insecure HTTP (port 80).

resource "aws_security_group_rule" "alb_ingress_user" {
  security_group_id = aws_security_group.hello_world_alb.id
  type = "ingress"
  from_port = 80
  to_port = 80
  protocol = "tcp"
  cidr_blocks = [local.ip_address]
}

ALB module

The module documentation will tell us how we need to structure it. Our requirements dictate we need the following:

Receive traffic on port 80
Forward traffic to backend targets on port 8080
Include health checks to ensure we do not forward requests to unhealthy instances

Translate these requirements into the context of the module and we have something like this:

module "hello_world_alb" {
  source = "terraform-aws-modules/alb/aws"
  version = "~> 5.0"

  name = "HelloWorldALB"

  load_balancer_type = "application"

  vpc_id = data.aws_vpc.default.id
  subnets = tolist(data.aws_subnet_ids.all.ids)
  security_groups = [aws_security_group.hello_world_alb.id]

  target_groups = [
    {
      name_prefix = "pref-"
      backend_protocol = "HTTP"
      backend_port = 8080
      target_type = "instance"
      health_check = {
        enabled = true
        healthy_threshold = 2
        unhealthy_threshold = 2
        interval = 6
      }
    }
  ]

  http_tcp_listeners = [
    {
      port = 80
      protocol = "HTTP"
      target_group_index = 0
    }
  ]
}

In order to keep this post simple I am not fronting services over HTTPS (secure HTTP) - I would strongly advise against doing this for non-test scenarios.

Another step we will need is to hook up our EC2 instances with the target group created.

resource "aws_lb_target_group_attachment" "hello_world_tg_att" {
  count = length(module.hello_world_ec2.id)
  target_group_arn = module.hello_world_alb.target_group_arns[0]
  target_id = element(module.hello_world_ec2.id, count.index)
  port = 8080
}

There's some clever Terraform going on here. All we're doing is looping over each of the created EC2 instances in the module and adding it to the target group, to receive traffic on port 8080.

Hitting the load balancer

Whereas earlier when we were testing the services by hitting the EC2 instances directly, we'll now be hitting the ALB instead. You can grab the ALB DNS name from the console.

$ curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80
Hello, World! From ip-172-31-31-223.eu-west-1.compute.internal

Nice - and we can see from here what instance we've hit in the backend, since we included the hostname in the response.

If we now hit the ALB one more time, we will get a different instance respond.

$ curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80
Hello, World! From ip-172-31-0-10.eu-west-1.compute.internal

This is the load balancer rotating between the backends available to it. We can see the rotation by repeatedly hitting the endpoint.

$ while true; do curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80 ; sleep 0.5; done
Hello, World! From ip-172-31-31-223.eu-west-1.compute.internal
Hello, World! From ip-172-31-43-11.eu-west-1.compute.internal
Hello, World! From ip-172-31-0-10.eu-west-1.compute.internal
Hello, World! From ip-172-31-31-223.eu-west-1.compute.internal
Hello, World! From ip-172-31-0-10.eu-west-1.compute.internal
Hello, World! From ip-172-31-43-11.eu-west-1.compute.internal
Hello, World! From ip-172-31-43-11.eu-west-1.compute.internal

So what does all this have to do with our maintenance document?

Now that we have a load balancer fronting our services, let's review executing our automation document from the last post and the problem it brings.

If an instance were to be rebooted during the AWS-RunPatchBaseline stage of the automation document, then there is a chance that a request would have been forwarded to that instance before the health checks against it have failed.

To simulate this, let's create a new automation document which simulates a reboot. We'll just take our existing patching document and replace the patch step with a reboot command, following the AWS guidelines to do this. I've also modified the health check to check to see if the new service came up okay - this may differ for your environment.

You can view the document on GitHub.

--------
schemaVersion: "0.3"
description: Executes a reboot on the instance followed by a healthcheck
parameters:
  InstanceIds:
    type: StringList
    description: The instance to target
mainSteps:
  - name: InvokeReboot
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}
      Parameters:
        commands: |
          flag_location=/home/ec2-user/REBOOT_STARTED
          if [! -f $flag_location]; then
            echo "Creating flag file at $flag_location"
            touch $flag_location
            echo "Reboot initiated"
            exit 194
          fi
          echo "Reboot finished, removing flag file at $flag_location"
          rm $flag_location          
  - name: ExecuteHealthcheck
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}
      Parameters:
        commands: |
          sleep 60
          if ! curl http://localhost:8080/; then
            exit 1
          fi

Then the Terraform code to create this document will look like this:

resource "aws_ssm_document" "reboot_with_healthcheck" {
  name = "RebootWithHealthcheck"
  document_type = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/reboot_with_healthcheck_template.yml",
    {
      output_s3_bucket_name = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix = "ssm_output/",
    }
  )
}

Testing for failure

After this has been applied in our environment let's get our test set up. In a terminal window from your machine have this script running in the background to simulate load.

$ while true;
do
  resp=$(curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80 2>/dev/null)
  if echo $resp | grep -q html; then
    error=$(echo $resp | grep -oPm1 "(?<=<title>)[^<]+")
    echo "Error - $error"
  else
    echo $resp;
  fi
  sleep 0.5
done

Then we need to invoke the new reboot document against an instance (see here for how we achieved this last time). Once it is running let's monitor the output of the command in your terminal.

Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Error - 502 Bad Gateway
Error - 502 Bad Gateway
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Error - 502 Bad Gateway
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Error - 502 Bad Gateway
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Error - 502 Bad Gateway
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal

Not good! 😧

These error messages are coming from the load balancer, indicating that the underlying backend wasn't able to complete the request. This is happening when the instance gets rebooted as specified in our document.

The load balancer hasn't had enough time to determine whether the instance is unhealthy or not - as dictated from our health check policy (2 failed checks with 6 seconds between them) - and so still forwards traffic to it even though it cannot respond.

We can actually view this disruption in CloudWatch Metrics too. ALBs expose metrics for us which we can monitor against, including:

RequestCount
- Informs us how many incoming requests the ALB is receiving
HTTPCode_ELB_5XX_Count
- How many HTTP 5XX error codes are being returned by the ALB

We can chart them together for visualisation.

HTTPCode_ELB_5XX_Count only reports on failures - if the metric is missing data points then no errors occurred at that time

While we are the only users hitting this, had this been a production box hit by 1,000s of users, each one of them would experience an issue with your application, and equals lost customers! 😡

What we need is a means of removing the node from the load balancer rotation so that we can safely perform maintenance on it.

Removing instances from load balancer rotation

Load balancer target groups have an API endpoint that allow you to drain connections from backends - where the load balancer stops any new requests being forwarded to that backend, and allows existing requests to complete. This can be done via - you guessed it - Automation Documents!

Graceful load balancer document

You can see the document in its entirety here. The steps that it performs are:

Check that the target group is healthy
- We want to ensure we're not fuelling a dumpster fire
Check that the instance we're targeting is in the target group we're modifying
- Otherwise what's the point? 🙃
Remove the instance from the target group and wait for it to be removed
Execute our maintenance document
Register the instance back and wait for it to be added back

Document metadata

As this is an automation document, the schemaVersion should be 0.3. We're using two parameters here to run the document:

TargetGroupArn - the ARN of the target group we are making modifications too
InstanceId - the instance that is undergoing maintenance

schemaVersion: "0.3"
description: Gracefully reboot instance with healthchecks
parameters:
  InstanceId:
    type: String
    description: The instance to target
  TargetGroupArn:
    type: String
    description: The target group ARN for the instance

Sanity checks

The first two steps of the document are sanity checking the target group to ensure preconditions are met before we introduce change. We're using the aws:assertAwsResourceProperty action to allow us to query against the AWS ELBv2 API (Elastic Load Balancer v2) and verify a response is what we expect it to be.

You can see all the API endpoints available for ELBv2 at this location.

The response of the DescribeTargetHealth endpoint returns an object that is structured like this.

{
  "TargetHealthDescriptions": [
    {
      "Target": {
        "Id": "i-083c8ca9c9b74e1cd",
        "Port": 8080
      },
      "HealthCheckPort": "8080",
      "TargetHealth": {
        "State": "healthy"
      }
    },
    {
      "Target": {
        "Id": "i-08a63b118a0b2a6b7",
        "Port": 8080
      },
      "HealthCheckPort": "8080",
      "TargetHealth": {
        "State": "healthy"
      }
    },
    {
      "Target": {
        "Id": "i-08c656b7160dd6729",
        "Port": 8080
      },
      "HealthCheckPort": "8080",
      "TargetHealth": {
        "State": "healthy"
      }
    }
  ]
}

This represents all the backends that are configured against the target group. The property selector $.TargetHealthDescriptions..TargetHealth.State specified in the document will check against all state fields to see if they are healthy. If any of them aren't then the document will be aborted. As mentioned before, this check is performed to ensure we're not causing more problems for ourselves if any of the nodes are unhealthy.

- name: AssertTargetGroupHealthBefore
  description: Assert the target group is healthy before we bounce Tomcat
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions..TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
  maxAttempts: 3
  timeoutSeconds: 60

In the next sanity check we're ensuring that the instance is definitely in the target group we want to remove it from. This is a slightly different query to the last step where we're specifically requesting for state health on that one instance.

- name: AssertInstanceIsInTargetGroup
  description: Assert the instance is a healthy target of the target group
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions[0].TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"
  maxAttempts: 3
  timeoutSeconds: 60

Remove the instance from rotation

Now our preconditions have been met we can remove the instance using the aws:executeAwsApi action. This action is similar to aws:assertAwsResourceProperty in that it calls an AWS API endpoint, but we're not checking the response of it - in fact the DeregisterTargets endpoint doesn't return anything for us to check against.

- name: DeregisterInstanceFromTargetGroup
    description: Proactively remove the instance from the target group
    action: aws:executeAwsApi
    inputs:
      Service: elbv2
      Api: DeregisterTargets
      TargetGroupArn: "{{ TargetGroupArn }}"
      Targets:
        - Id: "{{ InstanceId }}"

Once we've done that we need to verify the instance has definitely been removed. Remember that the target group allows for existing connections to complete their requests when it is draining, so deregistering the instance doesn't happen instantaneously - this is where the aws:waitForAwsResourceProperty helps us.

- name: WaitForDeregisteredTarget
  description: Wait for the instance to drain connections
  action: aws:waitForAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions[0].TargetHealth.State
    DesiredValues:
      - unused
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"
  maxAttempts: 1
  timeoutSeconds: 600
- name: AssertTargetIsDeregistered
  description: Assert the instance is no longer a target
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions[0].TargetHealth.State
    DesiredValues:
      - unused
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"

Execute maintenance

At this point we're 100% sure that the instance is now removed from the target group and is no longer receiving requests, so let's go ahead and use aws:executeAutomation to invoke the maintenance document from earlier. Remember it takes in the InstanceIds as a parameter to execute on, so we'll need to pass it there too.

We're specifying an onFailure too, this tells the document should the step fail then move onto this step instead of the default action which is to abort the rest of the document.

step:RegisterTarget is actually the next step after this one, which adds the instance back to the target group. Since it performs health checks for us and won't actually forward to an unhealthy instance, we'll let the target group make the call if this instance can receive traffic or not.

- name: RebootWithHealthcheck
  description: Reboot the instance with a healthcheck afterward
  action: aws:executeAutomation
  inputs:
    DocumentName: RebootWithHealthcheck
    RuntimeParameters:
      InstanceIds:
        - "{{ InstanceId }}"
  maxAttempts: 1
  onFailure: step:RegisterTarget

Add instance back to target group

In a similar vein to DeregisterTargets, this action will register the instance back to the target group.

- name: RegisterTarget
  description: Add the instance back as a target
  action: aws:executeAwsApi
  inputs:
    Service: elbv2
    Api: RegisterTargets
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"

Registering the instance happens instantaneously, but we will have to wait for the target group to perform initial health checks against the instance. Once we've asserted that it's healthy, then the document is complete!

- name: WaitForHealthyTargetGroup
  description: Wait for the target group to become healthy again
  action: aws:waitForAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions..TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
  maxAttempts: 1
- name: AssertTargetGroupHealthAfter
  description: Assert the target group is healthy after activity
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions..TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
  maxAttempts: 3
  timeoutSeconds: 60
  isEnd: true

Remember that this document only handles one instance at a time, it will typically be up to the caller (i.e. a maintenance window) to rate limit the execution of multiple instances one at a time. We explored this in the previous post.

Terraform additions and updates

The above document can be represented in Terraform to provision it.

resource "aws_ssm_document" "graceful_reboot_instance" {
  name = "RebootInstanceGraceful"
  document_type = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/graceful_patch_instance.yml",
    {
      reboot_with_healthcheck_document_arn = aws_ssm_document.reboot_with_healthcheck.arn,
    }
  )
}

We'll also need to update our maintenance window task to correctly reflect this new document, along with the new parameters it takes.

resource "aws_ssm_maintenance_window_task" "patch_with_healthcheck" {
  window_id = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type = "AUTOMATION"
  task_arn = aws_ssm_document.graceful_reboot_instance.arn
  priority = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "1"
  max_errors = "0"

  targets {
    key = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    automation_parameters {
      document_version = "$LATEST"

      parameter {
        name = "InstanceId"
        values = ["{{ TARGET_ID }}"]
      }

      parameter {
        name = "TargetGroupArn"
        values = [module.hello_world_alb.target_group_arns[0]]
      }
    }
  }
}

And lastly, we'll need to update the IAM role permissions for aws_iam_role.patch_mw_role.arn as it will be invoking more actions.

data "aws_iam_policy_document" "mw_role_additional" {
  statement {
    sid = "AllowSSM"
    effect = "Allow"

    actions = [
      "ssm:DescribeInstanceInformation",
      "ssm:ListCommandInvocations",
    ]

    resources = ["*"]
  }

  statement {
    sid = "AllowElBRead"
    effect = "Allow"

    actions = [
      "elasticloadbalancing:DescribeTargetHealth",
    ]

    resources = ["*"]

  }

  statement {
    sid = "AllowELBWrite"
    effect = "Allow"

    actions = [
      "elasticloadbalancing:DeregisterTargets",
      "elasticloadbalancing:RegisterTargets",
    ]

    resources = [module.hello_world_alb.target_group_arns[0]]
  }
}

Testing the new document

You can test the automation document by following the same process as before, else you can test the whole stack via changing the execution time of the maintenance window. I'll be following along with the latter.

While the document is running you can re-use the same command to hit the ALB endpoint from earlier to see how traffic is distributed amongst the instances. You'll first see that it will only execute on one instance at a time, which was the enhancement we introduced in the last post.

When we drill down into each invocation we can see the automation steps doing their magic.

We can have a look back at the ALB metrics again to see if we received any errors.

No error - no problem!

Conclusion

Each post prior to this one in the series has been leading up to where we are now - a means of achieving automated zero downtime maintenance for anything that sits behind an AWS ALB.

SSM is very much a bit of a beast and I hope that this series has helped clear the fog and given yourselves an idea of what you can do with SSM to automate a variety of tasks in your AWS estate.

Happy automating! 💪 🙌

Hugo Minify RSS Code Indentation Fix

Joseph Heyburn — Mon, 14 Dec 2020 00:00:00 +0000

It's certainly been a while since the previous post in this series, which has become the home of any updates I make to my Hugo website.

This post is a quick one but it's something I've been meaning to fix for a while. The RSS feeds that are generated will also include code blocks as defined through code fences or through {{< highlight >}} shortcodes in your post content.

However for some reason the code blocks generated in my RSS feeds were losing their indentation. Take this code snippet example from my latest post.

resource "aws_ssm_document" "patch_with_healthcheck" {
name = "PatchWithHealthcheck"
document_type = "Automation"
document_format = "YAML"
content = templatefile(
"documents/patch_with_healthcheck_template.yml",
{
healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
output_s3_bucket_name = aws_s3_bucket.script_bucket.id,
output_s3_key_prefix = "ssm_output/",
}
)
}

Whereas it should be rendering as:

resource "aws_ssm_document" "patch_with_healthcheck" {
  name = "PatchWithHealthcheck"
  document_type = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/patch_with_healthcheck_template.yml",
    {
      healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
      output_s3_bucket_name = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix = "ssm_output/",
    }
  )
}

This is happening during the GitHub Action that builds the website - I am using the --minify parameter which follows this configuration by default, and the highlighted line.

[minify]
  disableCSS = false
  disableHTML = false
  disableJS = false
  disableJSON = false
  disableSVG = false
  disableXML = false
  minifyOutput = false
  [minify.tdewolff]
    [minify.tdewolff.css]
      decimals = -1
      keepCSS2 = true
    [minify.tdewolff.html]
      keepConditionalComments = true
      keepDefaultAttrVals = true
      keepDocumentTags = true
      keepEndTags = true
      keepQuotes = false
      keepWhitespace = false
    [minify.tdewolff.js]
    [minify.tdewolff.json]
    [minify.tdewolff.svg]
      decimals = -1
    [minify.tdewolff.xml]
      keepWhitespace = false

So what's happening is the RSS XML that Hugo generates for us is then being minified to remove the whitespace generated in that file. This includes the whitespace that's used to indent the code in the RSS feeds. Not. Good.

The Fix

In order to fix this we just need to copy the default minify config as mentioned above and change the last line to keepWhitespace = true.

You can view my git commit that fixes this - should you want to include the fix too.

I'm using toml for my config, so make sure you change it to what your config is defined in (yaml / json)

I tried to experiment to see if the whole config was required or not, but it appears so - if you found a way to omit some lines then let me know.

That's it - hope this help you with fixing your RSS feeds too!

Automate Instance Hygiene with AWS SSM: Automation Documents

Joseph Heyburn — Fri, 04 Dec 2020 00:00:00 +0000

In part two of this series we looked at how we can automate SSM command documents using SSM Maintenance Windows.

This part will now explore another type of SSM Document; Automation.

tl;dr

Automation documents can allow us to combine command documents together
With this, we can utilise maintenance window error thresholds to stop further invocations
Dynamic invocation of command documents can also be achieved with automation documents

Prerequisites

All the code for this post can be found on GitHub.

You'll notice that we have changed some things around in this post, so if you've been using terraform apply in other posts to deploy to your AWS environment, you will notice some destructions.

Instead of having 1 Windows and 1 Linux EC2 instance, we're now using 3 Linux EC2 instances - to emulate an application running across multiple instances for redundancy. You don't actually need anything to be running on these instances, just have them visible in the Managed Instances console.

Note that the EC2 instances are tagged with the key App and value HelloWorld - we'll be using this to specify our automation document targets.

Automation Documents

Back in part one I gave a brief intro to automation documents. To save the click:

Automation document can call and orchestrate AWS API endpoints on your behalf, including executing Command documents on instances

Essentially we can combine two command documents into one with an automation document. But why would we want to do this?

Introducing proactive healthchecks

Well in the last post, we set up a maintenance window with two tasks; one for invoking AWS-RunPatchBaseline and another for PerformHealthcheckS3 (our healthcheck SSM Document) - both of these are command documents. Say if we had a policy that wanted to ensure that after AWS-RunPatchBaseline was invoked, we would always want the PerformHealthcheckS3 invoked afterward… the Automation Document would help us get there.

Not only that, the way that our maintenance window is currently structured is it will invoke AWS-RunPatchBaseline across all instances in scope at the same time. Once they are all done then it will invoke PerformHealthcheckS3 across all instances at the same time. This looks like this:

Given we have 2 instances; i-111, i-222
T+0: Invoke AWS-RunPatchBaseline on i-111, i-222
T+1: AWS-RunPatchBaseline finishes: i-111, i-222
T+2: Invoke PerformHealthcheckS3 on i-111, i-222
T+3: PerformHealthcheckS3 finishes: i-111, i-222

Say if you wanted to limit the rate of patching across these instances so that only one instance at a time was patched, and any healthcheck failures aborted the rest of patching, then simply changing max_concurrency from 100% to 1 for each maintenance window task will not achieve this.

Maintenance windows complete one task across all instances in scope before moving onto the next task. If we have Task 1 for Patching and Task 2 for Healthchecking (is that even a word?), then the maintenance window is going to patch all instances first before it performs healthchecks on the instances. There is no way to execute the tasks synchronously on one instance at a time.

This means that if a bad patch were to be installed in your estate, you could have this order of events:

Given we have 2 instances; i-111, i-222
T+0: Invoke AWS-RunPatchBaseline on i-111
T+1: AWS-RunPatchBaseline finishes: i-111
T+2: Invoke AWS-RunPatchBaseline on i-222
T+3: AWS-RunPatchBaseline finishes: i-222
T+4: Invoke PerformHealthcheckS3 on i-111
T+5: PerformHealthcheckS3 FAILS: i-111
T+6: Invoke PerformHealthcheckS3 on i-222
T+7: PerformHealthcheckS3 FAILS: i-222

Now the healthcheck has failed for i-222 as well because the bad patch landed on both instances, and production now has an outage.

Solution

Thankfully, Automation Documents help us avoid that - by combining the two command documents (AWS-RunPatchBaseline and PerformHealthcheckS3), we can mark this new automation document as a solo maintenance window task and have it invoked one at a time on instances, and have it abort further invocations if any sub-documents failed within in:

Given we have 2 instances; i-111, i-222
T+0: Invoke AWS-RunPatchBaseline on i-111
T+1: AWS-RunPatchBaseline finishes: i-111
T+2: Invoke PerformHealthcheckS3 on i-111
T+3: PerformHealthcheckS3 FAILS: i-111
T+4: Abort invoking AWS-RunPatchBaseline on i-222
T+5: Abort invoking PerformHealthcheckS3 on i-222

Notice how the failed healthcheck on the first instance caused the rest of task invocations to abort? Here is the order of events for a happy path.

Given we have 2 instances; i-111, i-222
T+0: Invoke AWS-RunPatchBaseline on i-111
T+1: AWS-RunPatchBaseline finishes: i-111
T+2: Invoke PerformHealthcheckS3 on i-111
T+3: PerformHealthcheckS3 succeeds: i-111
T+4: Invoke AWS-RunPatchBaseline on i-222
T+5: AWS-RunPatchBaseline finishes: i-222
T+6: Invoke PerformHealthcheckS3 on i-222
T+7: PerformHealthcheckS3 succeeds: i-222

So we know that automation documents allow us to combine command documents together - this is done using the aws:runCommand action, though there are many more actions available to you, some of which we'll explore later.

Combining command docs into automation

We combine command documents as below in YAML; like command documents they can also be defined in JSON.

For the first time we are defining a parameter called InstanceIds, which takes in a list of instance IDs to then pass down to the command documents, as they will need to know what instances to invoke the commands on. The value assigned to this parameter is retrieved back with the notation "{{ InstanceIds }}", which you can see being passed into the inputs of the sub-documents.

InstanceIds is a special parameter name that AWS recognises and so will provide an instance picker in the GUI of Execute Automation, as shown below.

The instance picker can be helpful if you only want to target a subset of instances in your estate.

We're also following logging best practices by having the command output logged to S3.

Other than that, there's not really a whole lot of difference between this and a command document, so far!

It's important to note the schemaVersion must be 0.3 for automation documents.

---
schemaVersion: "0.3"
description: Executes a patching event on the instance followed by a healthcheck
parameters:
  InstanceIds:
    type: StringList
    description: The instance to target
mainSteps:
  - name: InvokePatchEvent
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunPatchBaseline
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: jdheyburn-scripts
      OutputS3KeyPrefix: ssm_output/
      Parameters:
        Operation: Scan
  - name: ExecuteHealthcheck
    action: aws:runCommand
    inputs:
      DocumentName: PerformHealthcheckS3
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: jdheyburn-scripts
      OutputS3KeyPrefix: ssm_output/

Terraform automation documents

We can deploy these to AWS using Terraform once again. Note that the document_type is Automation and that we're using templating to set the variables in the document, such as referencing the PerformHealthcheckS3 command document ARN.

You can see the templated version of the document in GitHub.

resource "aws_ssm_document" "patch_with_healthcheck" {
  name            = "PatchWithHealthcheck"
  document_type   = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/patch_with_healthcheck_template.yml",
    {
      healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
      output_s3_bucket_name    = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix     = "ssm_output/",
    }
  )
}

View the above resource in GitHub.

Testing automation documents

Now let's go and test this by manually invoking it. This can be done by navigating to Automation within Systems Manager and clicking Execute Automation.

For a shortcut of invoking the commands, you can use the below command to invoke the below CLI command. Then you may skip to the results.

aws ssm start-automation-execution \
  --document-name "PatchWithHealthcheck" \
  --document-version "\$DEFAULT" \
  --target-parameter-name InstanceIds \
  --max-errors "0" \
  --max-concurrency "1" \
  --region eu-west-1

Setup

Navigate to the Owned by me tab and select the name of your created document, PatchWithHealthcheck , then click Next.

Because we want to test this document executing one at a time on an instance, we'll need to select the Rate control option.

We'll need to select what instances are our targets. The instances in scope for this document are tagged with the key App with the value HelloWorld, so let's use that as our criteria. Note this is the most scalable solution for targeting instances.

Then we need to specify how the rate of invocation should be controlled. Our criteria for this is:

Execute on 1 instance at a time
Abort further invocations if any produce a error

Therefore we need to set the concurrency to 1 and the error threshold to 0.

Once all done then click Execute.

Results

As the execution progresses you'll notice it invokes the document on one instance at a time. As it completes you'll have a screen that looks like the below as you click onto the execution detail page. Notice that the start and end times of each instance invocation do not overlap with one another.

Step name is the same as the instance ID in this case

We can dive into each step invocation (the blue text in the above screenshot) to view the commands that were invoked.

At this detail we can see the individual aws:runCommand actions performed on the instance.

Failure testing

Okay so we've confirmed the document now only invokes synchronously. Let's now test to see if further invocations are aborted when there is a failure.

To simulate the failure, we can borrow a trick from the first post in this series by flipping the healthcheck script from > to <. Once that change is deployed we can re-run the document using the same method as above.

We can see the failure, and that the automation did not invoke on more instances! We can drill down into the invocation to see why it failed.

From this page you can then continue to drill down to the run command output to determine the cause of the failure.

Configuring automation tasks for maintenance windows

Now that we've tested the automation document and we're happy to have it automated, let's get this added to a maintenance window task. We can reuse the same maintenance window as we created last time, but with some differences.

Maintenance window task

Since we are targeting an automation document, there are some differences we need to account for when compared to command tasks.

Specify the task_type as AUTOMATION
Update the task_arn to our new document accordingly
Ensure our new combined document is only invoked on one instance at a time, so max_concurrency is set to 1
Within task_invocation_parameters we use automation_parameters as opposed to run_command_parameters.
- document_version allows us to target a specific document version
- any parameters required by the document are defined within parameter

Remember that our document takes in InstanceIds as a parameter? Well you'll notice that the value is set to "{{ TARGET_ID }}". This is known in AWS as a pseudo parameter, whereby the instance ID returned by WindowTargetIds will be passed into the automation document.

Depending on what resource_type your aws_ssm_maintenance_window_target is set up as will result in a different value to {{ TARGET_ID }} - in our case ours is INSTANCE, so this becomes the instance ID.

See the AWS docs for a full breakdown.

resource "aws_ssm_maintenance_window_task" "patch_with_healthcheck" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "AUTOMATION"
  task_arn         = aws_ssm_document.patch_with_healthcheck.arn
  priority         = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "1"
  max_errors      = "0"

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    automation_parameters {
      document_version = "$LATEST"

      parameter {
        name   = "InstanceIds"
        values = ["{{ TARGET_ID }}"]
      }
    }
  }
}

You can view the above resource in GitHub.

Maintenance window target

We need to update the target to use tag lookups against the instances - this mimics how we tested our automation document earlier on.

resource "aws_ssm_maintenance_window_target" "patch_with_healthcheck_target" {
  window_id     = aws_ssm_maintenance_window.patch_with_healthcheck.id
  name          = "PatchWithHealthcheckTargets"
  description   = "All instances that should be patched with a healthcheck after"
  resource_type = "INSTANCE"

  targets {
    key    = "tag:App"
    values = ["HelloWorld"]
  }
}

You can view the above resource in GitHub.

Additional IAM policies

We'll need to also attach some new permissions to the IAM role for the maintenance window, aws_iam_role.patch_mw_role.arn, to allow the automation document to perform a lookup on instances by their tag as defined in the updated aws_ssm_maintenance_window_target.patch_with_healthcheck_target resource.

data "aws_iam_policy_document" "mw_role_additional" {
  statement {
    sid    = "AllowSSM"
    effect = "Allow"

    actions = [
      "ssm:DescribeInstanceInformation",
      "ssm:ListCommandInvocations",
    ]

    resources = ["*"]
  }
}

resource "aws_iam_policy" "mw_role_add" {
  name        = "MwRoleAdd"
  description = "Additonal permissions needed for MW"

  policy = data.aws_iam_policy_document.mw_role_additional.json
}

resource "aws_iam_role_policy_attachment" "mw_role_add" {
  role       = aws_iam_role.patch_mw_role.name
  policy_arn = aws_iam_policy.mw_role_add.arn
}

You can view the above resources in GitHub.

Simply put, we're creating an new IAM policy with anything additional that the maintenance window requires for it to operate. We'll use this policy to add any new actions if we need to in the future.

Testing automation documents in maintenance windows

Once you've got the config above applied you'll need to run a test. Just like last time, you can do this by changing the maintenance window execution time to something relatively close to your current time.

Once the execution is complete (and hopefully it was successful, if not then check back at the configuration), you'll see that there will be 3 task invocations, one for each instance, and that none of them overlapped one another.

Notice how the start and end times don't overlap

Just like for command documents, you can view the detail for each invocation made on the instance. Here we can see the two steps that make up our newly created command document.

But the whole point of this exercise was to proactively handle errors right? So let's introduce some by doing what we've done before and change the healthcheck script to fail.

Once the new "broken" script is applied we can rerun another test of the maintenance window.

This time round we can see that there was a failure in one task invocation, which then aborted further invocations on the remaining instances! Just as before, we can do a deep dive into the invocation to determine why it had failed.

From here you can follow the command invoked by the step to troubleshoot the failure

Bonus: Using automation to dynamically invoke command documents

In this post we've looked at a common maintenance task in the form of an SSM command document called AWS-RunPatchBaseline, and created an automation document that will always invoke our healthcheck script after this invocation.

You may have more command documents that perform some form of maintenance on instances, for which you would also want the healthcheck script to execute after as well.

Instead of copying and pasting these automation documents, we can create just one automation document which takes in the command document ARN as a parameter, dynamically invoke it, and then have a hardcoded step afterward for executing a healthcheck!

In it's raw YAML form, we would get a document that looks like this:

---
schemaVersion: "0.3"
description: Executes a maintenance event on the instance followed by a healthcheck
parameters:
  InstanceIds:
    type: StringList
    description: The instance to target
  DocumentArn:
    type: String
    description: The document arn to invoke
  InputParameters:
    type: StringMap
    description: Parameters that should be passed to the document specified in DocumentArn
    default: {}
mainSteps:
  - name: InvokeMaintenanceEvent
    action: aws:runCommand
    inputs:
      DocumentName: "{{ DocumentArn }}"
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}
      Parameters: "{{ InputParameters }}"
  - name: ExecuteHealthcheck
    action: aws:runCommand
    inputs:
      DocumentName: ${healthcheck_document_arn}
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}

Just like what we've done for InstanceIds, we're taking in the DocumentArn as a parameter and providing it as the input for the aws:runCommand step. Some documents will also take parameters, so we can allow the caller to specify them using InputParameters, which is defined as a StringMap type - allowing it to then be used in as parameters for the document we are invoking.

When we create this document in the console and then execute it, we can then dynamically add in the document we want to invoke.

Then just like any other document we can invoke it.

We can see that the input from the previous screen was passed to the step. Let's keep going until we hit the command invocation.

The document name was dynamically passed to the command execution

Terraforming dynamic command documents

So now we've confirmed that documents can be dynamically invoked, let's get this Terraformed. You can view this in GitHub.

resource "aws_ssm_document" "maintenance_wrapper" {
  name            = "MaintenanceWithHealthcheck"
  document_type   = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/maintenance_wrapper_template.yml",
    {
      healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
      output_s3_bucket_name    = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix     = "ssm_output/",
    }
  )
}

Not a lot has really changed in this when we compare it to our one from earlier, but the difference is in the parameters field for the document.

And then you can include it as a maintenance window task as below; I'm reusing the same maintenance window task as before.

resource "aws_ssm_maintenance_window_task" "patch_with_healthcheck" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "AUTOMATION"
  task_arn         = aws_ssm_document.maintenance_wrapper.arn
  priority         = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "1"
  max_errors      = "0"

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    automation_parameters {
      document_version = "$LATEST"

      parameter {
        name   = "InstanceIds"
        values = ["{{ TARGET_ID }}"]
      }

      parameter {
        name   = "DocumentArn"
        values = ["AWS-RunPatchBaseline"]
      }

      parameter {
        name   = "InputParameters"
        values = [jsonencode({ Operation = "Scan" })]
      }
    }
  }
}

You can then copy and paste the same task resource, only changing the values for the InputParameters and DocumentArn parameters accordingly. If the document you are calling doesn't take any parameters, then you can just omit that block.

You'll have to ensure that the IAM role the maintenance window task is assuming has the correct IAM permissions as required by the document you're calling.

Conclusion

What we've done in this post is taken our rudimentary command document, prone to introducing errors into our estate, and converted it to an automation document. With the right SSM maintenance window settings, you can ensure that any maintenance tasks you need to perform on your EC2 instances are done so in a manner that reduces the risk of errors in your environment.

Next time, we'll be taking this a step further to proactively remove EC2 instances from circulation when behind a load balancer for maintenance activities.

Automate Instance Hygiene with AWS SSM: Maintenance Windows

Joseph Heyburn — Mon, 16 Nov 2020 00:00:00 +0000

Last time we looked at writing our own SSM Command Document for the purpose of executing a healthcheck script on a set of EC2 instances across multiple platforms.

In this post we’ll be exploring how we can automate this using maintenance windows - also within the SSM suite. This is something I’ve covered before, but want to extend on that to show how its done.

tl;dr

We can use SSM Maintenance Windows to automate our newly created command documents on a schedule
Multiple command documents can be combined in a maintenance window, such as a patching event followed by a healthcheck
This provides us with a means of viewing historical invocations on whatever workflow we’ve automated
By storing command outputs to S3, we can ensure we can recover logs that are too large to display in the console
Using S3 Lifecycle Rules we can remove aged logs

Once again all the Terraform code for this post is available on GitHub. It is split into two parts:

aws-ssm-automation-1-barebones is for the barebones walkthrough
aws-ssm-automation-1-logging is for the logging enhancement

Intro to Maintenance Windows

Maintenance Windows, are a means of executing some automation workflow in your AWS estate on a schedule. Got an SSM Document you’ve written and want it automated? What about a Lambda you want invoked at a regular schedule? Or maybe it’s a Step Function? Whatever the use case, Maintenance Windows are for you - just don’t be fooled by the name - they don’t necessarily have to be just for maintenance!

Similarities to EventBridge Rules

“But wait” , I hear you ask, “don’t CloudWatch/EventBridge Rules also allow you to invoke events on a schedule too?"

Yes they do - both Maintenance Windows and EventBridge Rules (the bigger sibling of CloudWatch Rules) use cron expressions to define the schedule they should run on. The primary difference between the two is that Maintenance Windows allow you to specify the timezone that the cron expression adheres to, whereas EventBridge is tied to UTC.

No timezone, no party

So using maintenance windows can be handy if you’re in a non-UTC timezone and you don’t have to constantly convert your local timezone to UTC to schedule events. More importantly, maintenance windows will respect daylight savings time (DST) if your timezone observes it, so you can be sure your automation will be invoked at the same time in the specified timezone throughout the year.

On the other hand, EventBridge Rules are fixed to UTC; meaning if your timezone does observe DST, then you’ll find your automation could be off by an hour for some portion of the year (unless you change it of course - but who wants to be changing automation twice a year??).

I don’t think I’ve ever met a software engineer that’s a fan of DST!

Notably as well, you can view the execution history of maintenance windows as they’ve occurred in the past, allowing you to quickly see whether a particular invocation was successful or not - and drill down into any failures.

I’m not bashing EventBridge Rules, in fact, it is easier to set up than Maintenance Windows. But there’s always the right tool for the job.

For the rest of this post, we’re going to be exploring how to automate the command document we created last time. Later on in the series we’ll be looking at using maintenance windows to automate automation documents.

Automating command documents with maintenance windows

So what’s the purpose of creating the command document we achieved last time? Well we’re not going to be manually invoking it like what we have been doing so far - as engineers we need to be automating as many repetitive tasks as possible.

To summarise where we are now, we’ve produced a Command document which when executed, automates the following:

Downloads a healthcheck script from S3
Executes the healthcheck script, failing the command invocation if healthcheck does not pass

Healthchecks are important to run both continuously in our environment, as a means of monitoring and verifying the estate is working as intended, before your users notice. They are also necessary to run after a change has been introduced to the environment, such as a new code deployment, or even a patching event via the AWS-RunPatchBaseline document.

Barebones maintenance window with AWS-RunPatchBaseline

We’re going to use Terraform again to build out a minimal maintenance window. You can view the code for it here. You’ll notice the repository this file sits in is identical to the one from the first post, except I’ve included the new resources that will enable us to accomplish the requirement.

Here is a breakdown of each of the resources we’re going to create.

Maintenance Window

This creates the maintenance window resource, which is then referred to in the subsequent resources we create.

It’s nothing more than that cron expression I mentioned earlier, along with the timezone it should execute in
We specify how long the window lasts for, and the cutoff; both of which are specified in hours
- The cutoff indicates how long before the end of the window should AWS not schedule any new tasks in that window

resource "aws_ssm_maintenance_window" "patch_with_healthcheck" {
  name              = "PatchWithHealthcheck"
  description       = "Daily patch event with a healthcheck afterward"
  schedule          = "cron(0 9 ? * * *)" # Everyday at 9am UK time
  schedule_timezone = "Europe/London"
  duration          = 3
  cutoff            = 1
}

Maintenance Window Target

We need a means of telling the window what instances to target, and the aws_ssm_maintenance_window_target resource is how you do it. Below I’m demonstrating two methods of doing this:

Specify the instance IDs directly
- Handy if you have a fixed list of instances you only want to be included in the maintenance window
Target instances by their tags
- This is much for scalable, and means you don’t have to keep adding instance IDs to the list
- When the maintenance window executes, it will filter instances with this tag key and value combo for what to target

An optimum tag to use would be Patch Group described here - which I have mentioned previously. For the sake of this demo, we will keep it simple by targeting instance IDs.

resource "aws_ssm_maintenance_window_target" "patch_with_healthcheck_target" {
  window_id     = aws_ssm_maintenance_window.patch_with_healthcheck.id
  name          = "PatchWithHealthcheckTargets"
  description   = "All instances that should be patched with a healthcheck after"
  resource_type = "INSTANCE"

  targets {
    key = "InstanceIds"
    values = concat(
      module.windows_ec2.id,
      module.linux_ec2.id
    )
  }

  # Using tags is more scalable
  #   targets {
  #     key    = "tag:Terraform"
  #     values = ["true"]
  #   }
}

Maintenance Window Tasks

Patching task

Now for the tasks… remember that we want to execute our healthcheck SSM document after a patch event right? We need to build a task for executing the AWS-RunPatchBaseline document.

resource "aws_ssm_maintenance_window_task" "patch_task" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "RUN_COMMAND"
  task_arn         = "AWS-RunPatchBaseline"
  priority         = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "100%"
  max_errors      = 0

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    run_command_parameters {
      timeout_seconds  = 3600

      parameter {
        name = "Operation"
        values = ["Scan"]
      }
    }
  }
}

Let’s run through the main attributes:

window_id - the maintenance window to associate this task with
task_type - what kind of task this is
- since we’re executing a command document, the value here is RUN_COMMAND
task_arn - the ARN of the document you wish to run
- note the document name can also be used here, as demonstrated above
priority - defines in what order should tasks be executed in, whereby the lower the number given, the earlier the task is executed in the window
- e.g. a task priority of 1 gets executed before one with 10
- tasks with the same priority get executed in parallel
service_role_arn - tells which IAM role should be assumed to execute this task as
- we’ll get an explanation of this later
max_concurrency - specifies how many instances this task should be invoked on simultaenously
- it can take a percentage as a value, only applying to that percentage of target instances at a time
- i.e. 50% indicates only half of targeted instances will have the task executed at a time
- or it can take a fixed number such as 1; indicating that only one instance should have this task executed at a time
- 100% indicates this task will be invoked on all targets at the same time
max_errors - indicates how many errors should be thrown before we abort further invocations
- 0 indicates any error will abort the maintenance window and set its result to failed

The targets block allows us to define what instances to target this on - we’re referencing the aws_ssm_maintenance_window_target resource we created previously.

Lastly the task_invocation_parameters allows us to customise how the document should be ran via the parameter setting - which is passed to the document. For this example we’re only performing the Scan operation on the document, for testing purposes.

Scan will only check for missing patches - it won’t actually install them.

A full list of available commands can be found in the AWS-RunPatchBaseline command document.

In the console GUI we can see the available parameters for the AWS-RunPatchBaseline document

Healthcheck task

Next we want to define the healthcheck task.

resource "aws_ssm_maintenance_window_task" "healthcheck_task" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "RUN_COMMAND"
  task_arn         = aws_ssm_document.perform_healthcheck_s3.arn
  priority         = 20
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "100%"
  max_errors      = 0

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    run_command_parameters {
      timeout_seconds  = 600
    }
  }
}

There’s not a whole lot of difference here compared to the patching task; our healthcheck script takes no parameters so we can leave them out (although if yours does, you’ll need to add it here!), and the task_arn points to the command document we created last time.

Probably the most significant change though is the priority. Remember that the priority number indicates the ordering of tasks to be invoked? Our patching task had a priority of 10, whereby our healthcheck task is 20. This means the patch task will be invoked before the healthcheck one.

I could have set the priority of the patching and healthcheck tasks to 1 and 2 respectively to achieve the same thing.

However, giving some distance between them means you can programmatically add new tasks before/after each other.

Want a post-patch, pre-healthcheck task? Attach a new task with priority 15!

IAM role for maintenance window

We’ve been referencing aws_iam_role.patch_mw_role.arn as our task service_role_arn. You can view the code for it here - but let’s run through them quickly.

All we’re doing is creating an IAM role, allowing the EC2 and SSM AWS services to assume said role, and applying the predefined AWS policy AmazonSSMMaintenanceWindowRole to that role. This policy gives some basic permissions to the role which allow it to execute commands and more on the instances.

data "aws_iam_policy_document" "patch_mw_role_assume" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type = "Service"

      identifiers = [
        "ec2.amazonaws.com",
        "ssm.amazonaws.com",
      ]
    }
  }
}

resource "aws_iam_role" "patch_mw_role" {
  name               = "PatchingMaintWindow"
  assume_role_policy = data.aws_iam_policy_document.patch_mw_role_assume.json
}

data "aws_iam_policy" "ssm_maintenance_window" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonSSMMaintenanceWindowRole"
}

resource "aws_iam_role_policy_attachment" "patch_mw_role_attach" {
  role       = aws_iam_role.patch_mw_role.name
  policy_arn = data.aws_iam_policy.ssm_maintenance_window.arn
}

Testing the barebones maintenance window

Once we’ve ran terraform apply on all the above, we can test the maintenance window out. Currently we have it set to run at 9am UK time, which may or may not be a long time away - so change it manually in the console to a time not far away from your time now.

Once it’s done executing, you can navigate to the history tab to view the execution.

Let’s hope you have more luck on your first attempts running this than I did!

You can select any execution and drill down into it with the View details button.

You can see that the tasks got executed in the order we defined them in

We can go deeper in the execution details and pull out the result of individual commands by selecting View details on the task invocation.

The maintenance window has redirected us to the same page as when we manually invoked the command documents in the last post

Clicking on one of the instance IDs in the above screenshot will take us to the command output for that instance.

Logging command output to S3

Maintenance windows by default only capture the first 2500 characters of a command output, if your command outputs more than this then it gets truncated. This can be a problem if you have a task failure and need to examine the output for the reason why it failed.

Take the AWS-RunPatchBaseline output on a Linux instance for example. It’s pretty hefty, and so we lose a lot of context on what actually happened:

Method

To combat this, maintenance windows allow you to dump command output to an S3 bucket, so that you can retrieve it later. In the last post we created an S3 bucket to store our SSM scripts (aws_s3_bucket.script_bucket.arn), we can reuse that bucket to store our command logs too.

In order to do this there are some steps we need to take:

The S3 bucket policy needs to permit the EC2 instance role aws_iam_role.vm_base to s3:PutObject on "${aws_s3_bucket.script_bucket.arn}/ssm_output/*"
- ssm_output/ is the directory/prefix in the S3 bucket where we will store the logs
The KMS key used to encrypt objects in the target S3 bucket needs to permit instance role aws_iam_role.vm_base to kms:GenerateDataKey
The instance role aws_iam_role.vm_base needs permissions to do the above on its respective side

You can view the changes required in GitHub:

Once that is done we’ll need to add some new attributes to the maintenance window tasks, telling it where to dump the command output to.

resource "aws_ssm_maintenance_window_task" "task_name" {

  # ... other attributes hidden

  task_invocation_parameters {
    run_command_parameters {
      output_s3_bucket     = aws_s3_bucket.script_bucket.id
      output_s3_key_prefix = "ssm_output/"

      # ... other attributes hidden
    }
  }
}

When you have the new config written, then you can terraform apply and run another test on the maintenance window.

We now get a button that can redirect us to where the logs are stored in S3

If we click on this button, we can view the logs being stored in S3. Follow the path in S3 until you reach the S3 object containing the logs.

Now you can open this using the Object actions button in the top-right hand corner to view the entire logs!

Logging problems

It’s worth noting that SSM does not raise an error if an instance cannot push logs to S3 - the Amazon S3 button will redirect you to an object in S3 that does not exist. So if your logs are not appearing in S3 then ensure you’ve followed the steps above.

An example showing if logs were not successfully uploaded to S3 - if you get this then double check you’ve set everything up right!

Removing old command logs

Now that we have maintenance windows storing our logs in S3, we should ensure we’re maintaining a good level of hygiene by removing old logs - otherwise our S3 bucket is going to store more and more logs, costing us more money.

S3 has a feature called Lifecycle Rules which tells S3 how to handle objects throughout their lifecycle. We can tell it to move old files to a cheaper storage class, archive them to S3 Glacier (AWS’s long-term storage service), or just simply delete them!

Given we’re not exactly sentimental with logs, we can define a policy that will remove any logs older than 3 months (90 days).

This is very easy for us to add, we simply need to make the addition below to our aws_s3_bucket resource.

resource "aws_s3_bucket" "script_bucket" {
  # ... other attributes hidden

  # Remove old SSM command output logs
  lifecycle_rule {
    id      = "RemoveOldSSMOutputLogs"
    enabled = true

    prefix = "ssm_output/"

    expiration {
      days = 90
    }
  }
}

You can add more rules if you like, such as a transition block to move it to cold storage before deleting if you wish. Take a look at the Terraform documentation for the resource for example of this.

Logs older than 90 days certainly do not spark joy…!

Conclusion

With the first two posts of this series, you have enough to be able to create your own automated series of commands that can be executed on your EC2 instances.

You can also use SSM documents to retrieve files from instances - such as log files. Or even use them to update third-party software on the instances.

The next post and thereafter we’ll be exploring the Command Documents sibling; Automation Documents, and exploring how these can further enhance automation to other AWS services beyond EC2 instances.

Automate Instance Hygiene with AWS SSM: Command Documents

Joseph Heyburn — Fri, 30 Oct 2020 00:00:00 +0000

It’s been a while since my last post… which could be down to me trying to salvage something out of summer! 😅

In this post I want to talk a little bit more about AWS SSM. This was something I touched on when discussing patch baselines in a previous post, and within there is another service known as Automation. On first glance it may look pretty dull, but once you scratch the surface there are a number of capabilities it can unlock for you. I found a distinct lack of resources on how to write these documents, so this post will aim to help you get started on doing so!

This will be part one of a four part series about SSM Automation. The outline of the posts will be:

Re-introduction to SSM Documents in general, specifically the Command document by writing our own healthcheck document
How to automate command documents with SSM Maintenance Windows
Looking into Automation documents, and integrating them with maintenance windows
Advanced use case for Automation documents

tl;dr

To summarise the key parts of this post:

SSM can be used to automate several parts of your estate
Command documents are a means of executing logically indifferent commands across multiple platforms
How to write a basic command document in Terraform
How to write a verbose command document in Terraform

Prerequisites

This series assumes you have some knowledge of:

Terraform
These AWS services:
- EC2
- IAM
- S3

All source code for this post is available on GitHub, I’ll be referencing it throughout.

SSM Re-primer

I’ve mentioned SSM Documents before; to save you a click:

An SSM Document is essentially an automation script that you can perform on one or more instances at a time, with conditions to apply different sets of scripts depending on the operating system (OS) platform (i.e. Windows / Linux).

In that post I talk about how AWS-RunPatchBaseline is an example of a COMMAND document. Another type of document is known as AUTOMATION. AWS describes them as:

Command
- Uses command documents to run commands
- State Manager uses command documents to apply a configuration
- These actions can be run on one or more targets at any point during the lifecycle of an instance
- Maintenance Windows uses command documents to apply a configuration based on the specified schedule
Automation
- Use automation documents when performing common maintenance and deployment tasks such as creating or updating an Amazon Machine Image (AMI)
- State Manager uses automation documents to apply a configuration
- These actions can be run on one or more targets at any point during the lifecycle of an instance
- Maintenance Windows uses automation documents to perform common maintenance and deployment tasks based on the specified schedule

I like to use the below to differentiate between the two:

Command documents execute scripts on instances
Automation document can call and orchestrate AWS API endpoints on your behalf, including executing Command documents on instances

There are also other types too which are beyond the scope of this series.

SSM Managed Instances

In order to have command documents executed on your instances, they will need to become managed instances. This requires having the below set up correctly on your instances:

the SSM Agent installed on your instance
- done so by default on all Amazon Linux AMIs and Windows AMIs
connectivity from your instances to the following endpoints:
- https://ssm.REGION.amazonaws.com
- https://ssmmessages.REGION.amazonaws.com
- https://ec2messages.REGION.amazonaws.com
the correct IAM permissions applied on your EC2 instance profile
- these are all provided by the AWS IAM policy AmazonSSMManagedInstanceCore
- the Terraform example for this post shows the policy being attached to the EC2 IAM role
Optional : additional policies that may be required based on your use case

If your instances are appearing in the managed instances console then everything is set up correctly; if not then follow the troubleshooting guide.

Command documents

Documents are defined in either JSON or, for better or for worse, YAML. You can also define what OS each command should be executed on. This could be helpful if you wanted a healthcheck script to be executed across all (or a subset of) your instances in one action. As an example, my healthcheck script could be to check to see if the CPU is overloaded.

Constructing a healthcheck script

I could use the below to perform a healthcheck on a Windows box:

$Avg = (Get-WmiObject Win32_Processor | Measure-Object -Property LoadPercentage -Average | Select Average).Average
If ($Avg -gt 90) {
    Throw "Instance is unhealthy - Windows"
}
Write-Output "Instance is healthy - Windows"

And the equivalent for Linux would be:

# Sources:
# https://stackoverflow.com/a/9229580
# https://bits.mdminhazulhaque.io/linux/round-number-in-bash-script.html
avg_cpu=$(grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print int(usage)+1}')
if (( avg_cpu > 90 )); then
    echo "Instance is unhealthy - Linux"
    exit 1
fi
echo "Instance is healthy - Linux"

You’ll notice the scripts output the OS they are running on - this will serve as an explanation for when we run the scripts as a command document later on.

Note these scripts are just rudimentary examples of healthcheck scripts. Your healthcheck script may be checking that services are running ok, whether a task can be performed, etc. For the sake of this example, I’ve decided to do a simple check against the CPU load for demonstration.

Testing in AWS SSM Console

Let’s now test them in the AWS SSM Console. For this example I’ve spun up two EC2 instances, one Linux and one Windows, using Terraform. To run the commands we can navigate to Run Command in the console, and run AWS-RunPowerShellScript and AWS-RunShellScript for both Windows and Linux EC2s respectively. We don’t care about logging the output of the scripts just yet. Make sure when running the script you manually select what instance to run it on.

We can see they have executed fine. Let’s flip the condition in the healthcheck script so we can test them failing (done by changing > to <).

This is cool - in the script we can define what constitutes as a failure and have this propagate up to AWS - notice how the status was Failed. This will come in use later on in the series.

Terraforming command documents

Now let’s get these scripts Terraformed so we can reap the benefits of infrastructure-as-code. First we need to define the document in the YAML format.

# documents/perform_healthcheck.yml
---
schemaVersion: "2.2"
description: Perform a healthcheck on the target instance
mainSteps:
  - name: PerformHealthCheckWindows
    action: aws:runPowerShellScript
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      runCommand:
        - "$Avg = (Get-WmiObject Win32_Processor | Measure-Object -Property LoadPercentage -Average | Select Average).Average"
        - "If ($Avg -gt 90) {"
        - '  Throw "Instance is unhealthy- Linux"'
        - "}"
        - 'Write-Output "Instance is healthy - Windows"'
  - name: PerformHealthCheckLinux
    action: aws:runShellScript
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      runCommand:
        - "avg_cpu=$(grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print int(usage)+1}')"
        - "if (( avg_cpu > 90 )); then"
        - '  echo "Instance is unhealthy - Linux"'
        - "  exit 1"
        - "fi"
        - 'echo "Instance is healthy - Linux"'

GitHub URL for the above

Since a document is meant to perform an action (or group of actions) on a set of instances regardless of their operating system (OS), only the steps that apply to the OS platform for the instance they are being executed on will be invoked.

Therefore when we target this document to run on two types of EC2 instances, one Windows and one Linux, the step PerformHealthCheckWindows will be executed on the Windows box and vice versa for PerformHealthCheckLinux. This is because of the precondition key which filters on the platformType for the instance. Notably as well, we are targeting the appropriate actions for each platform; aws:runPowerShellScript for Windows and aws:runShellScript for Linux.

See here for a full list of what actions you can perform in a command document.

Using Terraform you can deploy out the document to your environment like so.

resource "aws_ssm_document" "perform_healthcheck" {
  name            = "PerformHealthcheck"
  document_type   = "Command"
  document_format = "YAML"

  content = file("documents/perform_healthcheck.yml")
}

GitHub URL for the above

Once deployed, we can navigate to System Manager in the AWS Console to invoke the document on our estate. This is done in the same manner as when we ran AWS-RunShellScript above, except now we are targeting PerformHealthcheck. Since our document has commands for both Linux and Windows, we can have it invoked across both platform types and only the scripts written for their platform will be invoked.

We can see that both executed successfully! You can view the output of each command invocation on each instance like previously. For the screenshot below of the Linux instance, only the Linux step was executed, whereas the Windows step was skipped. You’ll notice our message from earlier “- Linux” is there, reassuring us that only the Linux script was executed.

Let’s dive into some more intermediate documents.

Verbose command documents

The example above was a very basic example of such a document where we only had a few lines of code to execute. The healthcheck script you write for your service may have several more lines of code to execute, and trying to read lines of code in amongst the document markup format is not easy on the eyes. Here is a snippet of the AWS-RunPatchBaseline document as an example of what I mean. It is written in JSON and has over 100 lines in the runCommand section.

{
  // ...
  "mainSteps": [
    {
      "precondition": {
        "StringEquals": ["platformType", "Windows"]
      },
      "action": "aws:runPowerShellScript",
      "name": "PatchWindows",
      "inputs": {
        "timeoutSeconds": 7200,
        "runCommand": [
          "# Check the OS version",
          "if ([Environment]::OSVersion.Version.Major -le 5) {",
          "    Write-Error 'This command is not supported on Windows 2003 or lower.'",
          "    exit -1",
          "} elseif ([Environment]::OSVersion.Version.Major -ge 10) {",
          "    $sku = (Get-CimInstance -ClassName Win32_OperatingSystem).OperatingSystemSKU",
          "    if ($sku -eq 143 -or $sku -eq 144) {",
          "        Write-Host 'This command is not supported on Windows 2016 Nano Server.'",
          "        exit -1",
          "    }",
          "}",
          "# Check the SSM agent version",
          "$ssmAgentService = Get-ItemProperty 'HKLM:SYSTEM\\CurrentControlSet\\Services\\AmazonSSMAgent\\'",
          "if (-not $ssmAgentService -or $ssmAgentService.Version -lt '2.0.533.0') {",
          "    Write-Host 'This command is not supported with SSM Agent version less than 2.0.533.0.'",
          "    exit -1",
          "}",
          ""
          // ...
        ]
      }
    }
  ]
}

You can see the whole thing on AWS.

Even from this snippet it is hard to distinguish what is going on. Losing out on syntax highlighting means the code isn’t readable by any means. And since this is a JSON file format, we lose type hinting for the language we are writing the script in (PowerShell in this case). If the script was saved in as a PowerShell file (.ps1) then we’d get all those benefits.

We can fix all these issues by adopting a common pattern when composing command documents. We can have S3 store the script, then have the command document perform these actions:

Download the script in question from S3
Execute the script from the download location

Such a command document would have a composition as below, whereas this is performing the same healthcheck script previously.

# documents/perform_healthcheck_s3.yml
---
schemaVersion: "2.2"
description: Perform a healthcheck on the target instance
mainSteps:
  - action: aws:downloadContent
    name: DownloadScriptWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/jdheyburn-scripts/ssm_scripts/PerformHealthcheck.ps1"}'
  - action: aws:runPowerShellScript
    name: ExecutePerformHealthCheckWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      runCommand:
        - ..\downloads\PerformHealthcheck.ps1
  - action: aws:downloadContent
    name: DownloadScriptLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/jdheyburn-scripts/ssm_scripts/perform_healthcheck.sh"}'
  - action: aws:runShellScript
    name: ExecutePerformHealthCheckLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      runCommand:
        - ../downloads/perform_healthcheck.sh

GitHub URL for the above

Note the new action called aws:downloadContent - which you can view the documentation for here. Again we’re using the precondition key to ensure each platforms downloads their respective script. We’re also using the inputs key to instruct the action where it can download the script from; S3 in this case, at the given S3 bucket location. There is an optional destinationPath field which allows you to change where it downloads to.

Once the script is downloaded to the instance, we will need to have it executed. aws:downloadContent saves the script to a temporary directory for executing SSM command invocations on instances, so we need to reference it in the downloads directory for it; indicated by the ../downloads/perform_healthcheck.sh command.

Terraforming verbose command documents

This involves having your script first uploaded to S3. Thankfully, through the power of infrastructure-as-code, you can have this automatically deployed to your environment once the module is written.

Check the GitHub repository for the full example, for now let me break down what each bit is doing.

KMS

I want my S3 bucket to encrypt objects that are stored there, so we’ll need to create a KMS key and give it an IAM policy permitting any users of the account to decrypt using the key. You can make this more secure by only targeting the IAM roles that require it, instead of the whole account, adopting the principle of least privilege best practice 🔒

You can read up more about IAM policies here.

data "aws_iam_policy_document" "kms_allow_decrypt" {
  statement {
    sid    = "AllowKMSAdministration"
    effect = "Allow"

    actions = [
      "kms:Create*",
      "kms:Describe*",
      "kms:Enable*",
      "kms:List*",
      "kms:Put*",
      "kms:Update*",
      "kms:Revoke*",
      "kms:Disable*",
      "kms:Get*",
      "kms:Delete*",
      "kms:ScheduleKeyDeletion",
      "kms:CancelKeyDeletion"
    ]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:user/jdheyburn"]
    }

    resources = ["*"]
  }

  statement {
    sid    = "AllowDecrypt"
    effect = "Allow"

    actions = ["kms:Decrypt"]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
      # TIP: For increased security only give decrypt permissions to roles that need it
      # identifiers = [aws_iam_role.vm_base.arn]
    }

    resources = ["*"]
  }
}

resource "aws_kms_key" "script_bucket_key" {
  description = "This key is used to encrypt bucket objects"
  policy      = data.aws_iam_policy_document.kms_allow_decrypt.json
}

S3

Next we’re creating the S3 bucket to store the scripts; we’re encrypting it with the KMS key we created, aws_kms_key.script_bucket_key. We’re also applying an IAM policy on the bucket permitting any users of the account to download anything from the ssm_scripts/ prefix in the bucket (this prefix is where the scripts will be stored).

resource "aws_s3_bucket" "script_bucket" {
  bucket = "jdheyburn-scripts"

  # Encrypt objects stored in S3
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = aws_kms_key.script_bucket_key.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "s3_allow_script_download" {
  statement {
    sid    = "AllowAccountAccess"
    effect = "Allow"

    actions = ["s3:GetObject"]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
      # TIP: For increased security only give decrypt permissions to roles that need it
      # identifiers = [aws_iam_role.vm_base.arn]
    }

    resources = ["${aws_s3_bucket.script_bucket.arn}/ssm_scripts/*"]
  }
}

resource "aws_s3_bucket_policy" "script_bucket_policy" {
  bucket = aws_s3_bucket.script_bucket.id
  policy = data.aws_iam_policy_document.s3_allow_script_download.json
}

Store Scripts in S3

Now we are uploading the scripts from their location in the repository to S3, at the prefix where we defined an IAM policy to pull them from.

locals {
  perform_healthcheck_script_fname_windows = "PerformHealthcheck.ps1"
  perform_healthcheck_script_fname_linux   = "perform_healthcheck.sh"
}

resource "aws_s3_bucket_object" "perform_healthcheck_windows" {
  bucket  = aws_s3_bucket.script_bucket.id
  key     = "ssm_scripts/${local.perform_healthcheck_script_fname_windows}"
  content = file("scripts/${local.perform_healthcheck_script_fname_windows}")
}

resource "aws_s3_bucket_object" "perform_healthcheck_linux" {
  bucket  = aws_s3_bucket.script_bucket.id
  key     = "ssm_scripts/${local.perform_healthcheck_script_fname_linux}"
  content = file("scripts/${local.perform_healthcheck_script_fname_linux}")
}

SSM Document

resource "aws_ssm_document" "perform_healthcheck_s3" {
  name            = "PerformHealthcheckS3"
  document_type   = "Command"
  document_format = "YAML"

  content = templatefile(
    "documents/perform_healthcheck_s3_template.yml",
    {
      bucket_name   = aws_s3_bucket.script_bucket.id,
      linux_fname   = local.perform_healthcheck_script_fname_linux,
      linux_key     = aws_s3_bucket_object.perform_healthcheck_linux.id,
      windows_fname = local.perform_healthcheck_script_fname_windows,
      windows_key   = aws_s3_bucket_object.perform_healthcheck_windows.id,
    }
  )
}

Here we’re creating an aws_ssm_document as we had done before, but the document file we’re targeting this time is a template file.

Notice how we have the presence of ${bucket_name} and others? These are template variables. With the use of the Terraform function templatefile(), we can insert Terraform variables into the config to have the template name replaced with the value we’re passing in. In this case, ${bucket_name} will get replaced with the output of aws_s3_bucket.script_bucket.id, which is jdheyburn-scripts, and the same for the remaining variables.

# documents/perform_healthcheck_s3_template.yml
---
schemaVersion: "2.2"
description: Perform a healthcheck on the target instance
mainSteps:
  - action: aws:downloadContent
    name: DownloadScriptWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/${bucket_name}/${windows_key}"}'
  - action: aws:runPowerShellScript
    name: ExecutePerformHealthCheckWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      runCommand:
        - ..\downloads\${windows_fname}
  - action: aws:downloadContent
    name: DownloadScriptLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/${bucket_name}/${linux_key}"}'
  - action: aws:runShellScript
    name: ExecutePerformHealthCheckLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      runCommand:
        - ../downloads/${linux_fname}

GitHub URL for the above

EC2 IAM Permissions

For the pattern to work, you’ll need to attach an IAM policy to the instance to allow it to pull the scripts from the S3 bucket. If we don’t do this then the aws:downloadContent action will fail. From the Terraform above we’ve already applied the corresponding permissions on the S3 bucket, allowing all users and roles in the account to perform s3:GetObject on the scripts. We’ve also allowed done the same for performing kms:Decrypt on the KMS key that encrypts the S3 objects.

If you’ve been following the GitHub example, these required permissions have already been defined - for a recap:

data "aws_iam_policy_document" "ssm_scripts" {
  statement {
    sid    = "AllowS3"
    effect = "Allow"

    actions = ["s3:GetObject"]

    resources = [
      "${aws_s3_bucket.script_bucket.arn}/ssm_scripts/*",
    ]
  }

  statement {
    sid    = "AllowKMS"
    effect = "Allow"

    actions = ["kms:Decrypt"]

    resources = [aws_kms_key.script_bucket_key.arn]
  }
}

resource "aws_iam_policy" "ssm_scripts" {
  name        = "PullSSMScripts"
  description = "Enables instances to download SSM scripts from S3"

  policy = data.aws_iam_policy_document.ssm_scripts.json
}

resource "aws_iam_role_policy_attachment" "instance_download_scripts" {
  # Change this to point to the role(s) for your instances
  role       = aws_iam_role.vm_base.name
  policy_arn = aws_iam_policy.ssm_scripts.arn
}

This is an important concept to remember about setting IAM policies - you will need to ensure that the correct permissions are applied on both the source of the requestor, as well as the destination.

Testing verbose documents

Now, let’s give this command a spin in the console. We’re going to execute it the same way we did for other documents earlier, except now targeting PerformHealthcheckS3.

If you got any failures, make sure to dive into the failed invocation and see why it failed. Did it fail because of your healthcheck command? Then it is working as intended! Although if it failed on aws:downloadContent, check to make sure your instances are running the latest version of SSM agent. You can do this with the AWS-UpdateSSMAgent SSM document. Don’t be like me and spend hours troubleshooting against an out-of-date SSM agent! 😂

Joseph D. Heyburn

@jdheyburn

Trying to write my next blog post about AWS SSM, and had a load of hours wiped out trying to troubleshoot an issue with SSM agents... turns out AWS AMIs install these bad agent versions by default 😭

github.com/aws/amazon-ssm…

14:38 PM - 24 Oct 2020

0 0

Let’s now dive into the output of the Linux instance.

Just like with the previous document with the script embedded PerformHealthcheck, we can see the steps conditioned for Windows have been skipped (steps 1-2). Step 3 is where the document is doing real work, downloading the Linux script from the S3 location into the temp directory for SSM, and then executing it in step 4.

Conclusion

I think this has been my longest post thus far, and it’s only part one of this series. SSM is a bit of a beast and is often shrugged off as being a pain to set up and troubleshoot. I put that down to it encapsulating several other services, and a lack of tried and tested documented methods - which I hope this series resolves.

We covered in this post:

What SSM Documents are; and the differences between Command and Automation documents
How to create our own command documents to execute the same business logic on differing instance platforms
Adopting best practices by storing scripts in S3 and have a command document orchestrate the download and execution of said scripts
… all while written in Terraform!

If any mistakes were made in this post then please contact me - thanks for reading! 😃

Assertions in gotests Test Generation

Joseph Heyburn — Sun, 26 Jul 2020 00:00:00 +0000

I’ve been doing some programming in Go for a side project again, and I’m back using gotests to generate unit tests for functions. For this I’ve been referencing a post I’ve previously written in order to help me get them set up. If you’d like more context on the background I’d recommend reading there first.

Today I’ll be talking about a small enhancement to how the tests are generated to make use of the assert package within testify. Go already comes with enough for you to write tests, but assert provides me with more options for comparison in a natural language form. I’ll also be adding support for when a test case returns an unexpected error.

Don’t care about the waffle? Jump straight to it here.

Recap

From the last time we visited this, our test code took the format below.

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: errors.New("dog cannot have symbols in their name"),
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
                t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
                return
            }
            if got != tt.want {
                t.Errorf("validateDogName() = %v, want %v", got, tt.want)
            }
        })
    }
}

Although this works nicely, there is one issue - we’re not capturing unexpected errors. Or rather, if an error is returned to err, and tt.wantErr is set to nil, then the test will not fail.

Okay, so we still have the if got != tt.want condition to fail the test if needed. Although we still could have this condition pass, we want to make sure we capture the error. The test suite is doing currently is assuming we don’t care about the outcome of err, just because we didn’t want one as described by tt.wantErr.

I seem to remember assumptions being the ~~brother~~ mother of something…

Enhancing for Unexpected Errors

In order to enhance what we have already from the original modified function.tmpl, we could have it output something like this to capture unexpected errors.

// ... removed for brevity
for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
        got, err := validateDogName(tt.args.name)
        if err != nil && tt.wantErr == nil {
            t.Errorf("validateDogName() unexpected error = %v", err)
            return
        }
        if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
            t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
            return
        }
        if got != tt.want {
            t.Errorf("validateDogName() = %v, want %v", got, tt.want)
        }
    })
}

This enhancement is something we could implement fairly easily. On the other hand, the assert library gives us a lot more to play with. It essentially is doing the same as the above under the hood, albeit in a cleaner fashion… and I’m all for better code readability!

Rewriting for Assert

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: errors.New("dog cannot have symbols in their name"),
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
                return
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
                return
            }
            assert.Equal(t, tt.want, got)
        })
    }
}

The above code block is what we get when we take the test code above that is using the t.Errorf function call to record a test failure, and rewrite it to use assert.

What we now need to do is have gotests generate it for us.

Customising Gotests Generated Test v2

We’ll be following a process similar to when I last did this. I'm still using VSCode, so you’ll need to find the correct plugin for your editor.

Check out gotests and copy the templates directory to a place of your choosing
- git clone https://github.com/cweill/gotests.git
- cp -R gotests/internal/render/templates ~/.vscode/gotests/templates
In order for us to achieve the generated test using assert, this time we’re going to need to edit two files; function.tmpl and results.tmpl
- Overwrite the contents of function.tmpl with the contents of this Gist
- Overwrite the contents of results.tmpl with the contents of this Gist
Add the following setting to VSCode’s settings.json
- "go.generateTestsFlags": ["--template_dir=~/.vscode/gotests/templates"]

Now we can generate tests of functions that return the following:

only returns an error
a value, and an error
multiple values, and an error

I have the Go plugin for VSCode, so I just need to right click over a function to have the dropdown menu appear with an option to generate tests.

Only returns an error

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        wantErr error
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
            }
        })
    }
}

Returns a value, and an error

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
                return
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
                return
            }
            assert.Equal(t, tt.want, got)
        })
    }
}

Returns multiple values, and an error

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        want1   bool
        wantErr error
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, got1, err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
                return
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
                return
            }
            assert.Equal(t, tt.want, got)
            assert.Equal(t, tt.want1, got1)
        })
    }
}

Bonus: Only returns values

The below example is for a function that doesn’t produce any errors, but I’m including it for the sake of completeness.

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name string
        args args
        want bool
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := validateDogName(tt.args.name)
            assert.Equal(t, tt.want, got)
        })
    }
}

That’s It!

This is just a short update to an enhancement I previously made to gotests. The assert library is awesome for test cases and it’s great to have it autogenerated in my tests too.

Using Terraform to Manage AWS Patch Baselines at Enterprise Scale

Joseph Heyburn — Sun, 26 Jul 2020 00:00:00 +0000

If you’re new to AWS and patching principles then continue reading, else you can skip to juicy stuff below.

AWS Primer ☁️

EC2

Amazon Web Services (AWS) provides Cloud resources to those that require it, for a cost mind you. One resource in particular is a box-standard server known on AWS as EC2 Instances which you can secure shell (SSH) into and do whatever you like with it - the world is now officially your oyster! 🦪

Just like with your personal laptop, computer, mobile phone, etc., you need to patch your servers with the latest security updates to ensure that you are protected from any adversary gaining access to your devices.

If they were to gain unauthorised access, they can execute whatever they like on there; whether that be sniffing around your network for some sensitive files, or mining cryptocurrency on your paid-for resources. Just like you earlier - your world is now officially their oyster!

EC2 instances are operated by you, and as per the AWS Shared Responsibility Model, you are responsible for ensuring the software is kept secured.

SSM & Patch Manager

AWS have a service used to help with the administration of EC2 instances called Systems Manager (SSM*). SSM is a bit of a beast and covers a lot of different functionalitie, going deep on SSM is beyond the scope of this article - so you can read more about it on the AWS documentation.

* SSM formerly was an acronym for Simple Systems Manager - I guess it got more complex than they thought!

To help you keep your instances patched, AWS provided the Patch Manager - within there it contains a number of tools to help with this. These are:

Patch Baselines
- a policy which filters available patches for your instances to what should be installed on it
- filters which you can apply to a baseline include
- how many days since the patch was released
- the severity and classification of the patch
- or even explicitly deny individual patches if you know they introduce a bug
- e.g. only install security classified patches marked as critical severity which have been released more than 7 days ago
Patch Groups
- a label which joins together patch baselines, to what instances they should be applied to
- they apply to instances as a tag at a key named Patch Group

To then perform a patch event on an instance, you will need to execute an SSM Document known as AWS-RunPatchBaseline on your target instances.

An SSM Document is essentially an automation script that you can perform on one or more instances at a time, with conditions to apply different sets of scripts depending on the operating system (OS) platform (i.e. Windows / Linux).

AWS-RunPatchBaseline is one such example of a document that has both Windows and Linux stages, and it will check and install patches against the patch baseline that has been applied to your instance, via a patch group.

However if an EC2 instance is not assigned to a patch group, then AWS will pick the default baseline for that instances operating system

Say you had a security policy of ensuring instances must check for patches once a week - there’s no need to manually execute the SSM Document yourself. Maintenance Windows can help with that. They are essentially a cron expression that you define to then execute a task. In this case we can create a maintenance window to execute the AWS-RunPatchBaseline document on a weekly basis.

The resulting relationship of all the above looks like this:

Maintenance Windows ──┬── invokes ───> SSM Document ─── queries ───> Patch Baseline
                      |
                      |
                      └── targets ───> Patch Groups ─── assigned to ───> EC2 Instances

Infrastructure as Code Primer

Creating your resources through a web UI such as AWS Console is okay for learning in. But how do you ensure your infrastructure is repeatable across several environments? Not only that, how do you easily keep a history of the state of your application infrastructure?

This is where infrastructure as code (IAC) comes in. IAC is where all your infrastructure resources are defined in code, and published onto a source code repository of your choosing (GitHub, GitLab, etc.).

That way you can ensure the code that defines your infrastructure can be repeated across your different application environments, and be able to view history of code changes.

It also acts as a single source of truth that everyone in your team can depend on for what the application state looks like.

There are several IAC tools that you can use:

CloudFormation is AWS’s home-grown solution exclusively for AWS services
There are platform-agnostic tools such as Terraform
- written in its own proprietary language HashiCorp Configuration Language (HCL)
- I’ve talked about Terraform before in a previous post
Or for a language-agnostic, platform-agnostic tool - Pulumi is the one for you.

In the case of Terraform, it keeps the current state of your infrastructure stack in a file stored in a shared remote location. This is known as the state file, and it is used by Terraform to understand what resources Terraform is aware of in your platform.

Patch Baselines in an Enterprise Environment

When managing Terraform in an enterprise environment, it is a best practice to split up the infrastructure on the structure of the teams working on them, known as workspaces. This is something advised by Terraform themselves. For example, you would have a workspace for billing, and one for networking. That way you can ensure teams can work effectively without treading on each others’ toes.

The same principle can be applied to a security team that defines security policies, who will also write up patching policies. These policies will be followed by teams working in different workspaces to understand at minimum what patches should be installed on a server, and how quickly to install them.

Check out the University of Exeter’s own patching policy for an example.

For example, an enterprise may have a patching policy that states all patches with a severity marked as ‘Critical’ or ‘Important’ must be installed within 14 days of its release. If an out-of-band (OOB) patch is released, then it must be installed within 3 days.

These folk are the same lot who will also conjure up the Patch Baselines we learned about earlier.

They may deploy these patch baselines in a different Terraform workspace from yours. If that were the case, then if you wanted to refer back to these in your Terraform workspace, then the remote_state resource is something you might need.

# This example references a subnet_id created in another workspace
data "terraform_remote_state" "vpc" {
  backend = "remote"

  config = {
    organization = "hashicorp"
    workspaces = {
      name = "vpc-prod"
    }
  }
}

resource "aws_instance" "foo" {
  # ...
  subnet_id = data.terraform_remote_state.vpc.outputs.subnet_id
}

So the above works if all parties are using Terraform workspaces. But what if you’re using Terraform, and the security team (who are managing patch baselines) are using something different like the mentioned CloudFormation or Pulumi.

If you are creating patch groups in Terraform - you won’t be able to reference the patch baselines they’ve created, because they are in a different state file.

resource "aws_ssm_patch_group" "front_end_servers" {
  baseline_id = aws_ssm_patch_baseline.front_end_servers.id # Terraform does not know about this resource!
  patch_group = "front_end_servers"
}

You could of course replicate the patch baseline they’ve created in your Terraform code, but then that does not scale in an enterprise.

resource "aws_ssm_patch_baseline" "front_end_servers" {
  name = "front-end-servers"

  ... # variables removed for brevity
}

resource "aws_ssm_patch_group" "front_end_servers" {
  baseline_id = aws_ssm_patch_baseline.front_end_servers.id # Not good!
  patch_group = "front_end_servers"
}

This is what we want to avoid.

Data Sources For Patch Baselines

Data sources in Terraform allow you to pull resources in your platform that may exist in a separate state file to yours, or even a completely different IAC tool to Terraform, such as CloudFormation. What we need is a data source component for the aws_ssm_patch_baseline resource.

With a pull request I made to the terraform-aws-provider project - I added the ability to pull in patch baseline resources that exist in the AWS account you are targeting, meaning if the enterprise security team had deployed a patch baseline via a different stack, then you can reference that in your patch group.

# This resource is defined outside of the current working Terraform state
# So we are making a call to retrieve the ID of the resource in AWS
data "aws_ssm_patch_baseline" "front_end_servers" {
  owner            = "Self"
  name_prefix      = "FrontEndServers"
  operating_system = "CENTOS"
}

resource "aws_ssm_patch_group" "front_end_servers" {
  baseline_id = data.aws_ssm_patch_baseline.front_end_servers.id
  patch_group = "front_end_servers"
}

resource "aws_instance" "front_end_server" {
  ... # variables removed for brevity

  tags = {
    "Patch Group" = aws_ssm_patch_group.front_end_servers.id
  }
}

The documentation for its usage can be found here.

Another scenario might be where you want to reuse the patch baselines that AWS has created in your account. One such baseline is AWS-WindowsPredefinedPatchBaseline-OS-Applications, which patches both the Windows OS, and selected Microsoft applications installed on the Windows server.

Currently this is not the default baseline for Windows. So if you want to have this baseline assigned to a patch group, you could do something like this:

data "aws_ssm_patch_baseline" "windows_predefined_os_and_apps" {
  owner            = "AWS"
  name_prefix      = "AWS-WindowsPredefinedPatchBaseline-OS-Applications"
  operating_system = "WINDOWS"
}

resource "aws_ssm_patch_group" "active_directory" {
  baseline_id = data.aws_ssm_patch_baseline.windows_predefined_os_and_apps.id
  patch_group = "active_directory"
}

resource "aws_instance" "active_directory" {
  ... # variables removed for brevity

  tags = {
    "Patch Group" = aws_ssm_patch_group.active_directory.id
  }
}

Conclusion

This is a relatively short post - but it was just a quick introduction to AWS Patch Manager. I hope you find it helpful in applying your enterprise patching strategy to your product teams!

Reverse Proxy Multiple Domains Using Caddy 2

Joseph Heyburn — Wed, 10 Jun 2020 00:00:00 +0000

During lockdown, I’ve spent a bit of time improving our home network. The bigger picture of which I’ll write about in a future post. But for now, I came across some challenges with running Caddy 2 as a reverse proxy for multiple domains used internally.

If you’ve stumbled across this looking for the end config file for Caddy, then you can skip there.

Background

A few months back I kitted out my home with some Ubiquiti UniFi gear to fix our crappy Wifi at home, following inspiration from Troy Hunt and Scott Helme.

In order to administrate UniFi devices, you’ll need the UniFi Cloud Key which runs the Controller software to do just that. Although if you have a spare Raspberry Pi lying around, you can download the software for free and run it on there - this is what I did.

I’ve also wanted to protect my home network with a self-hosted DNS server, such as PiHole. I won’t go into depth about how that was done, but you can follow Scott Helme’s guide on how you can set the same up.

Both of these services can be accessed through web browsers at the IP address and ports where they are being hosted, such as http://192.168.1.10:8093/admin/ in the case of PiHole. Having to remember the IP address and the port can be a pain. We can front these services with a rememberable domain name which points to these services - of which I’ve written about in a previous post.

Securing with HTTPS

The web is evolving, and there is no reason why we should access services via insecure HTTP, that includes services that are only running on an internal network such as a home network. Web browsers nowadays give you a warning when you are connecting to website over an unencrypted connection.

Simply accessing over HTTP is not an option, when browsers present us with a huge warning message

Caddy is a web server similar to Apache, nginx, et al., but it is different in that it enables HTTPS by default and upgrades requests from HTTP to HTTPS. Managing certificates for HTTPS is a pain - so Caddy does that too, so long as you can prove you own the domain you are hosting requests at. We can use Caddy in a reverse proxy mode, allowing us to access services at endpoints such as https://pihole.domain.local in our browsers and forward them to the corresponding IP address hosting the service.

A reverse proxy is a service that simply forwards client requests onto the server on the clients behalf.

Proving Domain Ownership

Caddy uses Let’s Encrypt (LE) to provide certificates for domains. Since domains can be exposed publicly, we will have to prove ownership of the domain to have LE issue certificates on our behalf - so we’ll have to purchase the domain from a registrar. I talked about how to do this for this website in the past.

LE supports several challenge methods in order to prove you own the domain. This helps mitigates attacks by adversaries by claiming they own a domain such as natwest.co.uk - allowing them to create phishing attacks and steal banking information.

Since my network is only visible internally for the moment (i.e. the domain will only resolve to an IP address on my network) - I cannot use HTTP or TLS since these require the domain to resolve to a public IP address to a web server hosting a challenge file requested by LE. Therefore the only option I have is DNS challenge, where a randomly string generated by LE is placed into the TXT record of a DNS record to confirm ownership.

Building Our Caddy

For this exercise I’ll be using the latest version, Caddy 2, which allows for plugins to be built into the binary depending on your use case - including DNS challenge. This plugin isn’t included by default, so we’ll need to build our own Caddy binary. The tool to do this is called xcaddy.

UPDATE 2020-09-30

Looks like Caddy now comes with a nice web interface for downloading a Caddy binary with whatever plugins you desire. I just tested out the Linux arm 7 platform with just the github.com/caddy-dns/cloudflare plugin, and it worked as intended.

Once you've got the binary downloaded, copy it to the Pi then skip to Caddy Configuration.

To build using xcaddy, you need to make sure you have Go installed on your machine.

Note that I am building Caddy on my laptop, but running it on a Pi, so I will have to specify the architecture that Pi is running on so that Go can correctly build it.

# Download xcaddy
go get -u github.com/caddyserver/xcaddy/cmd/xcaddy

# Build custom Caddy binary for Raspberry Pi
GOOS=linux GOARCH=arm GOARM=7 xcaddy build --with github.com/caddy-dns/cloudflare

# Copy the new binary across to the Pi
scp caddy pi:/home/pi/caddy/

Caddy Configuration

The configuration I’m using can be seen below. Some things to note:

I’m using Cloudflare as the DNS name servers for the domain, even though I purchased my domain from namecheap
- This repeats an exercise I’ve done previously
- I’ve done this for two reasons:
- Caddy at the time of writing does not have a namecheap DNS challenge plugin
- It’s a proven method I know already
A CLOUDFLARE_API_TOKEN is required to have Caddy set the TXT record DNS challenge received from LE
- Guide for the same
Caddy is reverse proxying traffic to services running locally on the Pi
Caddy is not verifying the certificate being hosted by the UniFi Controller (insecure_skip_verify = true)
- The controller self-signs a certificate, and the reverse proxy has no means of establishing a chain of trust to verify the certificate
- It’s not a best practice to not verify the chain of trust, however I’m happy to accept the risk for now

Click here to see documentation on Caddy JSON config files.

Remember that the domain names aren’t actually publicly accessible. At a basic level we can update the /etc/hosts file of the machine we’re running on to add a record telling our machine how to resolve the domain.

sudo sh -c "echo \"192.168.1.10 pihole.joannet.casa\n192.168.1.10 unifi.joannet.casa\" >> /etc/hosts"

However, we’re already using PiHole as our own DNS server right? We can add the records there instead.

PiHole let’s you specify where local domain names should resolve to

The IP addresses you see above are pointing to the host running Caddy, the Raspberry Pi.

Verifying Caddy

Once the config file is built, you can perform a test run to confirm everything is working by executing this command.

sudo ./caddy run --config config.json

We need to execute using sudo so that we can expose the service to restricted ports 80 and 443 (HTTP and HTTPS respectively).

Now we have a memorable domain name fronting the service, and Firefox is happy that we’re encrypting the connection too. The certificate being produced in seen below.

Enabling Caddy Service

Since we’re not using the standard Caddy installation method, we will need to specify a service unit file so that Caddy starts up at the same time as the host - which is what PiHole and UniFi are doing currently.

First check to see if there is a stale service there already.

$ ls -la /etc/systemd/system/caddy.service
lrwxrwxrwx 1 root root 9 Jun 4 09:14 /etc/systemd/system/caddy.service -> /dev/null

If you get the above then remove the symlink so that we can create a file there.

rm /etc/systemd/system/caddy.service

Then populate the same file with the below, remembering the change the location of the Caddy config file to where it exists on your machine.

[Unit]
Description=Caddy Reverse Proxy
Wants=network-online.target
After=network.target network-online.target

[Service]
ExecStart=/usr/local/bin/caddy run --config /home/jdheyburn/homelab/caddy/config.json
Restart=on-abort

[Install]
WantedBy=multi-user.target

Finalise the new service with the two commands, enabling it on host startup and starting the service right now.

sudo systemctl enable caddy.service
sudo systemctl start caddy.service

Conclusion

For now I have all the above running bare-metal on one Pi instance, which produces a huge single point of failure in my network. In the future I’d like to see how converting these to Docker containers and having them distributed on multiple Pis would increase the resiliency of these services.

Until then, these basic but essential services are being hosted at easy to remember domains, transported over an encrypted connection, for me to easily administer the network for when it gets more complex over time.

Three Steps to Improve Hugo's RSS Feeds

Joseph Heyburn — Wed, 10 Jun 2020 00:00:00 +0000

This was originally posted on my personal blog from my Who Goes Blogging series - documenting my journey on setting up my Hugo-based portfolio website.

This walkthrough will be especially helpful if you have your own Hugo website, and want to cross-post across to dev.to using the Publishing from RSS feature.

I “fixed” the default RSS template used by Hugo to show the full article content, along with images, and also talk about how to have social media cards appear in the RSS items too.

I uploaded my completed RSS file to GitHub Gist 🚀

What is RSS?

RSS is a great way to “subscribe” to websites to ensure that you don’t miss content from them. The standard for how it is generated has been the same for decades now - however its still the best supported and most accepted way to receive updates. It is simply a standardised XML file that allows consumers such as RSS aggregators to parse the content of the post - for which there are many to choose from (RIP Google Reader).

My RSS aggregator of choice currently is inoreader - I’ll be using this in my examples

Hugo generates RSS XML files from a template that will loop over your content and expose this at an endpoint for RSS aggregators to subscribe to and periodically check for updates against. Hugo generates a bunch of RSS XML files at different sections of the website, allowing consumers to subscribe to the section that interests them most - you can even subscribe to tags if that XML is being generated!

Usually there is site top-level available at /index.xml - in the case of my site that would be https://jdheyburn.co.uk/index.xml. The source code for the template file Hugo uses to generate this is embedded in Hugo at this location.

The Problem

Some (but not all) websites only include the first paragraph of their post in an RSS update (aka summary), along with a message after that paragraph requesting the reader to visit the site in a browser to view the rest of the content.

This is a common feature of WordPress blogs - and it is done to have you load the full website and along with it, all the code for providing the owner with user analytics, and advertising - if they have it. So really they’re getting the benefit of being able to publish updates via a standardised approach (RSS), to then lure you onto the website.

Hugo kind of does this through the highlighted line in the templated RSS XML file:

<description>{{ .Summary | html }}</description>

.Summary will do what was just described, print out the first paragraph of your post. However unlike WordPress, it doesn’t actually say if there is more content. So users may not load your full website, instead thinking you’ve produced a very short article!

I’m not against the practice of previewing the post, if a blog provides revenue for the author(s) then you will need to force readers to view the full website for analytics and advertising (businesses gotta make money!).

Although I’m sure the initial purpose of RSS was only meant to be used to provide subscribers a pure text version of your content. At the end of the day, I believe the author should give the reader choice on mediums of consumption - and they accept the limitations that come from RSS.

So now, I’ll be making several changes to the Hugo template RSS XML file to produce the full site content, along with some other enhancements.

Rendering Full Site Content

Before we start editing files, we need to produce our own copy of the template RSS XML file. This is embedded directly within Hugo itself and not packaged in with your theme.

There’s every chance your theme is generating one for you. Run the below command to see if there is anything for you to copy from.
find themes/ -type f -name '*.xml'

We’ve discussed template lookup orders before in previous posts - we can overwrite the default of any file used in generating the website by placing it in your local project root.

We can start to make modifications to the rss.xml file by copying the default into the directory where Hugo will read it from.

# Assuming you are in project root
wget https://raw.githubusercontent.com/gohugoio/hugo/master/tpl/tplimpl/embedded/templates/_default/rss.xml -O layouts/_default/rss.xml

We know that line 35 containing .Summary is causing the issue here. All we need to do is change it to .Content - then Hugo will print out the entire content of your post. See the highlighted line below for the change.

 <item>
<title>{{ .Title }}</title>
<link>{{ .Permalink }}</link>
<pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
{{ with .Site.Author.email }}<author>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</author>{{end}}
<guid>{{ .Permalink }}</guid>
 <description>{{ .Content | html }}</description>
 </item>
{{ end }}
</channel>
</rss>

Let’s now view the changes in the RSS reader.

We can now see much more content is being produced - and with proper HTML this time!

Rendering images in RSS posts

Depending on how you are retrieving images - you may find that they are not displaying, and that the image alterative (alt) text is being displayed instead.

Note the highlighted alt texts of images

If you’re not already providing an alt field to your <img> blocks then you absolutely need to.

They help boost accessibility to your website should the user have a text-to-voice application reading your website.

They also provide a helpful description to your image in case it cannot load - which happens more often than you think!

Why it is breaking 😤

In part 4 of this series, we looked at placing resources for a post in the same directory as the content markdown file (also known as leaf bundles).

That means for a given directory tree structure:

$ tree content/blog
content/blog
└── some-blog-post
├── image-file-name.png
└── index.md

We would use the following shortcode to generate the <img> block in index.md.

{{< figure src="image-file-name.png" alt="Here is an image!" >}}

This produces the block:

<figure>
<img src="image-file-name.png" alt="Here is an image!">
</figure>

Since this contains no preceding forward-slash / in src, this becomes a relative path. This means the webserver will look for the image at the path relative to the current page.

So when we navigate to blog/some-blog-post/ in our web browser (as dictated from the above tree structure), your browser will request the image at blog/some-blog-post/image-file-name.png for you.

Why is this relevant? This same <img> block is being rendered in the description of your RSS XML file, including the relative image location. RSS is very simple and renders only complete URLs (e.g. https://jdheyburn.co.uk/blog/some-blog-post/image-file-name.png) - so when we just specify the relative path, it cannot find the image and thus prints out the alt text.

Note: if you specify images from a complete path such as /images/image-file-name.png under static/images, then this is not an issue for you.

Fix image rendering

The fix for this involves some Go magic. We need to prepend the permalink for the current post that’s being printed out to anything that contains a relative path <img> block.

We need to go back to the same line where we added .Content and change it to:

<description>{{ replaceRE "img src=\"(.*?)\"" (printf "%s%s%s" "img src=\"" .Permalink "$1\"") .Content | html }}</description>

This calls the Hugo function replaceRE which enables us to perform a find-and-replace using regular expressions (regex), and it takes three inputs:

PATTERN is the regex pattern used to find the text we want to remove
- "img src=\"(.*?)\""
REPLACEMENT is the text that we want to replace the PATTERN with
- (printf "%s%s%s" "img src=\"" .Permalink "$1\"")
INPUT is the string we want to perform the find-and-replace on
- .Content

In the example above, we’re placing a capturing group in the PATTERN so that it will capture the contents inside src and save them so that we can refer to it in the REPLACEMENT, while removing img src="".

REPLACEMENT is calling another Hugo function called printf to construct the replacement text. We’re pretty much just injecting the page .Permalink prior to the captured text from the PATTERN - so that RSS can then display the complete URL. While adding back the img src="" that was removed.

INPUT will be the page .Content, what we were printing out before.

Once we’ve made the change, images are now rendered properly!

UPDATE

Upon posting this, I noticed I needed to do the same thing for hyperlinks referencing headings with the same post. For instance the markdown below which renders a hyperlink to a heading…
[images to display in RSS content](#rendering-images-in-rss-posts)
Will render this in RSS:
<a href="https://jdheyburn.co.uk/#rendering-images-in-rss-posts">
Which does not take the reader to the right location at all. What we need to do is repeat the regex used above in rss.xml to catch these and replace them with the complete URL. See below for the snippet.
{{- $content := replaceRE "a href=\"(#.*?)\"" (printf "%s%s%s" "a href=\"" .Permalink "$1\"") .Content -}}
{{- $content = replaceRE "img src=\"(.*?)\"" (printf "%s%s%s" "img src=\"" .Permalink "$1\"") $content -}}
<description>{{ $content | html }}</description>

Adding cards for posts

Not dissimilar to social media cards, you can also have cards appear for your RSS posts in order to make them more attractive for readers to click on!

While in card view for inoreader, we can see featured images being displayed

You can do these in RSS via the enclosure tag…

<enclosure url="image-location.png" type="image/jpg"></enclosure>

If you have a look at our layouts/_default/rss.xml file, we don’t have this defined. We could go and add it in now, but I would like a unified social media image to be used across all platforms. This will enable me to focus my effort on generating a single social media card image that is shared across all methods of consumption.

Based on the above, my theme (hugo-coder), renders the tags required for social media cards through the internal template for twitter_cards. A snippet of the logic that determines this:

{{- with $.Params.images -}}
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="{{ index . 0 | absURL }}"/>
{{ else -}}
{{- $images := $.Resources.ByType "image" -}}
{{- $featured := $images.GetMatch "*feature*" -}}
{{- if not $featured }}{{ $featured = $images.GetMatch "{*cover*,*thumbnail*}" }}{{ end -}}
{{- with $featured -}}
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="{{ $featured.Permalink }}"/>
{{- else -}}
{{- with $.Site.Params.images -}}
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="{{ index . 0 | absURL }}"/>
{{ else -}}
...

The hierarchy of images used to render the social media card goes like this:

The first entry in the list of images found in the post front matter
Any image in the post leaf bundle containing feature in its name
Any image in the post leaf bundle containing either cover or thumbnail in its name
The default image defined at the site level params

Adding enclosure tags

We can effectively copy and paste the code being used for generating Twitter cards and modify it for enclosure tags. The snippet of code ultimately ends up looking like this 👇

{{- $pagePermalink := .Permalink -}}
{{- with .Params.images -}}
{{- $img := index . 0 -}}
<enclosure url="{{ printf "%s%s" $pagePermalink $img }}" type="image/jpg"></enclosure>
{{ else -}}
{{- $images := .Resources.ByType "image" -}}
{{- $featured := $images.GetMatch "*feature*" -}}
{{- if not $featured }}{{ $featured = $images.GetMatch "{*cover*,*thumbnail*}" }}{{ end -}}
{{- with $featured -}}
<enclosure url="{{ $featured.Permalink }}" type="image/jpg"></enclosure>
{{- else -}}
{{- with .Site.Params.images -}}
<enclosure url="{{ index . 0 | absURL }}" type="image/jpg"></enclosure>
{{- end -}}
{{- end -}}
{{- end }}

Walking through what this is doing:

Lines 1-4 will use the image defined in the images in the post front matter, if it exists
- I had to define {{-/* $pagePermalink := .Permalink */-}} on line 1 to make .Permalink available inside the with block at line 4
Line 7 will find an image containing feature in its name, at the post leaf bundle section, if an image is not found previously
Line 8 will revert to an image containing either cover or thumbnail, if an image is not found previously
Lines 12-13 will default to the image defined at the site level params

With lessons learnt trying to get images to display in RSS content, I am rendering the full URL of the image to ensure RSS is always able to retrieve it.

I’m going to follow lines 12-13 above and have a default card defined at the site level parameters. In my config.toml I’ll need to add the following…

[params]
images = ["images/jdheyburn_co_uk_card.png"]

…while also ensuring that I’ve placed the image under static/images/jdheyburn_co_uk_card.png.

Once all that is done and deployed - we will have an RSS post card that looks like this 🙌

We can now advertise our RSS subscription URL to readers. My theme hugo-coder supports this as a social button on the home page. We just need to add the below block to the config.toml file.

[[params.social]]
name = "RSS"
icon = "fas fa-rss"
weight = 6
url = "https://jdheyburn.co.uk/index.xml"
rel = "alternate"
type = "application/rss+xml"

Once done - our homepage will have the button displayed.

Summarising changes to RSS XML

You can view the gist below to see my complete RSS XML file after all the above has been added. If you wish to use it, you’ll need to place it in layouts/_default/rss.xml in your project directory.

Click here for a git diff view on the changes I made from the default RSS XML file.

Thanks for reading! 💯

Extending Gotests for Strict Error Tests

Joseph Heyburn — Tue, 07 May 2019 14:01:26 +0000

This is my first post on dev.to, X-posted from my new personal blog which can be found here. Hopefully I'll have dev.to publish from RSS
feeds once I've worked it out!

Happy to receive any feedback you may have! 😃

Strict Error Tests in Java

I love confirming the stability of my code through writing tests and practicing Test-driven development (TDD). For Java, JUnit was my preferred testing framework of choice. When writing tests to confirm an exception had been thrown, I used the optional parameter expected for the annotation @Test, however I quickly found that this solution would not work for methods where I raised the same exception class multiple times for different error messages, and testing on those messages.

This is commonly found in writing a validation method such as the one below, which will take in a name of a dog and return a boolean if it is valid.

public static boolean validateDogName(String dogName) throws DogValidationException {

    if (containsSymbols(dogName)) {
        throw new DogValidationException("Dogs cannot have symbols in their name!");
    }

    if (dogName.length > 100) {
        throw new DogValidationException("Who has a name for a dog that long?!");
    }

    return true;
}

For this method, just using @Test(expected = DogValidationException.class) on our test method is not sufficient; how can we determine that the exception was raised for a dogName.length breach and not for containing symbols?

In order for me to resolve this, I came across the ExpectedException class for JUnit on Baeldung which enables us to specify the error message expected. Here it is applied to the test case for this method:

@Rule
public ExpectedException exceptionRule = ExpectedException.none();

@Test
public void shouldHandleDogNameWithSymbols() {
    exceptionRule.expect(DogValidationException.class);
    exceptionRule.expectMessage("Dogs cannot have symbols in their name!");
    validateDogName("GoodestBoy#1");
}

Applying to Golang

Back to Golang, there is a built-in library aptly named testing which enables us to assert on test conditions. When combined with Gotests - a tool for generating Go tests from your code - writing tests could not be easier! I love how this is bundled in with the Go extension for VSCode, my text editor of choice (for now...).

Converting the above Java validateDogName method to Golang will produce something like:

func validateDogName(name string) (bool, error) {
    if containsSymbols(name) {
        return false, errors.New("dog cannot have symbols in their name")
    }

    if len(name) > 100 {
        return false, errors.New("who has a name for a dog that long")
    }

    return true, nil
}

If you have a Go method that returns the error interface, then gotests will generate a test that look like this:

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr bool
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: true,
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if (err != nil) != tt.wantErr {
                t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
                return
            }
            if got != tt.want {
                t.Errorf("validateDogName() = %v, want %v", got, tt.want)
            }
        })
    }
}

From the above we are limited to what error we can assert for, here any error returned will pass the test. This is equivalent to using @Test(expected=Exception.class) in JUnit! But there is another way...

Modifying the Generated Test

We only need to make a few simple changes to the generated test to give us the ability to assert on test error message...

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: errors.New("dog cannot have symbols in their name"),
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
                t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
                return
            }
            if got != tt.want {
                t.Errorf("validateDogName() = %v, want %v", got, tt.want)
            }
        })
    }
}

From the above there are three changes, let's go over them individually:

wantErr error
- we are changing this from bool so that we can make a comparison against the error returned from the function
wantErr: errors.New("dog cannot have symbols in their name"),
- this is the error struct that we are expecting
if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
- check to make sure the test is expected an error, if so then compare it against the returned error

Point 3 provides additional support if there was a test case that did not expect an error. Note how wantErr is omitted entirely from the test case below.

{
    name: "Should return true for valid dog name",
    args: args{
        name: "Benedict Cumberland the Sausage Dog",
    },
    want: true,
}

Customising Gotests Generated Test

Gotests gives us the ability to provide our own templates for generating tests, and can easily be integrated into your text editor of choice. I'll show you how this can be done in VSCode.

Check out gotests and copy the templates directory to a place of your choosing
- git clone https://github.com/cweill/gotests.git
- cp -R gotests/internal/render/templates ~/scratch/gotests
Overwrite the contents of function.tmpl with the contents of this Gist
Add the following setting to VSCode's settings.json
- "go.generateTestsFlags": ["--template_dir=~/scratch/templates"]

Once you have done that, future tests will now generate with stricter error testing! 🎉

Closing

I understand that the recommendations above will make your code more fragile, as the code is subject to any changing of the error message of say a downstream library. However for myself, I prefer to write tests that are strict and minimalise the chance of other errors contaminating tests.

I also understand that GoodestBoy#1 is probably a valid name for a dog! 🐶