DEV Community: BrettOps

I was on the Bright Founders Talk podcast!

Brett Weir — Mon, 17 Jul 2023 00:00:00 +0000

This week is a BrettOps first. I was honored to be a guest on Temy's Bright Founders Talk podcast, where I got to talk about my philosophy on infrastructure, work-life balance, and entrepreneurship.

As an add-on achievement, I actually found an opportunity to work in a prodigious Python packaging alliteration right in the middle of the interview. 😆

I want to say thank you to Temy for having me on, and to Matthew Wickham for being an excellent host.

Enjoy!

Meet cici-tools, a multi-tool for building GitLab CI/CD pipelines

Brett Weir — Mon, 03 Jul 2023 00:00:00 +0000

I've been working on a new project called cici-tools (pronounced "see-see"). It provides a set of command line tools for working with GitLab CI/CD files, where each tool does something useful in its own right. The direction of the project has changed quite a bit in trying to understand what is most needed and what can be reasonably built, but it's gotten to a good enough place to start talking about it.

This project is still experimental and the documentation is a work in progress. I can't promise that it works very well at the moment and would forgive you for not wanting to try it, but for the enterprising among you, I would love your feedback and to know if you found it useful.

Installation

cici-tools is available on PyPI, so you can install it with pip:

python3 -m pip install cici-tools

This will install the cici command into your local environment, which you can validate like so:

cici --version

$ cici --version
cici 0.2.5

Format CI files with `cici fmt`

The cici fmt tool mostly happened by accident while developing the cici tool. While it hasn't always been clear what I am building, it was always clear that it would modify GitLab CI files in some way.

In my early efforts, I wrote the tool to make as few changes as possible. Then I thought to create a new CI format that compiles back to GitLab CI. In the most recent iteration, cici now implements GitLab CI's schema directly in Python.

This latest approach has been the most time-consuming so far, but it has meant that reading a file in and writing it back out corrects the formatting in the process. Hence, cici fmt was born.

cici fmt can be run with or without files as parameters, like so:

cici fmt

$ cici fmt
.gitlab-ci.yml formatted

If no file is passed, it defaults to a file in the current directory named .gitlab-ci.yml.

When cici fmt is run, it will:

Add quotes to strings where the syntax would be ambiguous otherwise,
Reorder jobs in the file,
Fix indents and line spacing,
And some other random stuff.

Currently, it will also expand YAML anchors and extends keywords, because it shares a certain code path with cici bundle, even though it shouldn't. So it's not quite ready for prime time, but I'm working on it.

Once it's finished, it'll be pretty exciting to have a GitLab CI linter that doesn't require calling out to a GitLab instance.

Pin `include` versions with `cici update`

There's a question I've pondered for a long time. How does one create shared pipelines, push updates to everyone quickly, and also track changes over time?

There are two obvious choices, with their own obvious problems:

If everyone uses main / latest / what have you, everyone picks up changes immediately, and no one has any idea what versions are in use.
If everyone pins to a specific version, they know exactly what versions are in use and will likely never upgrade them unless they absolutely have to.

cici update offers a third choice: developers can continuously track the latest pipeline changes using a version-pinning tool so that they always know what versions they have, but are also able to pick up updates automatically.

Here's an example CI file:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/prettier
    file: include.yml
  - project: brettops/pipelines/python
    file:
      - lint.yml
      - setuptools.yml
      - twine.yml

Now call cici update:

cici update

$ cici update
brettops/pipelines/prettier pinned to 0.1.0
brettops/pipelines/python is the latest at 0.5.0

If you check back into that CI file, you'll see pinned versions:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/prettier
    ref: 0.1.0
    file: include.yml
  - project: brettops/pipelines/python
    ref: 0.5.0
    file:
      - lint.yml
      - setuptools.yml
      - twine.yml

That's it! That's all it does, but it helps me a lot.

Add the pre-commit hook to your project and cici update will run on every commit:

# .pre-commit-config.yaml
repos:
  # other hooks ...

  - repo: https://gitlab.com/brettops/tools/cici-tools
    rev: "0.2.5"
    hooks:
      - id: update

cici update pulls the latest GitLab Release for a pipeline project by date, so there are no versioning requirements for the upstream, but it does mean that the upstream needs to publish regular releases.

Bundle CI files with `cici bundle`

The cici bundle command splits a large CI file into many CI "bundles", one for each job. These bundled CI files have everything each job needs to run, so that each job can be consumed à la carte by downstream projects. It currently expands extends keywords, YAML anchors, and global variable declarations, with include expansion planned.

Let's use the brettops/pipelines/python pipeline as an example. It provides a large number of jobs that all depend on one or two base jobs. I won't attempt to reproduce its growing CI file, but here are a few jobs:

# .gitlab-ci.yml
# ...
python-black:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install black
    - $PYTHON -m black --check --diff .

python-isort:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install isort
    - $PYTHON -m isort --profile=black --check --diff .

python-mypy:
  extends: .python-base
  stage: test
  script:
    - *python-script-pip-install
    - $PYTHON -m pip install mypy
    - $PYTHON -m mypy "${PYTHON_PACKAGE}" --junit-xml report.xml
  artifacts:
    reports:
      junit: report.xml

python-pyroma:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install pyroma
    - $PYTHON -m pyroma -n "$PYTHON_PYROMA_MINIMUM_RATING" .
  rules:
    - exists:
        - setup.py
# ...

If you'd like to use only some of the jobs here, but not all of them, you've got yourself a pickle. Splitting it into multiple CI files means that they'll need to depend on another file containing .python-base or .python-base-small. If you try to include more than one of these split files, GitLab will refuse, citing diamond inheritance. Ouch!

To overcome this, cici bundle will act as a compiler and build final versions that are independent from one another. Here's a bundled version of the python-pyroma job from above:

# pyroma.yml
stages:
  - test
  - build
  - deploy

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_OPEN_MERGE_REQUESTS
      when: never
    - when: always

variables:
  # ...

python-pyroma:
  stage: test
  image: "${CONTAINER_PROXY}python:${PYTHON_VERSION}-alpine"
  variables:
    GIT_DEPTH: "1"
    GIT_SUBMODULE_STRATEGY: "none"
    PIP_CONFIG_FILE: "$PYTHON_PIP_CONFIG_FILE"
    PYTHON: "/usr/local/bin/python3"
  before_script:
    - |-
      if [[-n "$PYTHON_PYPI_GITLAB_GROUP_ID"]] ; then
        export PYTHON_PYPI_DOWNLOAD_URL="https://${PYTHON_PYPI_USERNAME}:${PYTHON_PYPI_PASSWORD}@${CI_SERVER_HOST}/api/v4/groups/${PYTHON_PYPI_GITLAB_GROUP_ID}/-/packages/pypi/simple"
        echo "Pulling PyPI packages from GitLab group ID $PYTHON_PYPI_GITLAB_GROUP_ID"
      elif [[-n "$PYTHON_PYPI_GITLAB_PROJECT_ID"]] ; then
        export PYTHON_PYPI_DOWNLOAD_URL="https://${PYTHON_PYPI_USERNAME}:${PYTHON_PYPI_PASSWORD}@${CI_SERVER_HOST}/api/v4/projects/${PYTHON_PYPI_GITLAB_PROJECT_ID}/packages/pypi/simple"
        echo "Pulling PyPI packages from GitLab project ID $PYTHON_PYPI_GITLAB_PROJECT_ID"
      fi
    - |-
      if [[-n "$PYTHON_PYPI_DOWNLOAD_URL"]] ; then
      cat > "$PIP_CONFIG_FILE" <<EOF
      [global]
      index-url = ${PYTHON_PYPI_DOWNLOAD_URL}
      EOF
      fi
  script:
    - $PYTHON -m pip install pyroma
    - $PYTHON -m pyroma -n "$PYTHON_PYROMA_MINIMUM_RATING" .
  cache: {}
  rules:
    - exists:
        - setup.py

The above job is fully expanded and no longer depends on another CI file.

Adopting cici bundle on your own project isn't very complex. The first thing you'll need to do is move your existing shared CI file into a new .cici/ directory as .gitlab-ci.yml:

mkdir -p .cici/
git mv include.yml .cici/.gitlab-ci.yml

The contents of your CI file can mostly stay the same, with the caveat that cici ignores hidden jobs (those that start with .), so those ones will not be bundled.

Ensure that every job in your CI file starts with the path of the project. For example, for brettops/pipelines/ansible, the jobs must start with ansible-. This restriction may be lifted in a future release.

It is also wise to prefix all global variables with the project path. So for brettops/pipelines/ansible, your variables should start with ANSIBLE_ (though this is not currently enforced by the tool).

Now you can run cici bundle:

cici bundle

$ cici bundle
pipeline name: python
bundle names: ['black', 'isort', 'mypy', 'pyroma', 'pytest', 'setuptools', 'twine', 'vulture']
created black.yml
created isort.yml
created mypy.yml
created pyroma.yml
created pytest.yml
created setuptools.yml
created twine.yml
created vulture.yml

As noted above, this creates new CI files. Add the pre-commit hook to your project, and your CI bundles will be rebuilt every time you try to commit:

# .pre-commit-config.yaml
repos:
  # other hooks ...

  - repo: https://gitlab.com/brettops/tools/cici-tools
    rev: "0.2.5"
    hooks:
      - id: bundle

Now you can use as many or as few components of your reusable CI file as you like:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/python
    file:
      - black.yml
      - isort.yml
      - mypy.yml
      - pyroma.yml
      - pytest.yml
      - setuptools.yml
      - twine.yml
      - vulture.yml

...which is awesome!

This tool is definitely still in development, so there is, again, no guarantee that it will work for any particular use case. Also, not all GitLab CI syntax is supported yet, but cici will loudly complain when it encounters syntax it doesn't recognize. Here is the currently supported syntax.

I'm slowly working to adopt cici-tools across the BrettOps pipeline catalog, starting with the more complex ones that really, really need this functionality. The old shared pipeline files are much harder to maintain, and while they will be kept around on a best-effort basis, it is highly recommended to transition over.

Conclusion

These three tools make up the cici command currently, but there are more on the way. A lot of possibilities have opened up by having a GitLab CI structurizer written in Python, which means I can now manipulate CI files all day long, in a type-safe, immutable way, using my favorite programming language.

Over the years, I've written a lot of scattered tools and scripts to perform automated edits and analyses to GitLab CI files, but I anticipate this effort will unify my approach a lot. Things that I expect to fall out of this effort include, but are not limited to:

Automatic pinning of job image versions
Extensions to GitLab CI, like sourcing job scripts from standalone bash scripts rather than only YAML files
Bundle-time resolution of included CI pipelines to reduce blast radius of bad pipeline changes
Converting GitLab CI/CD to / from other formats to some degree, both to provide an onramp onto GitLab from other CI systems, and to use GitLab CI syntax as a "write once, run everywhere" format
Simulating a mostly complete GitLab CI pipeline locally

Who knows what will come next, but I'm excited to see where this tool goes. Drop me a line at contact@brettops.io to tell me what you think! Happy coding!

Customize a Raspberry Pi image without any hardware

Brett Weir — Mon, 26 Jun 2023 00:00:00 +0000

Customizing SD card images is the worst—having to boot the device over and over again, and then grabbing the SD card to copy it back to your system, only to repeat it all over again the next time you need to make a change. Bleh.

Beyond that, this causes all kinds of problems in your operations story:

How would you reproduce your card image if you lost it?
How would you patch, upgrade, or otherwise modify the base image?
How do you know what is actually being delivered to customers?

I'm here to tell you that there's a better way! In this article, I'll show you a method for making all the changes you want to your embedded Linux operating system image without any actual hardware, even if the target architecture is different.

Yes, I am serious.

You'll learn how to prepare a chroot environment that will allow you to run commands inside your Raspberry Pi image as if you had the hardware running at your desk. It'll be possible to:

Install packages,
Customize the boot config,
Set up SSH keys,
And who knows what else!

The only limit is your imagination, and you can do it all right from the comfort of your computer.

This recipe targets Raspberry Pi because of its ubiquity, but can be adapted to almost any SD card image.

Prerequisites

A VirtualBox environment.
Vagrant installed.

Set up a Vagrant box

We'll complete the steps in this article inside a Vagrant virtual machine, because when you make mistakes while dealing with filesystem images, chances are, you'll end up with hung devices and needing to reboot often to recover.

Using Vagrant allows us to reboot a virtual machine instead and not disrupt our flow. It also means that you can complete this tutorial on any host machine, running Linux or not.

Create the following Vagrantfile in an empty directory, modifiying the values for vb.cpus and vb.memory as needed. Be sure to give the box as much oomph as you can spare, as your computer will get hungry when compressing and uncompressing the image:

# Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"
  config.vm.provider "virtualbox" do |vb|
    vb.cpus = 3
    vb.memory = "4096"
  end
end

With the Vagrantfile in place, start your Vagrant box and log in to it:

vagrant up
vagrant ssh

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/jammy64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/jammy64' version '20230302.0.0' is up to date...

...
...
    default: /vagrant => /home/brett/Projects/examples/card-image-ci
$ vagrant ssh
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-67-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage
...
vagrant@ubuntu-jammy:~$

Download the image

Go to the Raspberry Pi download page and find one you like. I chose Raspberry Pi OS Lite because the download is much smaller and I can add whatever I like to it. I also chose 64-bit because I have a newer Raspberry Pi.

You can download the exact image I chose by doing the following:

wget --progress=bar:noscroll https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-05-03/2023-05-03-raspios-bullseye-armhf-lite.img.xz

vagrant@ubuntu-jammy:~$ wget --progress=bar:noscroll https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-05-03/2023-05-03-raspios-bullseye-armhf-lite.img.xz
--2023-06-24 02:57:30-- https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-05-03/2023-05-03-raspios-bullseye-armhf-lite.img.xz
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 176.126.240.86, 46.235.230.122, 93.93.135.117, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|176.126.240.86|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 381558864 (364M) [application/x-xz]
Saving to: ‘2023-05-03-raspios-bullseye-armhf-lite.img.xz’

2023-05-03-raspios-bullseye-armhf-lite. 100%[=============================================================================>] 363.88M 9.43MB/s in 43s

2023-06-24 02:58:14 (8.44 MB/s) - ‘2023-05-03-raspios-bullseye-armhf-lite.img.xz’ saved [381558864/381558864]

Set up the filesystem

With the image downloaded, you're ready to prepare the image for access.

Uncompress the image

The first step is to uncompress the image if needed. The Raspberry Pi image downloaded above is compressed in the .xz format, so you'll need the xz and unxz commands. These should be pre-installed on the Vagrant box, but if they are not, they're easy to install:

sudo apt-get install -y xz-utils

Uncompress the image. You can add the -v flag to print the progress in real-time:

unxz -v 2023-05-03-raspios-bullseye-armhf-lite.img.xz

vagrant@ubuntu-jammy:~$ unxz -v 2023-05-03-raspios-bullseye-armhf-lite.img.xz
2023-05-03-raspios-bullseye-armhf-lite.img.xz (1/1)
  100 % 363.9 MiB / 1876.0 MiB = 0.194 44 MiB/s 0:42

You can tell it's uncompressed because it's now enormous, lol:

du -hs 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ du -hs 2023-05-03-raspios-bullseye-armhf-lite.img
1.3G 2023-05-03-raspios-bullseye-armhf-lite.img

Resize the image (optional)

If you're going to customize an OS image, you'll probably hit the existing disk size limit pretty quickly.

Install the amazing qemu-img utility:

sudo apt-get install -y qemu-utils

Inspect the image to find out what you're working with:

qemu-img info 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ qemu-img info 2023-05-03-raspios-bullseye-armhf-lite.img
image: 2023-05-03-raspios-bullseye-armhf-lite.img
file format: raw
virtual size: 1.83 GiB (1967128576 bytes)
disk size: 1.37 GiB

If you wanted to install, let's say, LibreOffice or something, it would need to be quite a bit larger. Let's resize the image file itself:

qemu-img resize 2023-05-03-raspios-bullseye-armhf-lite.img +2G

vagrant@ubuntu-jammy:~$ qemu-img resize -f raw 2023-05-03-raspios-bullseye-armhf-lite.img +2G
Image resized.

But that only resizes the image. You still need to grow the root partition that has all the stuff, and then expand the filesystem to fill the partition.

Let's find out which partition is which:

fdisk -l 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ fdisk -l 2023-05-03-raspios-bullseye-armhf-lite.img
Disk 2023-05-03-raspios-bullseye-armhf-lite.img: 3.83 GiB, 4114612224 bytes, 8036352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4c4e106f

Device Boot Start End Sectors Size Id Type
2023-05-03-raspios-bullseye-armhf-lite.img1 8192 532479 524288 256M c W95 FAT32 (LBA)
2023-05-03-raspios-bullseye-armhf-lite.img2 532480 3842047 3309568 1.6G 83 Linux

Looks like the second partition is what we want. Let's use the growpart command to expand it:

growpart 2023-05-03-raspios-bullseye-armhf-lite.img 2

vagrant@ubuntu-jammy:~$ growpart 2023-05-03-raspios-bullseye-armhf-lite.img 2
CHANGED: partition=2 start=532480 old: size=3309568 end=3842048 new: size=7503839 end=8036319

fdisk will now report the expanded size, which grew from 1.6G to 3.6G, as expected:

vagrant@ubuntu-jammy:~$ fdisk -l 2023-05-03-raspios-bullseye-armhf-lite.img
Disk 2023-05-03-raspios-bullseye-armhf-lite.img: 3.83 GiB, 4114612224 bytes, 8036352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4c4e106f

Device Boot Start End Sectors Size Id Type
2023-05-03-raspios-bullseye-armhf-lite.img1 8192 532479 524288 256M c W95 FAT32 (LBA)
2023-05-03-raspios-bullseye-armhf-lite.img2 532480 8036318 7503839 3.6G 83 Linux

Later on, after we mount the loopback devices, we'll be able to grow the filesystem to use the newly available space.

Add loopback devices

The next tool you're going to need is the losetup command, which allows you to manage loopback devices. We'll use it to create new loopback devices for the card image partitions and mount them to directories.

In my case, the first available device was /dev/loop6, but your specific device number may differ whenever this command is called. The losetup command returns the loopback device path, so we'll assign it to a variable to avoid referring to it directly:

DEVICE=$(sudo losetup -f --show -P 2023-05-03-raspios-bullseye-armhf-lite.img)
echo $DEVICE

vagrant@ubuntu-jammy:~$ DEVICE=$(sudo losetup -f --show -P 2023-05-03-raspios-bullseye-armhf-lite.img)
vagrant@ubuntu-jammy:~$ echo $DEVICE
/dev/loop6

You can use lsblk to inspect the newly available loopback devices:

sudo lsblk -o name,label,size $DEVICE

vagrant@ubuntu-jammy:~$ sudo lsblk -o name,label,size $DEVICE
NAME LABEL SIZE
loop6 3.8G
├─loop6p1 bootfs 256M
└─loop6p2 rootfs 3.6G

bootfs and rootfs are classic Raspberry Pi device labels, so it's looking like we're in good shape.

If you forget which device you had, run losetup -l to find it again:

losetup -l

vagrant@ubuntu-jammy:~$ losetup -l
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC
/dev/loop1 0 0 1 1 /var/lib/snapd/snaps/lxd_24322.snap 0 512
/dev/loop4 0 0 1 1 /var/lib/snapd/snaps/snapd_19457.snap 0 512
/dev/loop2 0 0 1 1 /var/lib/snapd/snaps/core20_1950.snap 0 512
/dev/loop0 0 0 1 1 /var/lib/snapd/snaps/core20_1822.snap 0 512
/dev/loop6 0 0 0 0 /home/vagrant/2023-05-03-raspios-bullseye-armhf-lite.img 0 512
/dev/loop3 0 0 1 1 /var/lib/snapd/snaps/snapd_18357.snap 0 512

In this case, it's /dev/loop6. Set the $DEVICE variable manually to continue:

DEVICE=/dev/loop6

If you opted to resize the image, you can grow the filesystem to its new size by running e2fsck and then resize2fs:

sudo e2fsck -f ${DEVICE}p2
sudo resize2fs ${DEVICE}p2

vagrant@ubuntu-jammy:~$ sudo e2fsck -f ${DEVICE}p2
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 49948/103584 files (0.1% non-contiguous), 341565/413696 blocks
vagrant@ubuntu-jammy:~$ sudo resize2fs ${DEVICE}p2
resize2fs 1.46.5 (30-Dec-2021)
Resizing the filesystem on /dev/loop5p2 to 937979 (4k) blocks.
The filesystem on /dev/loop5p2 is now 937979 (4k) blocks long.

Your filesystem is now its full size.

Mount partitions

The disk image is ready to be mounted to your local filesystem so that it appears as any other directory would, except that it will be the root filesystem of your Raspberry Pi image. Let's go!

You'll want to mirror the final filesystem layout as much as possible. That means opening up the root filesystem, finding /etc/fstab, and seeing what it has first.

Create the rootfs directory:

mkdir -p rootfs

Mount the rootfs partition onto the rootfs/ directory:

sudo mount ${DEVICE}p2 rootfs/

You should now be able to inspect the filesystem:

ls rootfs/

vagrant@ubuntu-jammy:~$ ls rootfs/
bin boot dev etc home lib lost+found media mnt opt proc root run sbin srv sys tmp usr var

From there, you can discover the /etc/fstab file:

cat rootfs/etc/fstab

vagrant@ubuntu-jammy:~$ cat rootfs/etc/fstab
proc /proc proc defaults 0 0
PARTUUID=4c4e106f-01 /boot vfat defaults 0 2
PARTUUID=4c4e106f-02 / ext4 defaults,noatime 0 1

This is promising! All we're missing are the contents of the rootfs/boot/ directory, which we might have guessed because the bootfs partition hasn't been mounted yet. Easy peasy—there's even already an empty rootfs/boot/ directory waiting for us, which we can make sure by doing the following:

ls rootfs/boot/

vagrant@ubuntu-jammy:~$ ls rootfs/boot/
vagrant@ubuntu-jammy:~$

So let's mount the bootfs partition there:

sudo mount ${DEVICE}p1 rootfs/boot/

All the fun Raspberry Pi stuff is there now:

vagrant@ubuntu-jammy:~$ ls rootfs/boot/
COPYING.linux bcm2708-rpi-zero.dtb bcm2710-rpi-zero-2-w.dtb bootcode.bin fixup4x.dat kernel7l.img start4x.elf
LICENCE.broadcom bcm2709-rpi-2-b.dtb bcm2710-rpi-zero-2.dtb cmdline.txt fixup_cd.dat kernel8.img start_cd.elf
bcm2708-rpi-b-plus.dtb bcm2709-rpi-cm2.dtb bcm2711-rpi-4-b.dtb config.txt fixup_db.dat overlays start_db.elf
bcm2708-rpi-b-rev1.dtb bcm2710-rpi-2-b.dtb bcm2711-rpi-400.dtb fixup.dat fixup_x.dat start.elf start_x.elf
bcm2708-rpi-b.dtb bcm2710-rpi-3-b-plus.dtb bcm2711-rpi-cm4-io.dtb fixup4.dat issue.txt start4.elf
bcm2708-rpi-cm.dtb bcm2710-rpi-3-b.dtb bcm2711-rpi-cm4.dtb fixup4cd.dat kernel.img start4cd.elf
bcm2708-rpi-zero-w.dtb bcm2710-rpi-cm3.dtb bcm2711-rpi-cm4s.dtb fixup4db.dat kernel7.img start4db.elf

Mount special host filesystems

Some commands you'll want to run, like those for installing packages, enabling services, or even connecting to the Internet, may require /dev, /proc, and /sys filesystems to be mounted inside the rootfs/ directory. They should currently be empty, except for rootfs/dev/:

ls rootfs/sys/
ls rootfs/proc/
ls rootfs/dev/

vagrant@ubuntu-jammy:~$ ls rootfs/sys/
vagrant@ubuntu-jammy:~$ ls rootfs/proc/
vagrant@ubuntu-jammy:~$ ls rootfs/dev/
console fd full null ptmx pts random shm stderr stdin stdout tty urandom zero

Once you've confirmed those directories exist and are empty, mount the filesystems by doing the following:

sudo mount -t proc /proc rootfs/proc/
sudo mount --bind /sys rootfs/sys/
sudo mount --bind /dev rootfs/dev/

Now those directories will be populated with a ton of magic stuff:

vagrant@ubuntu-jammy:~$ ls rootfs/sys/
block bus class dev devices firmware fs hypervisor kernel module power
vagrant@ubuntu-jammy:~$ ls rootfs/proc/
1 112 131 175 221 32 408 645 852 98 diskstats kallsyms misc slabinfo version_signature
10 1131 14 18 24 33 409 646 853 99 dma kcore modules softirqs vmallocinfo
102 1132 15 187 25 34 410 649 86 acpi driver key-users mounts stat vmstat
...
vagrant@ubuntu-jammy:~$ ls rootfs/dev/
autofs fd loop3 port shm tty15 tty28 tty40 tty53 tty9 ttyS20 ttyS5 vcs3 vcsu3
block full loop3p1 ppp snapshot tty16 tty29 tty41 tty54 ttyS0 ttyS21 ttyS6 vcs4 vcsu4
bsg fuse loop3p2 psaux snd tty17 tty3 tty42 tty55 ttyS1 ttyS22 ttyS7 vcs5 vcsu5
...

Enable ARM virtualization

At this point, we're almost ready to chroot into the environment. If we tried to use the Pi filesystem right now, we'd discover that none of the commands work:

./rootfs/bin/bash

vagrant@ubuntu-jammy:~$ ./rootfs/bin/bash
-bash: ./rootfs/bin/bash: cannot execute binary file: Exec format error

That's because the Raspberry Pi is an ARM platform, and our machine is x86-64.

That's okay though. We can add qemu-user-static to the filesystem so that ARM binaries run via QEMU when executed on our system:

sudo apt-get install -y qemu-user-static

If we run the ARM bash command again, we'll get... a different error!

vagrant@ubuntu-jammy:~$ ./rootfs/bin/bash
arm-binfmt-P: Could not open '/lib/ld-linux-armhf.so.3': No such file or directory

That's good. This error is different and expected because the bash binary is dynamically linked and has no way of linking to its nice ARM friends right now. We'll fix that next, when we finally chroot into our system.

Access the filesystem

It's the moment you've been waiting for. Let's run our fake Raspberry Pi!

You can "log in" to your filesystem with chroot, which will start an instance of the default shell:

sudo chroot rootfs/

$ sudo chroot rootfs/
root@ubuntu-jammy:/#

Wow, so cool! I can go home sweet home!

root@ubuntu-jammy:/# cd
root@ubuntu-jammy:~# ls
root@ubuntu-jammy:~# pwd
/root

I can more or less use it as though it were my machine. Give it a try.

You can run raspi-config, because why not:

raspi-config

You can install vim, because it's everyone's favorite code editor, right?

apt-get update -y
apt-get install -y vim

You can edit the hostname:

echo "my-cool-computer" > /etc/hostname
cat /etc/hostname

root@ubuntu-jammy:/# cat /etc/hostname
old-hostname
root@ubuntu-jammy:/# echo "my-cool-computer" > /etc/hostname
root@ubuntu-jammy:/# cat /etc/hostname
my-cool-computer

We can exit the chroot environment with exit:

exit

root@ubuntu-jammy:/# exit
exit
vagrant@ubuntu-jammy:~$

Pretty much anything that you could configure on the actual device can now be configured beforehand without the need to set up any hardware, which is pretty great.

It doesn't stop at hand edits either. You can pipe scripts into the chroot environment, if you have a set of tasks that you need to run often:

echo "ls /etc" | sudo chroot rootfs/ bash -

vagrant@ubuntu-jammy:~$ echo "ls -1 /etc" | sudo chroot rootfs/ bash -
NetworkManager
X11
adduser.conf
alternatives
apparmor.d
...

Or you can get even fancier. Ansible, for example, has a chroot connection plugin, so you can run Ansible playbooks to configure your Pi filesystem—again, without the need for actual hardware.

Anyway, let's assume we've set everything up the way we want. Yay! But we're not done yet. This filesystem won't do us any good until we repack it to be actually copied onto a real SD card. That comes next.

Tear down the filesystem

We now need to repeat everything we just did, but in the reverse order to get back to a packed-up, ready-to-ship card image.

Unmount filesystems

Unmount all the special filesystems we mounted:

sudo umount rootfs/dev/
sudo umount rootfs/sys/
sudo umount rootfs/proc/

Unmount the loopback devices for the bootfs and rootfs partitions:

sudo umount rootfs/boot/
sudo umount rootfs/

Delete loopback devices

Use losetup to detach the loopback devices that you created earlier to sever the final link between your disk image and your host operating system:

sudo losetup -d $DEVICE

Compress the image

Now that everything is disconnected, you can repack the SD card image with xz. This might take a significant amount of time, during which time your computer will purr like an angry kitten. Be patient, and add the -v flag if you like having literally any feedback on what's happening:

xz -v 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ xz -v 2023-05-03-raspios-bullseye-armhf-lite.img
2023-05-03-raspios-bullseye-armhf-lite.img (1/1)
  0.8 % 26.3 MiB / 32.3 MiB = 0.815 3.2 MiB/s 0:10

Eventually, it should finish. And then, huzzah! Look, our 3.6G image is compressed to 421MB:

du -hs 2023-05-03-raspios-bullseye-armhf-lite.img.xz

vagrant@ubuntu-jammy:~$ du -hs 2023-05-03-raspios-bullseye-armhf-lite.img.xz
421M 2023-05-03-raspios-bullseye-armhf-lite.img.xz

Ship it!

Your disk image is now in the exact same state that it was in when you downloaded it, except that it's actually not. The filesystem is now much larger and has all your super cool modifications ready to deploy to devices anywhere in the world.

Upload it to your FTP server, post it on USENET, or even put it on an actual physical SD card. Whatever you do, don't keep it to yourself, because the world needs your custom distribution of Sonic Pi, or whatever it was that you needed to customize your Pi image for.

Cleanup

If you want to save the card image you created, move it into the /vagrant/ directory. It will then be in the same directory as your Vagrantfile:

mv *.xz /vagrant/

Then exit the vagrant ssh session:

exit

vagrant@ubuntu-jammy:~$ exit
exit
$

Finally, destroy the box you created:

vagrant destroy

Conclusion

The tools and techniques discussed in this article are generally applicable to all kinds of embedded Linux disk images, not just Raspberry Pi. Once you know the filesystem layout, anything is possible!

You can even take all these instructions and dump them into a CI pipeline so that you never have to do this ever again!

Happy coding!

Hosting static sites with Cloudflare R2 and MinIO Client

Brett Weir — Mon, 19 Jun 2023 00:00:00 +0000

There are countless services nowadays for hosting static sites: GitHub, GitLab, Netlify, Surge, Porkbun, DigitalOcean, even Cloudflare.

If there are so many ways to get a static site online, why would anyone bother with setting up a plain ol' S3 bucket?

Well, there are lots of reasons:

Hosting multiple versions of a site. If you want v1, v2, and v3 at the same time, but don't want to commit your built sites to Git, then you need a writable location.
Hosting many sites from one domain. Maybe you're a hosting service! Or maybe you just provide hosting for multiple users in your company. You can build out a consistent workflow on top of S3 and host all the content in a single location.
Total control of how the sites are published. Maybe you want to vary available content by region, add authentication, use server-side analytics, or just configure how content is cached by the CDN. Building your own custom workflow will give you access to all the levers you need.

In this article, we'll develop a recipe for using Cloudflare R2 as a static site hosting service. You will:

Create a simple static site,
Publish the site to Cloudflare R2 with MinIO Client, and
Use Cloudflare Transform Rules to make your bucket behave more like a web server.

By the end of it, you will be the proud owner of a many-headed static site hydra that you'd never know was a simple S3 bucket underneath.

Prerequisites

The easiest way to meet all the prerequisites for this tutorial is to complete the previous tutorial in this series, Serve static assets with Cloudflare R2.

Here's a summary of the things you'll need:

A domain name proxied by Cloudflare.
A Cloudflare account.
An R2 bucket configured for public access.

For this tutorial, I created an R2 bucket called sites and made it accessible at sites.brettops.io. The domains and paths you will use for this article will be different, but the steps should otherwise be the same for you.

Step 0: Build a static site (optional)

If you came here because you already have a static site that you're ready to publish, use that and skip this section. For everyone else, you can set up an example site with me.

I'll be using MkDocs, because it's fast and simple and generates some nice boilerplate so that the site isn't completely empty. MkDocs is written in Python, so you'll need Python installed (which probably isn't an issue if you're on Linux).

Install the mkdocs package:

pip install mkdocs

This installs the mkdocs command. You can test that mkdocs is available by doing the following:

mkdocs --version

$ mkdocs --version
mkdocs, version 1.4.2 from /home/brett/.pyenv/versions/3.10.7/lib/python3.10/site-packages/mkdocs (Python 3.10)

Create a new mkdocs.yml project in the current directory:

mkdocs new .

$ mkdocs new .
INFO - Writing config file: ./mkdocs.yml
INFO - Writing initial docs: ./docs/index.md

One more thing: let's add a subpage for this site. You'll see why this is important later in the article:

cat > docs/about.md <<EOF
# About

Some more info about that.
EOF

You can run a local dev server to see your changes in action:

mkdocs serve

$ mkdocs serve
INFO - Building documentation...
INFO - Cleaning site directory
INFO - Documentation built in 0.05 seconds
INFO - [23:22:45] Watching paths for changes: 'docs', 'mkdocs.yml'
INFO - [23:22:45] Serving on http://127.0.0.1:8000/

Visit your local dev server at http://127.0.0.1:8000/. Our fancy new docs site isn't going to be anything to write home about, but it'll do the job.

When you're satisfied with your site, you can build a finished site for hosting like so:

mkdocs build

$ mkdocs build
INFO - Cleaning site directory
INFO - Building documentation to directory: /home/brett/Projects/examples/mkdocs-site/site
INFO - Documentation built in 0.05 seconds

This will create a site/ directory that contains our finished site, which is what we'll publish to Cloudflare R2.

Step 1: Deploy the site with MinIO Client

MinIO Client is far and away the best S3 command line tool I've found.

It's written in Go, so getting it onto your system is easy, and it supports a ton of commands that alternatives such as aws s3 or s3cmd simply don't have.

First, download and install the mc tool:

curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
sudo install mc /usr/local/bin/

$ curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 24.9M 100 24.9M 0 0 10.2M 0 0:00:02 0:00:02 --:--:-- 10.2M
$ sudo install mc /usr/local/bin/

For more installation options, see the official quickstart.

The mc command should be usable at this point:

mc --version

$ mc --version
mc version RELEASE.2023-06-15T15-08-26Z (commit-id=bf3924b58341eb7a71785653a29bf26ca9fac95e)
Runtime: go1.19.10 linux/amd64
Copyright (c) 2015-2023 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

mc allows you to configure a connection by creating an alias. You can have as many aliases configured as desired.

Use the mc alias set command to configure your R2 connection:

mc alias set NAME https://XXXXXX.r2.cloudflarestorage.com/ YYYYYY ZZZZZZ

Where:

NAME is the desired name for your alias. You'll have to type this often, so it's better to keep it short. I'll call mine r2.
XXXXXX is your Cloudflare account ID.
YYYYYY is your Cloudflare R2 access key ID.
ZZZZZZ is your Cloudflare R2 secret access key.

Once you've configured an alias, you can test it out or access it by prefixing the desired path with the alias:

mc ls r2/

$ mc ls r2/
[2023-06-17 18:38:11 UTC] 0B sites/

Hey, that's the sites bucket! Let's try accessing it!

$ mc ls r2/sites/
$

The above prints nothing. That's good! There's nothing in the bucket!

At this point, we've verified that the bucket works. Now we can put some stuff in it.

Note: The MinIO docs say to test the connection like this:
mc admin info r2
This does not work with Cloudflare. I don't know why.

It probably has to do with the fact that AWS S3 buckets have the bucket name in the domain, whereas Cloudflare R2 buckets have the bucket name in the path. 🤔

For hosting a static site, far and away the best tool for the job is the mc mirror command, which synchronizes files between two locations:

mc mirror SOURCE TARGET

In our case, we'll set it up to synchronize the local MkDocs site/ directory to the R2 bucket. We'll add the --overwrite flag so that it overwrites existing files if there are any differences, and we'll add the --remove flag so that it deletes files from the target that no longer exist in the source.

This will be great for when we create a pipeline to continuously publish changes to a site.

Double-check your command before running. You can very easily delete any existing data in the bucket if you're not careful with these commands.

mc mirror site r2/sites/latest/ --overwrite --remove

$ mc ls r2/sites/
$ mc mirror site r2/sites/latest/ --overwrite --remove
...site/sitemap.xml.gz: 1.38 MiB / 1.38 MiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 556.07 KiB/s 2s

If we browse to the published location, we'll be able to access the individual files we just uploaded:

We're not done yet though.

Step 2: Rewrite trailing slashes

You may have noticed that if you try to click on the About link, you get an error:

Web servers these days will almost always rewrite a URL that ends in a trailing slash (/) to an index.html file at the same path. In other words:

https://brettops.io/

Is the same page as:

https://brettops.io/index.html

This allows you to visit sites.brettops.io/latest/ and the contents of sites.brettops.io/latest/index.html, which is how it worked when we tested our site locally. Cloudflare doesn't do this by default, which is why our About link leads to nowhere.

We can tell Cloudflare to behave like this when serving our R2 site, using Rewrite URL Rules. That way, our links will work, and we'll be able to access our site at sites.brettops.io/latest/.

Go to the Cloudflare dashboard for your domain, and click Rules, then Transform Rules in the sidebar:

Click the Create rule button under the Rewrite URL tab:

Add an actually good name for your rule. It's the only way you'll be able to remember what it does without reading through your rule expressions:

Under If..., select Custom filter expression, and add the following expressions to the Expression Builder with an And between them:

Field	Operator	Value
Hostname	`equals`	`sites.brettops.io`
URI Path	`ends with`	`/`

Alternatively, you can edit the expression manually by clicking Edit expression and add the following:

(http.host eq "sites.brettops.io" and ends_with(http.request.uri.path, "/"))

Under Then..., then under Path, select the Rewrite to... option, select Dynamic, and add the following expression (see screenshot image below for reference):

concat(http.request.uri.path, "index.html")

This uses the concat function to append index.html to the URLs of matched request.

And under Query, select Preserve.

When you're ready, click Deploy. Then your new rule will be live:

At this point, you should be able to navigate to your site's URLs and see that they're accessible without adding index.html to the path:

Congratulations, we've built a fully functional static site hosting service!

Conclusion

Cloudflare R2 is deeply integrated with Cloudflare and easy to get started with. MinIO Client makes working with S3 clean and obvious. Together, they provide a slim, bare-bones hosting solution that is highly adaptable to different needs and use cases.

What's better, all of the steps taken in this article are easy to automate in a CI pipeline, allowing you to build a general solution for your team or company that scales with your users.

For simple use cases, I'd still rather use an off-the-shelf solution, but this is one of those tools that you keep in your toolbox, because you never know when you're going to need it. Sometimes, all you really need is something you can hack on and make your own.

The Bootstrapping Problem

Brett Weir — Mon, 05 Jun 2023 00:00:00 +0000

Infrastructure as code , or IaC for short, is often touted as a solution for all the world's problems.

Imagine for a moment: no more configuration drift, clean and consistent deliverables, and all operators finally on the same page. What more could anyone ask for? I myself am fanatic about IaC, and you can often find me raving about all the problems it solves.

But it's not all sunshine and roses. If you've spent any time trying to manage IaC deployments, you soon discover that all that fancy infrastructure tooling requires one thing to manage them: infrastructure.

By the time you've gotten to a point where you can even use Terraform, Kubernetes, or whatever other fancy tools, you likely already have multiple layers of deployments that are not and cannot be managed by the tools you've just adopted.

This is the Catch-22, the Achilles' heel, of infrastructure development.

This is the Bootstrapping Problem.

What is "bootstrapping"?

Bootstrapping in this context describes self-sustaining processes that grow in size and complexity by chaining successive stages. Examples abound:

New programming language compilers are compiled using older compilers, which are built using older compilers, which use compilers written in other languages, etc., all the way back until someone had to first write one in machine language.
Computers boot using successively larger bootloader, which each initialize the system just enough to support loading the next larger bootloader, until the operating system is fully loaded.
Software installers often have multiple stages so that earlier installers can update the later installers themselves.

Infrastructure goes through similar stages, whereby the introduction of a new capability to the organization makes it possible to introduce other capabilities that build on top. Each new layer depends on the previous layer and cannot exist without it:

The physical hardware must be working before you can install an OS.
A VM instance must be running before it can be provisioned with your software.
A container runtime must be installed before you can orchestrate containers.
And so on and so forth.

A case study

Let's see an example of the Bootstrapping Problem in action. Let's say you've got three infrastructure capabilities that you'd like to implement:

You want to host your own CI/CD runners. Bringing CI/CD runners home reduces your reliance on a third party and provides more control and visibility into what is critical infrastructure.
You want to use Terraform to deploy infrastructure. Adopting an immutable infrastructure strategy with Terraform leads to more predictable, flexible, and resilient deployments.
You want to use CI/CD to run Terraform. Standardizing around your CI/CD system simplifies your delivery system and makes cross-training your team members easier.

Each new capability sounds great, so you try to implement them.

You'll want to use Terraform to provision the runners, which you'll want to drive from CI/CD, which requires runners to be provisioned somewhere.

That, my friend, is a dependency cycle. Since this dependency cycle exists in the ordering of bootstrapping requirements, I like to call this situation a bootstrap loop.

Bootstrap loops are problematic because they must be resolved for bootstrapping to complete successfully. If the runners described above experience a failure, it will not be possible to use the existing Terraform automation to resolve it, because CI/CD will not be available to run it.

This can result in extended and difficult-to-resolve outages as teams scramble to find bootstrapping workarounds that circumvent existing processes.

How do bootstrap loops occur?

Bootstrap loops do not spontaneously occur, because it is impossible to provision infrastructure from nothing when it contains a bootstrap loop. Instead, they evolve over time as a result of changes in availability of infrastructure.

For the runner situation described above to occur, there must have existed another runner at some point to kick off the whole process:

One of a few things might have happened:

The team used to pay for Cloud Runner minutes, but doesn't anymore.
The team decided to tear down their old runner box after deciding that they no longer needed it.
The person who ran the old runner box left the company.

Whatever the reason, there used to be a failover mechanism to kickstart this loop, but that mechanism disappeared when the organization forgot why they had it.

The Bootstrapping Problem can thus be described as the tendency to create bootstrap loops on accident as part of normal operation.

Solving the Bootstrap Problem

Preventing the introduction of bootstrap loops is essential to building a resilient infrastructure stack that can recover from failure without operator intervention.

1. Know the end state

Have an end state in mind for your infrastructure. You won't get there if you don't know what you want:

What tools will you use?
How will things fit together?
Where will you host?
Who needs access?

Answering these questions well requires experience. If you don't know what you need, find someone who does. It'll help you prevent expensive early mistakes.

2. Know the order of dependencies

If you are part of an organization of any size or complexity, it may seem like a constellation of interdependent services just exists and has happily done so for all time.

But this is not the case!

Everything depends on something else. Though you may not have experienced a scenario in which a bootstrapping requirement is currently unsatisfiable, these scenarios do exist.

Your first line of defense is knowing what things your thing needs in order to start successfully, and protecting that information.

3. Write everything down

Document, document, document! The earliest parts in this process are the most critical to document. The steps (create an account, save recovery codes, create an API key) seem so simple and obvious, and that's why they're likely to be forgotten.

In addition, these early steps exist outside of the platform you're building, so it is unlikely that they can be effectively automated.

4. Define a bootstrap procedure

Define a plan for getting from where you are to where you want to be. You'll likely need to solve the Bootstrapping Problem multiple times to arrive at your end state.

Here's a hypothetical plan to bootstrap a Kubernetes cluster with Ansible:

Manually provision an SSH key to a newly installed Debian box. Add this key to secure keyring X and document at location Y.
Run the Ansible playbook to provision the box as a PXE server. Commit the playbook to Git.
Create a Debian pre-seed to install the OS and add an SSH key to every box that joins the network.
Run the Ansible playbook to provision these boxes as Kubernetes cluster nodes.
Pass the resulting kubeconfig to Terraform to bootstrap base cluster services.
...

By defining this procedure, we always know the order of dependencies, and we know what layer to consult to recover if the layer above it is broken.

5. Use a third party

It's okay—stand on the shoulders of giants!

Even if you plan to host your own GitLab instance, having a few projects on GitLab.com to kickstart your delivery infrastructure is a good idea.

Build there for a few iterations until you have an self-hosted production environment to run GitLab. Then keep the GitLab.com infrastructure in your back pocket so that you are ready to bootstrap again if you ever need to.

Conclusion

Bootstrapping is a universal problem.

Your work is a result of your earlier work, and other people's work, much like you are a product of your past experiences, the experiences of those around you, and all of human history.

It really is turtles all the way down.

In that case, we should strive to know where we came from, how we got to where we are today, and preserve our history to defend our future.

If you do that much, you'll be able to weather any storm that comes your way, operational or otherwise.

Loading config files in Python

Brett Weir — Mon, 29 May 2023 00:00:00 +0000

Config files are everywhere. There are lots of reasons your app might need to have one:

You have configuration that you want to persist beyond a reboot.
Your configuration represents a physical state; for example, it contains the settings for peripheral devices, a stored procedure for accomplishing a task, or maybe it expresses the layout of the live user interface.
Your app's configuration cannot be easily expressed as a series of variables. CI pipelines, workflows, etc. feature a lot of complex nesting, repeated blocks, and even internal linking.
You want the app to be able to persist its own changes to configuration, like changing of windows sizes, menu settings, or credentials. In this case, the config file is functioning more as a database than something the user writes.

In all of these cases, the structure of the config is very important and likely long-lived. Mistakes in your config syntax will be hard to undo, so it pays to have a plan upfront, and design for it to be extended and documented.

In this article, we'll learn how to load YAML config files in a way that is clean, easy to support, and easy to extend. We'll do this by creating our own YAML task automation syntax, which we'll call taskbook files:

# taskbook.yml

group: # name of group

tasks: # list of tasks
  - name: # name of task
    module: # module to use
    options:
      # key / value options

  # ...

We'll write a program to read them, which we'll call Taskable *. When finished, it will be easy to determine fields that are supported, validate config values safely, add more fields for future needs, and even access config values within our program as properties.

*Any similarity to Ansible playbook syntax, real or imagined, is purely coincidental. 😂

Create a command line tool

Let's create a file called taskable.py, to contain our implementation of Taskable:

# taskable.py
import argparse

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()

if __name__ == " __main__":
    main()

This provides the scaffolding for an argparse command line interface (for more info, see our article on Python CLIs).

You can run the script as follows:

python3 taskable.py

$ python3 taskable.py
usage: taskable.py [-h] file
taskable.py: error: the following arguments are required: file

To be able to read in a file, we need to create the file first, which we'll do next.

Create a taskbook file

I'll be using YAML for the config files because it's easy to read and I'm comfortable with it, but you can easily support JSON or TOML, as they offer similar APIs.

Create a taskbook.yml file and add the following:

# taskbook.yml
group: localhost
tasks:
  - name: copy file.txt to the place
    module: saucy.copy
    options:
      source: file.txt
      dest: /etc/file.txt

  - name: install a package
    module: cheesy.package
    options:
      name:
        - fzf
        - tree
      upgrade: true

  - name: enable the service
    module: lettuce.service
    options:
      enable: true
      start: true

At this point, we'll be able to run the following:

python3 taskable.py taskbook.yaml

However, nothing will happen because our app doesn't print anything yet.

Read in the YAML file

YAML files are easy to read with Python. There are multiple libraries available, but pyyaml is the de facto standard and is often installed on whatever system you're already on.

If you don't have pyyaml (or you're using a virtual environment because you're awesome), install it now:

pip install pyyaml

Then, in your taskable.py file, import the yaml package and read in the YAML file:

import yaml
...
data = yaml.safe_load(args.file)

Our taskable.py file so far:

# taskable.py
import argparse
import yaml

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)

if __name__ == " __main__":
    main()

At this point, you will be able to read in the YAML file, but there's still no output just yet. We could stop here and access its values as nested dictionaries and arrays, like so:

data["tasks"][0]["module"]

...but there are a couple problems with this.

First, there's no validation at all, so a malformed config file has unpredictable results. Second, strings are opaque data, so IDE auto-completion won't work; changing a field name will require manually searching through the code to do so; and I hope you never misspell a field name.

No, we can do a lot better, and we will, starting by building a model of our data in the next section.

Create the data model

We need a way to express our data format so that it's functional. For this purpose, I prefer to use attrs, which gives us data validation, makes our classes more performant, allows us to access our fields as properties with dramatically less boilerplate, and more.

Let's install attrs:

pip install attrs

Then add the following to your taskable.py file:

from typing import Any
...
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

Our taskable.py file so far:

# taskable.py
import argparse
from typing import Any

import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)

if __name__ == " __main__":
    main()

These two classes—Task and Taskbook—fully express the taskbook format. We won't instantiate them ourselves though, because we'll learn a method to do so automagically in the next section.

Structurize into models

"Structurize" is a $6 word (that I may have made up) that translates to, "load all your data into fancy model classes." I'm using it because "de-serialize" sounds awful and is harder to type. 😝

The easiest way to structurize your YAML data into attrs classes is by using the cattrs package. The simplest usage looks like this:

import cattrs
taskbook = cattrs.structure(data, Taskbook)

Let's add it to our taskable.py file:

# taskable.py
import argparse
from typing import Any

import cattrs
import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)
    taskbook = cattrs.structure(data, Taskbook)

if __name__ == " __main__":
    main()

That's all you need! cattrs will load the data into attrs classes after only being given the expected top-level class, which is Taskbook here.

If you need to tweak the behavior, cattrs provides a hook mechanism. It's a bit cumbersome, but it's easier than writing all the structurization code from scratch.

In the next section, we'll work on doing something useful with our data.

Use the data

At this point, we've fully structurized our data into classes, which means we can access our config data like this:

taskbook.tasks[0].module

This makes our code much easier to read and work with. Now we'll try using it to do stuff.

"Run" tasks

What good is our script if it can't run tasks? Let's add something to simulate "running" our hypothetical tasks, by adding the following to our taskable.pyfile:

...
print("group", taskbook.group)
for task in taskbook.tasks:
    print(f"run {task.module}: {task.name}")
...

Our taskable.py file so far:

# taskable.py
import argparse
from typing import Any

import cattrs
import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)
    taskbook = cattrs.structure(data, Taskbook)
    print("group", taskbook.group)
    for task in taskbook.tasks:
        print(f"run {task.module}: {task.name}")

if __name__ == " __main__":
    main()

Running our hypothetical tasks will output the following:

python3 taskable.py taskbook.yml

$ python3 taskable.py taskbook.yml
group localhost
run saucy.copy: copy file.txt to the place
run cheesy.package: install a package
run lettuce.service: enable the service

It's not hard to imagine connecting this skeleton to real module implementations to drive real task execution.

List used modules

Maybe we'd like to inspect our taskbook to find out what modules it uses. This would be useful, for example, to install necessary modules before running our tasks.

Let's add a -l / --list option to list used modules and exit without running the tasks:

...
parser.add_argument("-l", "--list", action="store_true")
...
if args.list:
    used_modules = sorted(list(set(task.module for task in taskbook.tasks)))
    for module in used_modules:
        print(module)
    return
...

Our taskable.py file so far:

# taskable.py
import argparse
from typing import Any

import cattrs
import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("-l", "--list", action="store_true")
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)
    taskbook = cattrs.structure(data, Taskbook)
    if args.list:
        used_modules = sorted(list(set(task.module for task in taskbook.tasks)))
        for module in used_modules:
            print(module)
        return
    print("group", taskbook.group)
    for task in taskbook.tasks:
        print(f"run {task.module}: {task.name}")

if __name__ == " __main__":
    main()

Running taskable.py with the list mode enabled:

python3 taskable.py -l taskbook.yaml

$ python3 taskable.py -l taskbook.yaml
cheesy.package
lettuce.service
saucy.copy

Woot! Static analysis! And it was easy to implement because our data model is so well-defined.

Summary

In this tutorial, we've built up a versatile config loading mechanism.

This setup works equally well for tiny command line utilities as it does for large and complex data formats files like task workflows, specifications, and so on. You can continue growing your application by adding new fields and new data models, and avoid the malignant technical debt that springs from a muddled early config implementation.

The best part? Your configuration will be stable and serve the bedrock and foundation of your application, now and in the future. In the words of Eric S. Raymond:

Smart data structures and dumb code works a lot better than the other way around.

— Eric S. Raymond

Stay smart, people! 😄

Six reasons to start a business

Brett Weir — Mon, 22 May 2023 00:00:00 +0000

There are good and bad reasons to start a business. If you're hoping to get rich quick, never work again, or be the next ZuckerJobs, you might be disappointed. The odds of becoming a billionaire are only somewhat better than being struck by lightning.

On the other hand, you don't have to spend all your time chasing huge valuations and grand exits. Many people, myself included, start businesses because it makes their lives work better for them.

In this article, I'll share some reasons why I started my own cloud consulting business, to show that even a modest business can add value to your life, and give you something to talk about forever.

Reason #1: Freedom of choice

How much free time do you want? How hard do you want to work? How much money is enough? How often will you get to visit your friends and family?

Whatever you do in life, there will be tens or hundreds of other things that you won't be able to do. This is our good friend, opportunity cost.

As a small business owner, I've found that I have a choice in how I spend my time. I don't ask for permission to take time off, I can plan meetings around my schedule, and no one is looking over my shoulder.

Reason #2: Continuity

Whenever I've left a company in the past, my whole life has been disrupted: the work changes, the tech changes, my insurance changes, and I have to start all over at my new job.

Starting a business has been a good solution for many of these life disruptions:

I maintain my own tech stack, tools, and knowledge base, which means I can deliver higher quality work more quickly for new clients.
I have a consistent online presence that doesn't need to be rebuilt all the time, so my network can grow with each new job instead of starting over.
My insurance doesn't change because it's provided by my own company.

With a stable foundation, I spend much less time worrying about retooling and spend more time doing useful work.

Reason #3: Saying "no"

In a corporate setting, there is almost always tension between what you want to build and what they want you to build, but when you're directly employed by an organization, you can't really say "no", or at least not for long.

When you work for yourself, it's not like that. If your business has any amount of traction, you have the option to say "no" to jobs that don't work for you, are outside your expertise, or are for clients that you just don't feel comfortable with.

Someone is asking you to violate the laws of physics? Your boss wants you to lie about the fitness of a product for a purpose? If it's your company, you can just say no, allowing you to dodge that bullet and possibly sleep better at night too.

Remarkably, exercising your right to say "no" can make you more attractive to customers you want to have: ones that deal fairly and treat you like a human.

Reason #4: Having a voice

There is a handbook for interacting with customers, writing about what you're doing, or what your organization values. To the extent allowable by law, you get to define all of these relationships, and you get to ask yourself: who am I? What am I building?

When you have your own small, closely held entity, you are, in large part, the business, and the identity of you and your business are difficult to distinguish (except for tax purposes).

Starting a business has forced me to communicate more often, to many audiences with different ideas about things. In doing so, I have continued to refine my messaging, understand my own identity, and put on paper what my values actually are. It is forever a work in progress, but it's a task that I did not expect would shape my life in the way that it has.

Reason #5: Knowing yourself

Having a business, your own business, forces you to think about what's truly important to you.

What's important enough that would be worth taking a massive pay cut? Or being stressed out? Or needing to learn a bunch about tax code? I personally know people who have tried to have their own business, and in the process, realized that, ya know, maybe having their own business isn't for them and that the working world wasn't so bad.

If you're looking for powerful self-reflection, starting a business provides lots of opportunities for that.

Reason #6: Learning

When you start a business, your job on any given day is always changing. Whether it's code-slinging, hardware, documentation, bookkeeping, marketing, filming, editing, paperwork; there is always something more to do, and often it's something you've never done before.

I've learned a lot from regular jobs, but I'd wager that every one year I've worked as a solo founder has been worth ten plugging away at a job somewhere. It's not enough to do your little part at a startup. You have to know everything, you have to understand everything, and you have to be able to communicate everything to everyone.

On top of that, every second counts. Every conversation, every email, every line of code counts, and there is a visceral connection between the work you do and what value it does or doesn't bring to the organization.

You just don't get that kind of feedback from a performance review.

Conclusion

You don't have to sell your soul to live a good life. It's okay to be small, and there are lots of ways to go about it:

Becoming a consultant
Building a tiny SaaS product
Franchising a business
Selling things online

It's different for everyone, and it won't be easy; I'm not getting rich any time soon with BrettOps. But I like to set my own hours, I don't like asking for permission, and I like having an interesting story to tell at the end of the day.

If any of that sounds like you, then starting your own small business might be just what the doctor ordered.

If my reasons for starting BrettOps align with yours, you can reach out. I love meeting new people.

What is Terraform?

Brett Weir — Mon, 15 May 2023 00:00:00 +0000

HashiCorp's Terraform is a powerful tool that enables higher-order infrastructure management across tools and providers. It does this by:

Providing a standardized, declarative front end for remote APIs
Supporting gradual change, rollback, and disaster recovery
Making your infrastructure self-documenting
Supporting immutable infrastructure deployment patterns

That all sounds great, but what does any of that mean? To understand Terraform, I find it best to understand what it replaces.

What about ...

Let's say you have a server somewhere. You need to put stuff on it. How should you approach it?

... hand edits?

The easiest possible solution is to hack away. They have a UI for a reason, right? Log right in, tweak the config, install some things, add your SSH keys, and away you go. This will work famously, and potentially for a very long time. That is, of course, until it doesn't.

Cracks will start to show sooner or later. Someone will ask you to change something. Your app will get more users. You'll need to migrate something to somewhere. You'll hand off some work to the new team member. The server's OS reaches end-of-life. Over time, these miscellaneous changes accumulate and are then forgotten as the team moves on to new work. This is called configuration drift , and it is ruinous because it is both catastrophic and inevitable. Any server that sits there long enough will eventually succumb to this fate.

... change orders?

One solution is to neurotically control and gate all changes to systems, requiring operators to fill out paperwork and get changes approved and signed off by multiple people.

This "helps" to prevent changes by demoralizing the team and making them resistant to working on or improving the system at all. If, despite that, someone does eventually try to make a change, then it does little to prevent erroneous or incidental changes.

... scripts?

Another solution is writing a script to do the deployment. This is a stunning advancement in practice, as it acknowledges that making changes yourself is probably a bad idea. It also helps you remember how you did something, why, and lets you do it again if needed.

A problem you quickly discover with this approach is the pleasure of coding for all possible circumstances. Writing a script to install a package? Okay, great. What should I do if the package is already installed? Should I install the newer version? Remove and reinstall? Purge existing configuration associated with the package? What if there's a conflicting package?

Coding for every eventuality on a long-lived system is nearly impossible.

Another problem is that, you're still running your script on a live, existing server, which means your install script isn't the only thing potentially making changes. Your system auto updater is also making changes, and so are the support techies who are panic fixing the bug that got introduced. By extension, there's no great way to roll back your changes, because there is no well-defined answer to knowing what changes have actually been made. Sure, your can restore the image from a backup, but what exactly does that do (or undo)?

... configuration management?

Configuration management (CM) tools like Chef, Puppet, and Ansible have their uses. In many ways, the benefits and drawbacks of CM tools are the same as for writing scripts.

They bring extra features to the table to support managing multiple machines, and come equipped with packages and modules to make coding idempotent changes easier. They still suffer from incomplete knowledge of system state, and you still gotta code for the great many situations that you may run into while provisioning.

So how do you manage a server over its lifetime?

It's a trick question: you don't.

Enter Terraform

Terraform is a different kind of tool entirely. It's a configuration management tool, but it approaches things so differently that it's really in a class of its own.

Terraform abandons the idea that things should be long-lived at all. It treats things it manages as immutable black boxes. You might be able to change a label or a tag, but for any change that might change the nature of a resource, like installing package or changing certain parameters, Terraform prefers to just delete it and start over.

What?!

This is a big departure from endlessly tweaking systems forever. Now, instead of operators getting mired in a sea of patches and updates and configs, resources become atomic building blocks that can be assembled and reassembled as needed.

To demonstrate how Terraform works, we'll deploy a DigitalOcean droplet.

Project setup

Terraform projects are written in their own configuration language, and Terraform files end in .tf. Terraform syntax is declarative, so ordering doesn't matter, and resources are defined at a directory, or "module", level, so you can spread your resources out across multiple .tf files in a directory, and name them whatever you like.

Create a directory called whatever you like, and add main.tf andterraform.tf files:

mkdir droplet-deployment/
cd droplet-deployment/
touch main.tf terraform.tf

Finding providers

The next thing to do is find a provider.

A provider is a plugin that wraps an API so that it can be modeled declaratively using HCL code. Companies, organizations, and individuals maintain Terraform providers because they understand the value that Terraform provides, and know that many developers will refuse to integrate services that don't offer Terraform providers. 😄

We can search the Terraform Registry, where we find the digitalocean/digitaloceanprovider, maintained by none other than DigitalOcean themselves. Let's add it to the terraform.tf file:

# terraform.tf
terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.28"
    }
  }
}

We can add a credential in whatever way desired to connect to our DigitalOcean account. I prefer environment variables:

export DIGITALOCEAN_TOKEN="dop_v1_XXXXXXXXXXXXXXXX"

Run terraform init to download the providers and configure your state:

terraform init

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding digitalocean/digitalocean versions matching "~> 2.28"...
- Installing digitalocean/digitalocean v2.28.1...
- Installed digitalocean/digitalocean v2.28.1 (signed by a HashiCorp partner, key ID F82037E524B9C0E8)

...
Terraform has been successfully initialized!
...

Then we can then declare our droplet resource according to the example:

# main.tf
resource "digitalocean_droplet" "app" {
  image = "ubuntu-22-04-x64"
  name = "app"
  region = "sfo3"
  size = "s-1vcpu-1gb"
}

We haven't created anything yet though. That comes next.

Making plans

Terraform has an excellent view of the systems that it manages. It knows:

The actual state of the system, by querying the target API,
The expected state of the system, by reading the state file, and
The desired state of the system, by reading the HCL files in your repository.

Through all these views of the system, Terraform is able to build a directed graph of changes required to achieve the desired state. This is known, in Terraform jargon, as a "plan".

After writing out your HCL code, you can create a plan using the terraform plan command:

terraform plan

On running the plan, Terraform will tell you exactly what changes it intends to make, so you can review them before Terraform does anything:

$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # digitalocean_droplet.app will be created
  + resource "digitalocean_droplet" "app" {
      + backups = false
      + created_at = (known after apply)
      + disk = (known after apply)
      + graceful_shutdown = false
      + id = (known after apply)
      + image = "ubuntu-22-04-x64"
      + ipv4_address = (known after apply)
      + ipv4_address_private = (known after apply)
      + ipv6 = false
      + ipv6_address = (known after apply)
      + locked = (known after apply)
      + memory = (known after apply)
      + monitoring = false
      + name = "app"
      + price_hourly = (known after apply)
      + price_monthly = (known after apply)
      + private_networking = (known after apply)
      + region = "sfo3"
      + resize_disk = true
      + size = "s-1vcpu-1gb"
      + status = (known after apply)
      + urn = (known after apply)
      + vcpus = (known after apply)
      + volume_ids = (known after apply)
      + vpc_uuid = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.

This is huge. Terraform allows you to see and understand the changes you're about to make before you make them. You know, so you don't do fun things like clobber the database or delete the firewall.

You can visualize your plan with the graph subcommand, which renders Graphviz code to render into an image:

terraform graph | dot -Tsvg > plan.svg

Here, we see that Terraform will instantiate the DigitalOcean provider and use that to create our droplet.

Applying plans

When you are feeling confident in the changes you're about to make, you can "apply" them, which executes your plan's proposed changes to arrive at the new state.

$ terraform apply
...
...
...
Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

digitalocean_droplet.app: Creating...
digitalocean_droplet.app: Still creating... [10s elapsed]
digitalocean_droplet.app: Still creating... [20s elapsed]
digitalocean_droplet.app: Still creating... [30s elapsed]
digitalocean_droplet.app: Still creating... [40s elapsed]
digitalocean_droplet.app: Creation complete after 42s [id=355390759]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

If we browse to the DigitalOcean dashboard, the resource is here, it's live, it's ready to go!

Well, not quite ready. I mean, there's no SSH key, no firewall, no monitoring. We'll look at that in the next section.

Making changes

Once you deploy a machine, the first thing you'll want to do is make changes to it, I guarantee you. And isn't that why we're here? 😄

Here's a fairly obvious and immediate change you'll want to make to this image. It has no SSH key, so there's currently no way at all to access it.

Normally, you might bust out ssh-keygen and have at it, but then you're back where you started, making hand-edits to things with no way to track it. Instead, add the hashicorp/tlsprovider to generate SSH keys from your Terraform configuration:

# terraform.tf
terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.28"
    }
    tls = {
      source = "hashicorp/tls"
      version = "4.0.4"
    }
  }
}

Since you added a new provider, you'll need to run terraform init -upgrade:

terraform init -upgrade

$ terraform init -upgrade

Initializing the backend...

Initializing provider plugins...
- Finding digitalocean/digitalocean versions matching "~> 2.28"...
- Finding hashicorp/tls versions matching "4.0.4"...
- Using previously-installed digitalocean/digitalocean v2.28.1
- Installing hashicorp/tls v4.0.4...
- Installed hashicorp/tls v4.0.4 (signed by HashiCorp)

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!
...

Then you'll need to add two resources: one to create the key itself, and another to add the key to DigitalOcean. Then modify the droplet resource to add the key to the machine:

# main.tf
resource "tls_private_key" "app" {
  algorithm = "ED25519"
}

resource "digitalocean_ssh_key" "app" {
  name = "app ssh key"
  public_key = tls_private_key.app.public_key_openssh
}

resource "digitalocean_droplet" "app" {
  image = "ubuntu-22-04-x64"
  name = "app"
  region = "sfo3"
  size = "s-1vcpu-1gb"

  ssh_keys = [digitalocean_ssh_key.app.id]
}

At this point, if we run graph, we should see this:

It's not required to run plan before apply, because Terraform will automatically create a plan anyway if it needs to. Let's just run terraform apply, and examine closely what's going on:

terraform apply

$ terraform apply
digitalocean_droplet.app: Refreshing state... [id=355390759]
...
...
...
Plan: 3 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

digitalocean_droplet.app: Destroying... [id=355390759]
digitalocean_droplet.app: Still destroying... [id=355390759, 10s elapsed]
digitalocean_droplet.app: Still destroying... [id=355390759, 20s elapsed]
digitalocean_droplet.app: Destruction complete after 21s
tls_private_key.app: Creating...
tls_private_key.app: Creation complete after 0s [id=d680d387c543c08d6d015ad893164bf97a6bf495]
digitalocean_ssh_key.app: Creating...
digitalocean_ssh_key.app: Creation complete after 1s [id=38310647]
digitalocean_droplet.app: Creating...
digitalocean_droplet.app: Still creating... [10s elapsed]
digitalocean_droplet.app: Still creating... [20s elapsed]
digitalocean_droplet.app: Still creating... [30s elapsed]
digitalocean_droplet.app: Still creating... [40s elapsed]
digitalocean_droplet.app: Creation complete after 42s [id=355414489]

Apply complete! Resources: 3 added, 0 changed, 1 destroyed.

You and I have added two resources, and modified the existing one, but Terraform is reporting that we're adding three and destroying one. 😲

This is because the droplet resource will not attempt to modify an existing virtual machine, for all the reasons we've discussed. Instead, it will delete the instance and create a new one from scratch. This means that our machine is fresh, completely blank, and unmodified.

This could be frustrating if we had saved a bunch of stuff on that machine. If, instead of doing that, we just accept that our whole machine will be destroyed periodically, then we have the closest thing in existence to a precisely known machine image, because we fully record the steps to create the machine , and execute those steps each time to create it.

If we extend this idea to our infrastructure, we can do things like:

Rotate the SSH keys on our boxes any time we like
Use a different SSH key for every box
Swap the OS image or modify our runtime environment whenever we need to

This gives us a lot of freedom to change things that we didn't previously have.

Cleaning up

Imagine yourself, messing around with dozens of clusters. You're so cool, you're spinning them up like nobody's business, trying out GPUs and storage drivers and all kinds of fun stuff. Only problem is, they're expensive! How do you clean up after yourself in a sane way?

With every other tool in existence, you get to clean it up yourself!

If you used Ansible, you get to write a destroy playbook.
If you wrote a script, you're now writing a destroy script.
If you did it by hand through the web UI, here are the keys to the building and make sure lock up when you're done.

Terraform, on the other hand, already knows what exists and how it would delete it. All it needs is to be told that you no longer want your resources anymore, with the destroy subcommand:

Warning: This will delete all your stuff!

terraform destroy

$ terraform destroy
tls_private_key.app: Refreshing state... [id=d680d387c543c08d6d015ad893164bf97a6bf495]
digitalocean_ssh_key.app: Refreshing state... [id=38310647]
digitalocean_droplet.app: Refreshing state... [id=355414489]
...
...
...
Plan: 0 to add, 0 to change, 3 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

digitalocean_droplet.app: Destroying... [id=355414489]
digitalocean_droplet.app: Still destroying... [id=355414489, 10s elapsed]
digitalocean_droplet.app: Still destroying... [id=355414489, 20s elapsed]
digitalocean_droplet.app: Destruction complete after 21s
digitalocean_ssh_key.app: Destroying... [id=38310647]
digitalocean_ssh_key.app: Destruction complete after 1s
tls_private_key.app: Destroying... [id=d680d387c543c08d6d015ad893164bf97a6bf495]
tls_private_key.app: Destruction complete after 0s

Destroy complete! Resources: 3 destroyed.

And that's it! All of your resources are gone, and you don't need to wonder if you're going to get billed next month for things you forgot to delete.

Summary

Because every resource is created via a declarative specification that is committed to version control , Terraform deployments are inherently self-documenting. There is no question as to what you deployed, when, and how.

Because Terraform knows how to resolve differences between states, there is also no question about what changes would be required to move from one state to another, and there is always a path to do so, via Terraform.

It's difficult to overstate how valuable this is. Terraform makes past, present, and future infrastructure states representable and gives you a complete, executable view of your systems, allowing you to understand, modify, and even reproduce systems in exquisite detail.

I can't imagine working on infrastructure without Terraform. It enables my work in a way that no other tool has up to this point, and it is the glue that holds all the rest of my tools, projects, and projects together.

If you haven't tried Terraform, I highly recommend it.

Download the completed example project:
https://gitlab.com/brettops/examples/droplet-deployment

DEV Community: BrettOps

I was on the Bright Founders Talk podcast!

Meet cici-tools, a multi-tool for building GitLab CI/CD pipelines

Installation

Format CI files with cici fmt

Pin include versions with cici update

Bundle CI files with cici bundle

Conclusion

Customize a Raspberry Pi image without any hardware

Prerequisites

Set up a Vagrant box

Download the image

Set up the filesystem

Uncompress the image

Resize the image (optional)

Add loopback devices

Mount partitions

Mount special host filesystems

Enable ARM virtualization

Access the filesystem

Tear down the filesystem

Unmount filesystems

Delete loopback devices

Compress the image

Ship it!

Cleanup

Conclusion

Hosting static sites with Cloudflare R2 and MinIO Client

Prerequisites

Step 0: Build a static site (optional)

Step 1: Deploy the site with MinIO Client

Step 2: Rewrite trailing slashes

Conclusion

The Bootstrapping Problem

What is "bootstrapping"?

A case study

How do bootstrap loops occur?

Solving the Bootstrap Problem

1. Know the end state

2. Know the order of dependencies

3. Write everything down

4. Define a bootstrap procedure

5. Use a third party

Conclusion

Loading config files in Python

Create a command line tool

Create a taskbook file

Read in the YAML file

Create the data model

Structurize into models

Use the data

"Run" tasks

List used modules

Summary

Six reasons to start a business

Reason #1: Freedom of choice

Reason #2: Continuity

Reason #3: Saying "no"

Reason #4: Having a voice

Reason #5: Knowing yourself

Reason #6: Learning

Conclusion

What is Terraform?

What about ...

... hand edits?

... change orders?

... scripts?

... configuration management?

Enter Terraform

Project setup

Finding providers

Making plans

Applying plans

Making changes

Cleaning up

Summary

Format CI files with `cici fmt`

Pin `include` versions with `cici update`

Bundle CI files with `cici bundle`