DEV Community: Seán Kelleher

Live Stream: Writing a Programming Language from Scratch

Seán Kelleher — Sat, 05 Jun 2021 22:03:57 +0000

At 17:00 GMT tomorrow (June 6) I'll be be streaming a coding session from Twitch, where I'll be implementing an interpreter for a custom programming language. This may be interesting to those who want to learn the basics of how programming languages are implemented, or for those who already know but haven't yet implemented one themselves. I'll also be providing commentary and answering any questions that may come up during the session.

Edit: The stream is now available at https://www.twitch.tv/videos/1047868433.

The Target Language

The language I'll be developing will be a custom C-style interpreted language. I intend to use this custom language as a skeleton for playing around with different syntaxes/styles for a further programming language project that I plan to work on.

The language itself will be a fairly typical C-like language consisting of a number of common features from modern languages:

Basic types: Booleans, integers and strings
Complex types: Lists, objects (maps) and functions/methods
Python-like strong, runtime typing, with optional typing in future
Python-like string/list access
Python-like built-in functions like len, keys, etc.
Rust-like control flow syntax (e.g. if true { ... } and for x in xs { ... })
Rust/JS-like array/object composition (spreading) and decomposition
Go-like error handling

Here are some example snippets:

x = 1
if x < 2 {
    print('T')
} else {
    print('F')
}

# Output:
#
# T

i = 0;
while i < 3 {
    print(i);
    i += 1;
}

# Output:
#
# 0
# 1
# 2

xs = ['b', 'c'];
ys = [xs.., 'a', xs..];
[_, _, ..zs] = ys
for z in zs {
    print(z);
}

# Output:
#
# a
# b
# c

fn half(n) {
    return n / 2
}

fn map(f, xs) {
    for i in 0..len(xs) {
        xs[i] = f(xs[i])
    }
}

print(map(half, [4, 5, 6]))

# Output:
#
# [2, 2, 3]

The Source Language

I'll be writing the interpreter in Rust using LALRPOP. The data structuring and pattern matching capabilities of Rust make it a perfect fit for building and walking ASTs.

Finally

Please feel free to join me and ask questions. Also, please let me know in the comments here if there's anything that you'd like to see and I'll do my best to cover it.

Docker `with_build_env.sh` Script

Seán Kelleher — Sat, 29 May 2021 15:01:02 +0000

In Docker for the Build Process, I introduced the idea of extracting the build phase of a project into a dedicated build_img.sh script, which uses the project's Dockerised build environment to build the "run" image for the project, allowing a project to be built on a host environment without installing any extra tooling other than Docker. In this post I'll expand on this idea and show how we can use this script as a basis for a new, reusable script that allows us to easily work with the build environment of our project.

Basic Usage

The idea here is to create a with_build_env.sh script which, at its most basic, will take a command and run it in the build environment, with the local project directory also mounted in this build environment. This means that we could, for example, run bash scripts/with_build_env.sh make without even installing make locally, but have all the artefacts output to our local project directory.

Headless usage

The first way this script can be run is the "headless" way, which is the approach that will be primarily used in the build pipeline. This runs a command in the build environment:

bash scripts/with_build_env.sh make posts

More complicated commands involving Bash operators can also be performed using the likes of bash -c:

bash scripts/with_build_env.sh bash -c 'make check && make posts'

Interactive usage

The second way this script can be run is the "interactive" way, which will only be used locally in general. This typically involves running an interactive shell in the build environment. This will allow you to run build environment tools on your project, even if they're not installed on your local environment.

This approach will usually be performed using sh/bash as the command, and using a flag to indicate that the current command should be run interactively:

bash scripts/with_build_env.sh --dev bash

This launches a Bash process in the build environment. This allows you to work within your local directory but use the tools from your build environment.

The reason for using a flag to distinguish interactive use from headless use is that build pipelines generally don't provide a certain mechanism (a TTY) for interactive use, so attempting to run a command in interactive mode in the build pipeline will fail.

Basic Implementation

The following is a basic with_build_env.sh script based on the ideas presented in Docker for the Build Process:

org='ezanmoto'
proj='hello'
build_img="$org/$proj.build"

bash scripts/docker_rbuild.sh \
    "$build_img" \
    "latest" \
    --file='build.Dockerfile' \
    .

workdir='/go/src/github.com/ezanmoto/hello'

docker run \
    --rm \
    --mount="type=bind,src=$(pwd),dst=$workdir" \
    --workdir="$workdir" \
    "$build_img:latest" \
    "$@"

This just performs two actions: it rebuilds the image for the build environment, if necessary, and then runs the provided command in the build environment. Rebuilding the image isn't necessary if you're using a remote image, but this step is useful for keeping your image up-to-date if your project defines its own build environment, as it's common for the requirements of projects to grow beyond base images quite quickly.

Using this script to build the current project can then be as straightforward as bash scripts/with_build_env.sh make. However, there are a number of drawbacks to the script as it currently is, such as the fact that the build runs as root and dependencies aren't cached. The following sections show optional, incremental steps that can be used to improve this basic implementation.

One additional note about with_build_env.sh is that, while the general idea and approach of each instance of the script is the same, each specific instance of the script may vary. This is because the utility of this script will generally change from project to project and language to language. For example, when a with_build_env.sh script wants to keep the build cache between Docker runs, the specifics of where the cache lives is handled directly by the with_build_env.sh script, and this changes depending on the language and build tooling being used.

Layers

The basic with_build_env.sh script presented above gives a lot of the benefits touted in Docker for the Build Process right out of the box. However, you are likely to encounter different issues depending on exactly how you'll be using the script. For example, the first issue that is likely to be encountered when working with this script is to try and run it in a build pipeline, which will probably result in Docker failing with an error output of the input device is not a TTY. Another problem is that the issued command is run by root by default which, while not necessarily a problem in and of itself, can cause a little friction when files created in the build environment are owned by root in the host directory. This section outlines the most common and problematic issues that may be encountered, along with possible resolutions.

Interactive use

The first issue that is often encountered when working with the with_build_env.sh script is the fact that Docker will need the --interactive and --tty flags when running locally, but will need them to be removed when running in the build pipeline. For this I generally introduce some basic argument parsing to the script to allow it to either be run in "dev" (interactive) mode, or not:

+docker_flags=''
+case "$1" in
+    --dev)
+        docker_flags="$docker_flags --interactive --tty --publish=$2"
+        shift 2
+        ;;
+esac
+
 org='ezanmoto'
 proj='hello'
 build_img="$org/$proj.build"

 bash scripts/docker_rbuild.sh \
     "$build_img" \
     "latest" \
     --file='build.Dockerfile' \
     .

 workdir='/go/src/github.com/ezanmoto/hello'

 docker run \
+    $docker_flags \
     --rm \
     --mount="type=bind,src=$(pwd),dst=$workdir" \
     --workdir="$workdir" \
     "$build_img:latest" \
     "$@"

Passing --interactive and --tty is sufficient for enabling this functionality, but I have a convention of also having the --dev option take a port forwarding argument, which is used to expose a port from the container. This is because I often find myself making use of ports to achieve some level of communication between the host and the container, such as running the main service I'm working on in the build environment and accessing it from the host. This addition does mean that the command for launching an interactive Bash shell has to be modified slightly, but it also means that we avoid having to restart the session in order to expose the container to the host:

bash scripts/with_build_env.sh --dev 3000:3000 bash

User mapping

This isn't as much of an issue in the build pipeline, but when using the with_build_env.sh script as presented above, one issue is that the default user in the build environment is root. This is fine in a functional sense - you'll generally be able to build the project without issue and won't run into ownership problems. However, it quickly becomes very tedious from a usability perspective - any files that are created in the container are owned by root, requiring sudo to remove them locally, and accidentally performing git operations can result in pain down the line as some of the files in your .git directory can have their ownership altered.

As a more usable solution I usually pass --user="$(id --user):$(id --group)" to the docker run command. This means that we're now running as our host user when using the build environment, so any files we create will have the desired ownership:

 docker run \
     $docker_flags \
     --rm \
+    --user="$(id --user):$(id --group)" \
     --mount="type=bind,src=$(pwd),dst=$workdir" \
     --workdir="$workdir" \
     "$build_img:latest" \
     "$@"

User mapping caveats

One issue with mapping the user as presented is that, while we're using the correct user and group IDs in the container for local development, this user doesn't actually exist within the build environment. This means that any tools that rely on the user's $HOME, including many Git and SSH-based commands, simply won't work. Such commands will either need to be run outside the build environment (such as git commit), or else the build environment will need to be set up with a functional user to carry out specific commands with sudo. It's not always necessary for the user to exist in the container, but if it is then that user can be created in the build environment with the desired user ID and group ID.

Caching

A convention of with_build_env.sh is to --rm containers after each use, which is useful to avoid accidentally depending on ephemeral aspects of the build environment. However, this means that any project dependencies stored outside the project directory must be re-downloaded with each launch of the build environment.

The solution to this is to cache the downloaded files. This is a big area where the script will change based on the programming language and tooling being used.

The first step is to create an area to persist the cached directories. For this I create named volumes with open permissions:

tmp_cache="$org.$proj.tmp_cache"
pkg_cache="$org.$proj.pkg_cache"
tmp_cache_dir='/tmp/cache'
pkg_cache_dir='/go/pkg'

docker run \
    --rm \
    --mount="type=volume,src=$tmp_cache,dst=$tmp_cache_dir" \
    --mount="type=volume,src=$pkg_cache,dst=$pkg_cache_dir" \
    "$build_img:latest" \
    chmod \
        0777 \
        "$tmp_cache_dir" \
        "$pkg_cache_dir"

The specific image being used here isn't important (it just needs to have chmod present) but we use the build environment for this for simplicity. We give the directory open permissions because volumes are owned by root when created by docker, and we want to allow any user to be able to download dependencies and run builds in the build environment.

It's also useful to prefix the name of the volume with the name of the project (ezanmoto.hello., in this example) to help isolate builds across project boundaries. See the "Caveats" section, below, for more details.

The last piece of the puzzle then is to mount the volume when using the build environment, and to let the build tools that we're using know about this:

 docker run \
     $docker_flags \
     --rm \
+    --env=XDG_CACHE_HOME="$tmp_cache_dir" \
+    --mount="type=volume,src=$tmp_cache,dst=$tmp_cache_dir" \
+    --mount="type=volume,src=$pkg_cache,dst=$pkg_cache_dir" \
     --user="$(id --user):$(id --group)" \
     --mount="type=bind,src=$(pwd),dst=$workdir" \
     --workdir="$workdir" \
     "$build_img:latest" \
     "$@"

We can see that the Go build tools are told about the cache directory through the use of the XDG_CACHE_HOME environment variable.

This setup allows the persistence of project dependencies across docker runs without baking them into the image or storing them locally. The cache can also be cleared by running docker volume rm ezanmoto.hello.tmp_cache ezanmoto.hello.pkg_cache, and the cache area will be automatically remade the next time with_build_env.sh runs.

Rust

The following shows how a similar caching mechanism could be implemented for Rust. This snippet is taken from another project of mine:

docker run \
    --rm \
    --mount='type=volume,src=dpnd_cargo_cache,dst=/cargo' \
    "$build_img:latest" \
    chmod 0777 /cargo

docker run \
    --interactive \
    --tty \
    --rm \
    --mount='type=volume,src=dpnd_cargo_cache,dst=/cargo' \
    --env='CARGO_HOME=/cargo' \
    --user="$(id -u):$(id -g)" \
    --mount="type=bind,src=$(pwd),dst=/app" \
    --workdir='/app' \
    "$build_img:latest" \
    "$@"

It can be seen that Rust uses the CARGO_HOME environment variable to locate the cache.

Caveats of cache volumes

It should generally be fine to use the same cache across projects, but some tools can encounter difficulties when the same caching volume is shared between projects (I've experienced this when using Go in particular, where I've encountered issues with the checksum database).

The simplest solution in this scenario is to prefixing the volume name with the name of the project in order to isolate volumes per project. However, do note that this issue can still arise even in a single project - for example, when changing between branches with different lockfiles. In this scenario it's helpful to remove the cache volume and allow with_build_env.sh to recreate it:

docker volume rm ezanmoto.hello.tmp_cache ezanmoto.hello.pkg_cache

Benefits

An immediate benefit of with_build_env.sh is that, assuming direct Docker access is available to us in the build pipeline, we can use the same commands locally those used in the build pipeline. This makes our pipeline easier to set up, but also means that we can run our local code in the build environment before we commit to help ensure that code that builds locally will also build in the build pipeline.

Some other benefits:

We have a simpler mechanism to open a shell in the build environment, which can be used for building, testing and debugging.
We can more easily work with a project without having to manually install its dependencies and required tools, which helps us avoid issues when trying to run the project for the first time. This is particularly helpful for onboarding new developers.
Many of the benefits outlined in Docker for the Build Process also apply. In particular, it now becomes much easier to start the interactive build environment.

Conclusion

The with_build_env.sh script allows us to abstract the process of running commands inside the build environment, making it easier for us to build, run and debug code in the replicated build environment. This enables greater parity with the build pipeline, and further helps to avoid the classic "it works on my machine" issue.

This article was originally published on seankelleher.ie.

Docker for the Build Process

Seán Kelleher — Sun, 25 Apr 2021 12:22:51 +0000

This article covers the use of Docker as part of the build process, for both local development and continuous integration builds. In particular, it addresses the practice of building projects using docker build, and the alternative approach of using docker run to perform builds.

Many articles already espouse the benefits of using Docker for local builds, as part of the "dev loop", but to recap a few:

Running your local builds in Docker gives you better parity with the central build pipeline when the latter is also using Docker.
Your local builds can run in a minimal environment, reducing dependencies and the risk of depending on tools and resources that aren't available in the central build pipeline. This also helps with the issue of using mismatched versions of tools.
It's possible to work on a project without having any of the tools/languages/frameworks installed locally. This is particularly useful when working on projects for a short term, or for working on different projects that depend on conflicting versions of the same tools.

Building in Docker Images

When using Docker as part of the build process, some projects build the project artefacts as part of an image build. A typical Dockerfile to define a service may look like the following:

FROM golang:1.14.3-stretch

RUN \
    apt-get update \
    && apt-get install -y \
        fortune-mod \
    && ln -s /usr/games/fortune /bin/fortune

ENV GO111MODULE=on

COPY go.mod go.sum /go/src/github.com/ezanmoto/hello/

WORKDIR /go/src/github.com/ezanmoto/hello

RUN go mod download

COPY . /go/src/github.com/ezanmoto/hello

RUN go build ./...

ENTRYPOINT ["./hello"]

One of the most obvious problems with this approach is that this image defines both the build environment and the run environment. This leads to a number of issues such as increased image size, an increased number of attack vectors (because of all of the extra packages required for the build), and the more subtle issue of mixing contexts (for example, is a particular dependency required for the build time or the run time?). Thankfully, Docker added multi-stage Docker builds, which allows us to separate the definition of the build image and the run image, and allows us to make the run image as minimal as possible.

However, another problem exists in the form of the COPY commands:

COPY . /go/src/github.com/ezanmoto/hello means that a new image needs to be built every time we want to use Docker to test a code change.
COPY . /go/src/github.com/ezanmoto/hello actually results in the entire codebase being copied every time that Docker is used to build the project (assuming something in the codebase has changed). This is more of an issue in bigger projects, where it can take a few seconds to copy all files, adding delays to the development loop.
docker build has to be run a lot in this setup for debugging purposes; every time you change the Dockerfile to try something different you'll need to rebuild the image, further compounding any delays encountered by re-downloading project dependencies and copying the codebase into the image. It's not very interactive.
COPY go.mod go.sum /go/src/github.com/ezanmoto/hello/ can be a pain-point when working with Docker images. With dependency managers such as npm install and go get that can handle automatically fetching packages, it's useful to be able to quickly try out different combinations. However, because the COPY step will cause all packages to be re-downloaded instead of just the updates, this adds friction, and a resistance to using Docker in the development loop, when it comes to testing and updating different dependencies.

Building in Docker Containers

A straightforward alternative is to build artefacts in containers instead of in images, utilising bind-mounts instead of COPY. For example, instead of the image-based build in the previous section, the following can be used:

build.Dockerfile:

FROM golang:1.14.3-stretch

RUN \
    apt-get update \
    && apt-get install -y \
        fortune-mod \
    && ln -s /usr/games/fortune /bin/fortune

ENV GO111MODULE=on

WORKDIR /go/src/github.com/ezanmoto/hello

build_img.sh:

proj='ezanmoto/hello'
build_img="$proj.build"
run_img="$proj"

bash scripts/docker_rbuild.sh \
    "$build_img" \
    "latest" \
    --file='build.Dockerfile' \
    .

mkdir -p target
docker run \
    --rm \
    --mount="type=bind,src=$(pwd),dst=/go/src/github.com/ezanmoto/hello" \
    "$build_img:latest" \
    bash -c '
        set -o errexit

        go mod download
        CGO_ENABLED=0 go build -o target ./...
    '

bash scripts/docker_rbuild.sh \
    "$run_img" \
    'latest' \
    .

Dockerfile:

FROM alpine:3.11.3

RUN apk add fortune

COPY target/hello /bin

ENTRYPOINT ["/bin/hello"]

There is a notable increase in the amount of code that's present here, but there are also some immediate benefits:

Fewer image builds: The build image only needs to be built once, until its actual definition changes, as opposed to being built every time there's a code change. The run image only needs to be built in the CI environment unless it's being tested locally. When debugging the run image, rebuilding the image multiple times has less friction because changing files in the project won't break the cache.
No redundant copying: The build is being run using the host directory so the delay incurred from copying the build context over and over is avoided, without needing to play around with .dockerignore.
Project dependencies that are located within the project directory (e.g. .node_modules) can be kept from previous runs, even if the dependency/lock files change, and even if the definition of the build environment changes. Volumes can be used to cache dependencies that exist outside the project directory.

Here are some other small benefits that I like:

Succinct definition of the build environment: This makes it easier for developers that prefer to develop locally to actually mirror the exact build environment that'll be used in the build pipeline, by following the steps outlined in build.Dockerfile.
The definition of the build environment, run environment and build instructions are all cleanly separated.

However, the biggest benefit that I get from this approach lies in the fact that I can use the build environment interactively, as outlined in the following section.

Interactive Build Environment

Now that the definition of the build environment has been separated from the build instructions and the definition of the run environment, it's possible to work interactively within the build environment with the local project directory mounted:

docker run \
    --interactive \
    --tty \
    --rm \
    --volume="$(pwd):/go/src/github.com/ezanmoto/hello" \
    'ezanmoto/hello' \
    bash

This starts a Bash session "in" the build environment, but with the project directory bind-mounted within the container. This means that any changes made inside the container are reflected locally, and vice-versa. This setup has numerous advantages when used as part of the development loop:

Assuming that the build pipeline uses the same build image, there is now almost total parity between the development environment and the build pipeline environment, meaning that there's less variability after a developer pushes changes. A developer can easily build locally, without creating new Docker images, in almost the same conditions as the build will occur in the build pipeline.
A subtle benefit of the previous point is that developers will be using the same version of dependencies as the build pipeline. For example, a new team member may use Go 1.14 locally but the build pipeline might still be on Go 1.12 for various reasons. New Go features will work locally but will break the build. Being able to run with the same versions of tools locally means that there is less chance of this occurring in practice.
Again following on from the previous point, updates are effortless, and safe across projects. For example, the image for the build pipeline may get updated from Go 1.12 to 1.15. It can often be a daunting task to update local installs, especially if there isn't a simple mechanism for removing old versions. With a specially-defined build image, local software doesn't need to be updated, but developers can instead simply work in the new build environment without installing the build tools locally. This also means that issues won't arise when a developer is working on two projects at the same time that require different versions of the same dependencies.
Developers don't have to install programs locally at all! As a somewhat extreme example, even though I work primarily in Go and Rust, I don't have them installed on my host. Instead, they're installed in my build images that I work in interactively. This also means that environments can be cleaned effortlessly after finishing a project - removing the build images for that project removes all the programming languages and tools that were being used for that project.
Setting up the development environment for a new developer is now handled automatically by the docker build process. Developers that want to replicate the setup locally can follow the instructions defined in build.Dockerfile manually.
Outside of aspects like bound directories, the build environment is decoupled from the host. This means that there's less risk of accidentally depending on things that are present in the host environment that aren't going to be present in the build pipeline. A simple example of this could be depending on Linux tools that are native to the Ubuntu development host when Debian is being used as the build environment.

Disadvantages

While the approach outlined above is my personal approach and preference for managing Docker images in a project, there are some notable caveats to it:

The presented approach depends on the use of bind-mounting volumes for the build environment. This can work well in practice when using Linux images on Linux hosts, but may be less practical on other platforms. Furthermore, build environments that nest Docker containers may encounter extra complexity with working with bind-mounted volumes, as paths are referenced relative to the host's filesystem.
When working interactively in the build environment you may quickly realise that users that you have defined locally don't exist in the build environment. Furthermore, trying to map users between these environments isn't the most straightforward - for example, do you do it at build time or at run time? This is perhaps one of the trickier aspects of running builds in containers instead of images, and could be a big argument in favour of the image approach. This is because any required users are usually better-defined in the image approach, typically being set up using the likes of adduser/useradd at the start of the image definition.
Some people may consider the bind-mount approach to be less "pure" because the container is exposed to the host, and may worry about the reproducibility of the setup, since reproducibility is one of the main benefits of Docker. However, the approach is no less reproducible than the approach that performs COPY . /src, as the entire host context is copied into the image. With both approaches, it is the responsibility of the build pipeline to ensure that the environment is clean and set up for reproducible results.
As mentioned in the previous section, this approach achieves almost total parity between the development environment and the build pipeline environment, but that doesn't mean that subtle differences can't emerge between the two.

For example, I encountered an issue with this setup when using Samson for CI/CD. By default, Samson doesn't actually do a full clone of a repository when running a new build/deployment, but instead creates a new Git worktree using symbolic links. This meant that the Git repository mounted in the build environment was referencing a file on the host, which couldn't be resolved in the build environment. I wasn't using worktrees locally, so this issue wasn't occurring in my local environment.

The resolution was straightforward, but less than ideal: to force a full checkout for projects that needed it. Still, it highlights the fact that differences between the local environment and the build pipeline can still manifest with this approach, and special attention should be made when working with symbolic links in particular.

Conclusion

Building code in Docker containers using docker run is generally faster, more space-conserving and more amenable to debugging than building code as part of a docker build. Separating the build environment definition from build instructions also allows for greater parity between the development environment and the build pipeline, and allows for easier management of project dependencies.

This article was originally published on seankelleher.ie.

Docker Build `--replace`

Seán Kelleher — Sat, 17 Apr 2021 13:32:24 +0000

This article covers "docker build replace", a script that I use in projects that contain Dockerfiles, which aims to help overcome some of the main drawbacks I encounter when using Dockerfiles in a project.

The docker build command is great for helping to achieve reproducible builds for projects, where in the past developers had to rely on setting up the correct environment manually in order to get a successful build. One big drawback of docker build, however, is that it can be very costly in terms of storage when running it multiple times, as each run of the command will generally leave unnamed images around. Cleanup can be straightforward, but requires continual pruning.

The need to remove unused images is particularly felt when trying to develop and debug Dockerfiles. Trying to come up with a minimal set of instructions that will allow you to run your processes the way that you want can require several docker build runs, even after you've narrowed down the scope with an interactive docker run session. Such a sequence may well require a few Docker image purges over the course of a session as your disk is continually overbooked by old and redundant images. This is compounded further if your Docker image makes use of a command such as COPY . /src, where each change to your root project will require a new image build.

This is where docker build --replace comes in, where Docker automatically removes the old image with the same tag when a new copy is built, and skips the build entirely if it's up-to-date. The only problem is that this flag doesn't currently exist.

`docker_rbuild.sh`

I wrote docker_rbuild.sh ("Docker replace build") to approximate the idea of docker build --replace by making use of the build cache:

# `$0 <img-name> <tag>` builds a docker image that replaces the docker image
# `<img-name>:<tag>`, or creates it if it doesn't already exist.
#
# This script uses `<img-name>:cached` as a temporary tag and so may clobber
# such existing images if present.

if [ $# -lt 2 ] ; then
    echo "usage: $0 <img-name> <tag> ..." >&2
    exit 1
fi

img_name="$1"
shift
tag="$1"
shift

docker tag "$img_name:$tag" "$img_name:cached" &>/dev/null
if docker build --tag="$img_name:$tag" "$@" ; then
    docker rmi "$img_name:cached" &>/dev/null
    # We return a success code in case `rmi` failed.
    true
else
    exit_code=$?
    docker tag "$img_name:cached" "$img_name:$tag" &>/dev/null
    exit $exit_code
fi

This tags the current copy of the image so that it can be reused for caching purposes, and then kicks off a new build. If the build was successful then the "cache" version is removed, theoretically meaning that only the latest copy of the image you're working on should be present in your system. If the build fails then the old tag is restored. If there are no updates then the cached layers are used to create a "new" image almost instantly to replace the old one.

With this, local images are automatically "pruned" as newer copies are produced, saving time and disk space.

Idempotency

One benefit of docker_rbuild.sh is the fact that, now that docker build isn't leaving redundant images around with each build, it is more practicable to use it in scripts to rebuild our images before we run them. This is useful when a project defines local images so that we can rebuild the image before it's used, every time that it's used, so that we're always using the latest version of the image without having to manually update it.

An example of where this can be convenient is when you want to use an external program or project that uses a language that isn't supported by your project. For example, the build process for this blog's content uses Node.js, but consider the case where I wanted to use a Markdown linter defined in Ruby, such as Markdownlint. One option is to add a Ruby installation directly to the definition of the build environment, but this has a few disadvantages:

It adds an installation for a full new language to the build environment just to support the running of one program.
It isn't clear, at a glance, that Ruby is only being installed to support one tool, and to someone new to the project it can look like the project is a combined Node.js/Ruby project.
The above point lends itself to using more Ruby gems "just because" it's available, meaning that removing the Ruby installation later becomes more difficult.

One way to work around this is to encapsulate the usage with a Dockerfile, like markdownlint.Dockerfile, and a script that runs the tool:

markdownlint.Dockerfile:

FROM ruby:3.0.0-alpine3.13

RUN gem install mdl

ENTRYPOINT ["mdl"]

markdownlint.sh:

if [ $# -ne 1 ] ; then
    echo "usage: $0 <md-file>" >&2
    exit 1
fi
md_file="$1"

proj='ezanmoto/hello'
sub_img_name="markdownlint"
sub_img="$proj.$sub_img_name"

docker run \
    --rm \
    --volume="$(pwd):/app" \
    --workdir='/app' \
    "$sub_img" \
    "$md_file"

This addresses some of the above issues:

Ruby isn't installed directly into the build environment, meaning that the build environment is kept focused and lean.
In markdownlint.Dockerfile, the Ruby installation is kept with the program that it's used to run, making the association clear.
The entire Ruby installation can be removed easily by deleting markdownlint.Dockerfile. This can be useful if we decide to replace the tool with a different linter, like this one written for Node.js. Another reason why we might remove markdownlint.Dockerfile is if the external project starts maintaining its own public Docker image that can be used instead of managing a local version.

Despite the benefits, there are two subtle issues with this setup. The first is that ezanmoto/blog_content.markdownlint will need to be built somehow before markdownlint.sh can be run, which may be a manual process, and it would also be a surprising error to find out that an image is missing for a script.

The second issue is that if one developer builds the local image, and a second developer updates the image definition, the first developer will need to rebuild their copy of the local image before running markdownlint.sh again or risk
unexpected results.

We can solve both of these issues by running docker_rbuild.sh before running ezanmoto/blog_content.markdownlint:

markdownlint.sh:

bash scripts/docker_rbuild.sh \
    "$sub_img" \
    'latest' \
    --file="$sub_img_name.Dockerfile" \
    .

docker run \
    --rm \
    --volume="$(pwd):/app" \
    --workdir='/app' \
    "$sub_img" \
    "$md_file"

This causes the image to be always be rebuilt before it's used, meaning that we're always working with the latest version of the image, and this build step will most often be skipped due to caching (though attention should be paid to
the commands used in the image build, as the use of commands like COPY can limit the effectiveness of the cache).

Use With `docker-compose`

I find docker-compose particularly useful for modelling deployments. However, like developing Docker images, getting the docker-compose environment correct can require continual fine-tuning of Docker images, especially for defining minimal environments. This can again result in lots of wasted space, especially when used with docker-compose up --build.

With that in mind, I now remove the build property from services defined in docker-compose.yml. This then requires the images to be built before docker-compose is called, which I normally handle in a script that will build all of the images used in the docker-compose.yml file before the file is called:

docker-compose.yml:

version: '2.4'

services:
    hello.seankelleher.local:
        image: nginx:1.19.7-alpine
        ports:
          - 8080:8080
        volumes:
          - ./configs/hello.conf:/etc/nginx/conf.d/hello.conf:ro

    hello:
        image: ezanmoto/hello

scripts/docker_compose_up_build.sh:

set -o errexit

proj='ezanmoto/hello'
run_img="$proj"

bash scripts/docker_rbuild.sh \
    "$run_img" \
    'latest' \
    .

docker-compose up "$@"

Conclusion

Having an idempotent rebuild for Docker images means that it's more feasible to rebuild before each run, much in the same way that some build tools (e.g. cargo) update any changed dependencies before attempting to rebuild the codebase. While Docker doesn't have native support for this at present, a script that takes advantage of the cache can be used to simulate such behaviour.

This article was originally published on seankelleher.ie.

DEV Community: Seán Kelleher

Live Stream: Writing a Programming Language from Scratch

The Target Language

The Source Language

Finally

Docker `with_build_env.sh` Script

Basic Usage

Headless usage

Interactive usage

Basic Implementation

Layers

Interactive use

User mapping

User mapping caveats

Caching

Rust

Caveats of cache volumes

Benefits

Conclusion

Docker for the Build Process

Building in Docker Images

Building in Docker Containers

Interactive Build Environment

Disadvantages

Conclusion

Docker Build `--replace`

docker_rbuild.sh

Idempotency

Use With docker-compose

Conclusion

`docker_rbuild.sh`

Use With `docker-compose`