DEV Community: Friedrich Kurz

Extending AstroJS Markdown Processing With Remark and Rehype Plugins

Friedrich Kurz — Sat, 17 Aug 2024 10:03:08 +0000

AstroJS

AstroJS is a framework for static website building that has a lot of exciting features like

reducing page size (and therefore faster loading speed);
being framework agnostic (React, Svelte, what have you are all supported);
lowering client-side scripting (a lot of processing is done on the server and AstroJS ships pure HTML by default);
being component-based and supporting component de-hydration (components are loaded lazily when needed);
support for MDX (Markdown + JSX), TypeScript, CSS Pre-Processing, and CSS frameworks (like Tailwind CSS); and
integrated SEO automations (like automatic sitemap generation, RSS feeds, and pagination)

🔗 (astro.build) Introducing Astro

Another great aspect of AstroJS is that it supports extensions of Markdown processing via Remark and Rehype plugins.

Remark and Rehype

Both Remark and Rehype are frameworks to transform documents by first transforming them to abstract syntax trees (AST) and then applying pluggable functions to the AST before converting it to the target format.

Remark parses Markdown and MDX files and applies transformations. Rehype parses HTML and applies transformations. Both frameworks have a large ecosystem of community plugins available.

💡 Remark and Rehype may be combined if we export HTML after transformation with Remark, for example by using the remark-rehype plugin. AstroJS automatically does this for use, so we do not need to worry about the transformation and may directly write plugins for Remark and Rehype.

Extending AstroJS With Remark and Rehype

A Remark or Rehype plugin is a higher-order function—a function returning another function in this case—that takes an options argument for configuration and then returns a function that receives the abstract syntax tree (AST) of the parsed Markdown or HTML document.

The abstract syntax tree for both tools looks something like this:

{
  type: 'root',
  children: [
    { type: ..., children: [Array], position: [Object] },
    // ...
  ],
  position: {
    start: { line: 1, column: 1, offset: 0 },
    end: { line: 44, column: 11, offset: 1201 }
  }
}

Plugins for both Remark and Rehype may be registered in the Markdown or MDX integrations in astro.config.mjs. Below, is an example configuration of the MDX integration with a Remark plugin theRemarkPlugin:

import { defineConfig } from "astro/config";
import theRemarkPlugin from "src/remark/the-remark-plugin"

// https://astro.build/config
export default defineConfig({
  integrations: [
    mdx({
      remarkPlugins: [theRemarkPlugin]
    })
  ]
});

💡 If we want to add a Remark or Rehype plugin to the Markdown processor, we have to add it to markdown.remarkPlugins respectively markdown.rehypePlugins of the configuration object passed to defineConfig instead.

Example: Enabling `rehype-mermaid` in AstroJS

Lets, assume we want to add rendered Mermaid diagrams to our page by transforming the source code in fenced code blocks like the following:

```mermaid
flowchart LR

A[Hard] -->|Text| B(Round)
B --> C{Decision}
C -->|One| D[Result 1]
C -->|Two| E[Result 2]
```

Thankfully, a Rehype plugin to generate Mermaid diagrams rehype-mermaid from HTML <pre>or <code> elements with Mermaid diagram source code already exists. Adding it to our Markdown or MDX processing pipeline is therefore as easy as installing it with NPM and adding it to the configuration as shown above.

🔗 (github.com) Rehype Mermaid Plugin

The plugin's documentation, however, states that it only converts <pre class="mermaid"> and <code class="language-mermaid"> elements to rendered diagrams in SVG format.

And, unfortunately, if we add a fenced code block with language mermaid to our post, the element produced by remark is a <pre> element with a class list like astro-code github-dark. To get our rendered Mermaid diagram, we have to add the class mermaid to this element.

With the tooling provided for Rehype via projects in the unist ecosystem, this is luckily rather straight forward. To help finding the nodes of the AST that we are concerned with and transforming them as need, we may use the visit function from unist-util-visit.

This function, may—for simple scenarios like ours—be defined with only two parameters

ast (the AST's root) and
a visitor function that applies our changes.

Omitting some implementation details, our Rehype plugin addMermaidClass to add the mermaid class to Mermaid code blocks then may be written as follows:

import { visit, CONTINUE } from "unist-util-visit"
import type { Plugin } from 'unified';
import type { Root, Element } from 'hast';

/* ... */

const visitor = (node: any) => {
    /* ... */
}

const addMermaidClass: Plugin<void[], Root> = () =>
  (ast: Root) => visit(ast, visitor)


export default addMermaidClass

Let's now take a look at the implementation of the visitor function.

const dataLanguageMermaid = "mermaid"
const typeElement = "element"
const tagNamePre = "pre"
const classMermaid = dataLanguageMermaid

const isPreElement = (node: any) => typeof node.type !== undefined && node.type === typeElement
    && node.tagName !== undefined && node.tagName === tagNamePre
    && node.properties !== undefined && node.properties.dataLanguage === dataLanguageMermaid

const visitor = (node: any) => {
  if(!isPreElement(node)) {
    return CONTINUE
  }

  const element = node as Element
  const properties = element.properties
  const className = properties.className as Array<string>
  properties.className = [...className, classMermaid]

  return CONTINUE
}

The important things to note are

the first conditional which assures that we are visiting the AST representation of a <pre> element and skips all other elements and
the in-place modification of the node.properties.className array.

💡 Note that the visit function should return an object of type Action instead of the visited node. In this case, we return the CONTINUE action that tells Remark to continue traversing the AST.

The last step is to import the plugin and add it to the configuration of the MDX integration before the rehypeMermaid plugin. My AstroJS at the time of writing looks like this.

import { defineConfig } from "astro/config";
import icon from "astro-icon";
import tailwind from "@astrojs/tailwind";
import mdx from "@astrojs/mdx";
import rehypeMermaid from 'rehype-mermaid'
import addMermaidClass from "src/rehype/add-mermaid-class"

// https://astro.build/config
export default defineConfig({
  integrations: [
    tailwind(),
    icon(),
    mdx({
      rehypePlugins: [addMermaidClass, rehypeMermaid]
    })
  ]
});

💡 Adding the addMermaidClass plugin before the rehypeMermaid plugin is important since the array order defines the execution order of the plugins and we need to change the class name before passing the AST on to the rehypeMermaid plugin.

The rendered result of the Mermaid diagram is shown below:

More on AstroJS

Check out my blog for more posts about AstroJS:

How to define and work with a Rust-like result type in NuShell

Friedrich Kurz — Fri, 21 Jun 2024 11:51:00 +0000

Motivation

Rust—among other modern programming languages—has a data type Result<T, E> in its standard library, that allows us to represent error states in a program directly in code. Using data structures to represent errors in code is a pattern known mostly from purely functional programming languages but has gained some traction in programming languages that follow a less restrictive language paradigm (like Rust).

The idea of modelling errors in code is that—following the functional credo—everything the function does should be contained in the return value. Side effects, like errors, should be avoided wherever possible.

This is a nice pattern, since it forces you to address that code may not succeed. (If you don't ignore the return value, of course. :D) Combining the use of Result return types with Rust's pattern matching also improves code legibility, in my opinion.

Error handling in NuShell

NuShell is a very modern shell and shell language that draws some heavy inspiration from Rust and that I really enjoy writing small programs and glue code in (especially for CI/CD pipelines).

In contrast to Rust, NuShell has a try/catch control structure to capture and deal with errors. There is no result type in the standard library at the time of writing.

NuShell's try/catch, moreover, has the major downside, that you cannot react to specifics of an error, since the catch block doesn't receive any parameter like an exception object, that would allow us introspection on what went wrong.

So what can we do? Well, we may just define a Result type ourselves and use it. Since NuShell also has pattern matching using the match keyword, we can write some pretty readable code with it.

Consider, for example, malformed URLs when using the http command (in this case the protocol is missing):

nu> http get --full www.google.com
Error: nu::shell::unsupported_input

  × Unsupported input
   ╭─[entry #64:1:1]
 1 │ http get --full www.google.com
   · ────┬───        ───────┬──────
   ·     │                  ╰── value: '"www.google.com"'
   ·     ╰── Incomplete or incorrect URL. Expected a full URL, e.g., https://www.example.com

The code above will crash. As mentioned before, we could use try/catch. But the problem remains, how do we enable the calling code to react to errors?

use std log 

try {
    return http get --full www.google.com # Missing protocol
} catch {
    log error "GET \"www.google.com\" failed."
     # Now what?
}

Using a result type (and some convenience conversion functions into ok and into error), we can write a safe http get function as follows:

def safe_get [url: string] { 
  try {
    let response = http get --full $url
    $response | into ok
  } catch {
    {url: $url} | into error
  }
}

We could use it in our code like this:

nu> match (safe_get "https://www.google.com") {                                    
    {ok: $response} => { print $"request succeeded: ($response.status)" },
    {error: $_} => { print "request failed" }
}
request succeeded: 200

And for the failure case:

match (safe_get "www.google.com") {                                              
    {ok: $response} => { print $"request succeeded: ($response.status)" },
    {error: $_} => { print "request failed" }
}
request failed

Now the calling code can react to failure by disambiguating the return values and processing the attached data.

Addendum

Here are the helper functions into ok and into error for completeness sake.

export def "into ok" [value?: any] {
    let v = if $value == null { $in } else { $value }

    {ok: $v}
}

export def "into error" [cause?: any] {
    let c = if $cause == null { $in } else { $cause }

    {error: $c}
}

More on Nushell

Check out my blog for more posts about Nushell:

Dev Containers on Kubernetes With DevSpace

Friedrich Kurz — Tue, 14 May 2024 05:33:33 +0000

Motivation

It is probably undisputed that a robust, easy to use, and quickly set up development environment is one of the key drivers of high developer productivity. Typical productivity boosters include, for example,

reduced onboarding time;
lowered maintenance efforts;
homogenization of development environments (e.g. across CPU architectures); and
easy access to required configurations and resources.

A major concern when talking about development environments is, of course, packaging and distribution because we want to quickly and reliably ramp up—and also tear down—development environments in order to assure the productivity gains described earlier.

A good way of handling the packaging problem is containerization. This is the case because container images not only may be used to package the tooling required for the development process and allow us to run workloads within a pre-configured environment; but, they may also be versioned, uploaded, and downloaded using established infrastructure components (i.e. image repositories).

Unsurprisingly, there are quite a lot of tools that operate in the space of providing development setup using containers. Some of the more prominent ones include

(I personally refer to these tools as dev container tools.)

DevSpace

DevSpace, the topic of this post, also falls in the category of dev container tools. It has the advantage over the aforementioned choices, however, that it provides a very generic and customizable approach to bootstrapping development environments. Three of the most striking features to me are the

configurable out of the box SSH server injection, as well as the
two-way sync capability between local host file system and development container file system, and that
DevSpace development containers run on Kubernetes.

The first point is great because remote development using SSH is a tried and true approach with lots of tooling support (e.g. VS Code, IntelliJ, and Neovim all support remote development via SSH). Consequently, it lets developers stay flexible w.r.t. their editor/IDE choice.

Having a fast and reliable two-way sync mechanism is also great to have because it gives us quick and easy, albeit limited, persistence (limited to the synced directories of course) without having to configure persistent volumes or having to mount directories as you would using only Docker to run a development container. Since containers should be ephemeral, this is a very easy way to keep changes that you want to persist stored away safely without much additional setup.

As for the last point, running on Kubernetes is a great way to organize and quickly ramp up, as well as tear down, development resources. E.g. we may use Kubernetes namespaces to scope resources for a specific developer; additionally, we may use Kubernetes abstractions to provide and manage access to

configuration and credentials,
downstream network resources (e.g. giving access to third-party systems via external name services), or
physical resources (like GPUs).

Lastly, you gain the capability to conveniently lift and shift your development from a local Kubernetes cluster to a remote Kubernetes cluster by simply changing the Kubernetes configuration.

DevSpace, moreover, is an official Cloud Native Computing Foundation (CNCF) project with over 4k stars on GitHub (at the time of writing) and it is, therefore, very likely to be actively worked on in the foreseeable future.

Basic Development Workflow with DevSpace

💡 This section is a short tutorial illustrating the development workflow with DevSpace. If you want to try it out, please have a look at the proof of concept repository available on my GitHub. It includes set up instructions for starting a dev container on AWS including code for infrastructure setup and an example dev container Dockerfile.

To use DevSpace, we first have to install it. For example, on an Linux/ARM64 machine:

curl -L -o devspace "https://github.com/loft-sh/devspace/releases/latest/download/devspace-linux-arm64" && \
sudo install -c -m 0755 devspace /usr/local/bin

ℹ️ See here for more installation options.

Assuming that we already have access to a Kubernetes cluster and that we have pointed kubectl to use the corresponding context, we should—as a best practice—create a unique namespace—e.g. devspace—for our development environment and then tell DevSpace to use the targeted context and namespace.

$ kubectl create namespace devspace
namespace/devspace created
$ devspace use namespace devspace
info The default namespace of your current kube-context 'kind-kind' has been updated to 'devspace'
         To revert this operation, run: devspace use namespace

done Successfully set default namespace to 'devspace'
$ devspace use context "$(kubectl config current-context)"
done Successfully set kube-context to 'arn:aws:eks:eu-central-1:174394581677:cluster/devspace-eks-QbUEJaxD'

💡 Creating a unique, separate namespace for every developer or feature is a very good way to prevent conflicts. E.g. if we need to change external configuration (e.g. ConfigMaps or Secrets) or change the API of a service during feature development, keeping these changes isolated in a dedicated namespace, prevents breaking the workflow of other developers.

We also need to tell DevSpace what kind of dev container to deploy for us. The way to do this is to use the DevSpace configuration file devspace.yaml. Below is an excerpt from the PoC repository mentioned earlier. With a few omissions for the sake of brevity. (In particular, the .pipelines section that I unfortunately at this point in time only have superficial knowledge on.)

version: v2beta1
name: devspace

# ...

deployments:
  the-dev-container:
    helm:
      chart:
        name: component-chart
        repo: https://charts.devspace.sh
      values:
        containers:
          - image: "${THE_DEV_CONTAINER_IMAGE}"
            imagePullPolicy: IfNotPresent
            resources:
              requests:
                memory: "500Mi"
                cpu: "500m"
              limits:
                memory: "1Gi"
                cpu: "1"

dev:
  the-dev-container:
    imageSelector: "${THE_DEV_CONTAINER_IMAGE}"
    ssh:
      localPort: 60550 
    command: ["sleep", "infinity"]
    sync:
    - path: ./:/home/dev

vars:
  THE_DEV_CONTAINER_IMAGE:
    source: env
    default: ubuntu:22.04

Let's go over the configuration file step by step.

First, the .deployments section essentially defines what resources to deploy to the configured Kubernetes cluster. For our use case, the most pivotal deployment is our dev container. The Pod running our dev container will be deployed using Helm (since we specify an .deployments.the-dev-container.helm object). More specifically, we use the component-chart Helm chart provided by the DevSpace team. We pass values to the Helm chart with the .deployments.the-dev-container.helm.values property. Crucially, the dev container image which is parameterized using the DevSpace variable THE_DEV_CONTAINER_IMAGE specified in the .vars section, so we are able to reuse the DevSpace configuration for different images.

ℹ️ Note that the-dev-container was arbitrarily chosen. You can pick whatever name you like best as long as it conforms to syntax requirements.

ℹ️ The .deployments section, moreover, allows multiple deployment specifications. We could theoretically also deploy ancillary resources like databases using separate Helm charts.

ℹ️ Details on the configurable Helm values for the component-chart Helm chart can be found in the chart documentation.

Moving on to the .dev section. We configure a development configuration for the deployment defined earlier, by adding an .dev.the-dev-container object. We again have to specify the image here (using the THE_DEV_CONTAINER_IMAGE variable). This time, however, it is used as a selector so that DevSpace can find the Kubernetes Pod running the dev container.
More interestingly, we tell DevSpace to allow SSH access to the deployed dev container by adding an .dev.the-dev-container.ssh object. It is worthy of note, that the port may be fixed to a specific local port number (via .dev.the-dev-container.ssh.localPort) so that we don't have to change the configuration of tools trying to connect to our dev container via SSH whenever we redeploy our dev container.

💡 Running devspace dev with enabled SSH will add an entry in ~/.ssh/config similar to the following one:
# DevSpace Start the-dev-container.devspace.devspace
Host the-dev-container.devspace.devspace
HostName localhost
LogLevel error
Port 60550
IdentityFile "/home/lima.linux/.devspace/ssh/id_devspace_ecdsa"
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
User devspace
# DevSpace End the-dev-container.devspace.devspace
DevSpace also creates a public and private key pair for authentication. This allows us to connect via SSH via
ssh -i ~/.devspace/ssh/id_devspace_ecdsa -l devspace -p 11817 localhost
or—using the SSH configuration—by simply running
ssh the-dev-container.devspace.devspace

Additionally, we add a blocking command to the dev container in .dev.the-dev-container.command, so that our pod doesn't terminate right away.
Lastly, the .dev.the-dev-container.sync property tells DevSpace to sync file changes from and to our current working directory (./) to the dev user's home directory on the dev container (/home/dev).

As mentioned earlier, we parameterized the dev container image using DevSpace variables (declared in the .vars section). So, the last thing we have to do before we can actually launch our dev container with DevSpace, ist to build and upload a suitable dev image to a registry that our cluster has access to. (Or, use a public dev container image.)

Let's assume for now that we have built and pushed a dev container image tagged 174394581677.dkr.ecr.eu-central-1.amazonaws.com/devspace-devcontainer:latest and that it is available to the provisioned cluster. We then may utilize the variable THE_DEV_CONTAINER_IMAGE declared in the devspace.yaml above to deploy a dev container with our custom dev container image using the DevSpace CLI's --var option:

$ devspace dev --var THE_DEV_CONTAINER_IMAGE="174394581677.dkr.ecr.eu-central-1.amazonaws.com/devspace-devcontainer:latest"
info Using namespace 'devspace'
info Using kube context 'arn:aws:eks:eu-central-1:174394581677:cluster/devspace-eks-3Hij2z5x'
deploy:the-dev-container Deploying chart /home/lima.linux/.devspace/component-chart/component-chart-0.9.1.tgz (the-dev-container) with helm...
deploy:the-dev-container Deployed helm chart (Release revision: 1)
deploy:the-dev-container Successfully deployed the-dev-container with helm
dev:the-dev-container Waiting for pod to become ready...
dev:the-dev-container Selected pod the-dev-container-devspace-847f75dd44-9httz
dev:the-dev-container sync  Sync started on: ./ <-> /home/dev
dev:the-dev-container sync  Waiting for initial sync to complete
dev:the-dev-container sync  Initial sync completed
dev:the-dev-container ssh   Port forwarding started on: 60550 -> 8022
dev:the-dev-container ssh   Use 'ssh the-dev-container.devspace.devspace' to connect via SSH

💡 Note the lines Sync started on: ./ <-> /home/dev and Port forwarding started on: 60550 -> 8022 that tell us that the continuous two-way sync of the local working directory to the dev container directory /home/dev was established; respectively, that the SSH server is listening on the local port 60550 .

And that's it. 🚀 As a simple test, we may run the hostname command on the remote dev container.

$ ssh the-dev-container.devspace.devspace 'hostname'
the-dev-container-devspace-847f75dd44-9httz

As expected from a container running in a Kubernetes Pod, this will return the Pod identifier.

Assuming sufficient CPU and memory are allocated to the container, we could now continue by attaching shells, and connecting IDE or editor via SSH and start developing. After we're done, we simply detach from the container and run devspace purge to clear up the deployed resources.

Discussion

DevSpace is a CNC-sponsored project that allows us to easily deploy dev containers on a Kubernetes cluster. When compared to other tools for dev container workflows, it has the advantage that it supports

connection via SSH, that it has a
robust two-way sync mechanism, and that it
deploys to Kubernetes.

As shown above, once a Kubernetes cluster and a container image registry that is accessible from the cluster are available, the development workflow is simple and very flexible w.r.t. to the development tooling. Moreover, we may use Kubernetes abstractions to support some of the desired features of a development environment (e.g. resource isolation via namespaces).

DevSpace therefore is a very reasonable choice for development teams that have access to both these infrastructure resources since it allows them to leverage the benefits of dev containers with minimal setup requirements and is very flexible w.r.t. to tooling choice.

Detecting Kubernetes API Deprecations with pluto

Friedrich Kurz — Wed, 27 Jul 2022 13:22:24 +0000

The Kubernetes API is changing all the time. With these changes come deprecations and eventual removals of parts of the API. To be able to keep an up-to-date Kubernetes cluster version, we have to identify deprecated APIs and update them. This may become tedious in larger clusters with hundreds of resources but tools like pluto can help.

What does API deprecation mean in Kubernetes?

The operations governing the lifecycle of all Kubernetes resources are provided via RESTful API endpoints by the Kubernetes API server. In other words, the Kubernetes API is the frontend of the Kubernetes control plane.

Resource APIs are associated with URIs like

/apis/GROUP/VERSION/* for cluster-scoped resources, and
/apis/GROUP/VERSION/namespaces/NAMESPACE/* for namespace-scoped resources.

Some resources are also grouped into the core group (or, legacy group). Those are available via the special API endpoint /api/{version} (see API groups). Pods, for example, are part of the core group. A request to list all pods in a given namespace my-namespace is mapped to the following HTTP GET request.

GET /api/v1/namespaces/my-namespace/pods

(See the API documentation for the Pod resource for reference.)

The problem with Kubernetes API deprecations

Kubernetes specifies a deprecation policy that defines what it means if parts of an API become deprecated. Essentially, deprecation means that the associated endpoints of the Kubernetes API server are flagged for removal and subsequently deleted. Since the API server governs the resource lifecycle, using a resource with a removed API version, will prevent the deployment of that resource. Consequently, if we fail to update our resource API versions, we will either be stuck with an outdated Kubernetes version; or, updating to the new Kubernetes version will prevent deployments of certain resources. Both are undesirable states since we will either

continue using an unstable Kubernetes version, or
our Kubernetes deployment will be incomplete.

Deploying resources with removed API versions

To get a clearer picture, let's have a look at the second problem and see how Kubernetes responds if we try to deploy a resource using a removed API version. To do this, we spin up a local Kubernetes cluster with k3d

$ k3d cluster create
INFO[0000] Prep: Network
INFO[0004] Created network 'k3d-k3s-default'
INFO[0004] Created image volume k3d-k3s-default-images
INFO[0004] Starting new tools node...
INFO[0005] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.4.3'
INFO[0007] Starting Node 'k3d-k3s-default-tools'
INFO[0007] Creating node 'k3d-k3s-default-server-0'
INFO[0009] Pulling image 'docker.io/rancher/k3s:v1.23.6-k3s1'
INFO[0027] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0028] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.4.3'
INFO[0030] Using the k3d-tools node to gather environment information
INFO[0030] Starting new tools node...
INFO[0030] Starting Node 'k3d-k3s-default-tools'
INFO[0033] Starting cluster 'k3s-default'
INFO[0033] Starting servers...
INFO[0033] Starting Node 'k3d-k3s-default-server-0'
INFO[0040] All agents already running.
INFO[0040] Starting helpers...
INFO[0040] Starting Node 'k3d-k3s-default-serverlb'
INFO[0049] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap...
INFO[0051] Cluster 'k3s-default' created successfully!
INFO[0051] You can now use it like this:
kubectl cluster-info

Then switch Kubernetes context e.g. using kubectx

$ kubectx k3d-k3s-default
Switched to context "k3d-k3s-default".

Now, we take a look at which version the Kubernetes API server is running by executing kubectl version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

As we can see from the Server Version: output, our k3d cluster is running Kubernetes v1.23.

By taking a look at the API deprecation guide we can see that two versions of the Ingress API, extensions/v1beta1 and networking.k8s.io/v1beta1, were removed in v1.22 of Kubernetes. So let's try to deploy an Ingress resource with that API version and see what happens. Below we have an example manifest that I shamelessly stole from the The Ingress Resource section of the official Kubernetes documentation and put it in a file ingress-pre-1.22.yaml.

# ingress-pre-1.22.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80

And then let's try to deploy it to our v1.23 Kubernetes cluster.

$ kubectl apply -f ingress-pre-1.22.yaml
error: resource mapping not found for name: "minimal-ingress" namespace: "" from "ingress-pre-1.22.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
ensure CRDs are installed first

As we can see, the API returns an error indicating that networking.k8s.io/v1beta1 does not include an Ingress type anymore. If we change the API version to networking.k8s.io/v1, however,

# ingress-1.22.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80

our Ingress will be created without a hitch.

$ kubectl apply -f ingress-1.22.yaml
ingress.networking.k8s.io/minimal-ingress created

Detecting API deprecations with pluto

In a more realistic scenario, we already have resources deployed to our cluster and want to keep their API versions up to date so that we can update our cluster version safely.

The question therefore is, how do we spot resources with deprecated and soon-to-be-removed API versions? One answer to that question is to check the deprecation guide mentioned earlier and check which resource API versions will be removed in the upcoming Kubernetes update. It is important to note, however, that if we skip several versions, we will have to repeat this check for all versions in between our current and target Kubernetes versions as well.

In large clusters with dozens of resource types and versions, this can become tedious and error-prone. Luckily, there are tools like pluto by FairwindOps which assist us in spotting deprecated and soon-to-be-removed resource API versions.

Let's for example deploy our Ingress resource to a Kubernetes cluster with an API server version earlier than v1.22 (e.g. v1.19). Creating a k3d cluster with a non-current Kubernetes API version is possible by passing the --image option to k3d specifying a k3s image for our desired Kubernetes version (e.g. v1.19.16-k3s1, full list of images is available on Dockerhub):

$ k3d cluster delete
INFO[0000] Deleting cluster 'k3s-default'
INFO[0002] Deleting cluster network 'k3d-k3s-default'
INFO[0005] Deleting 2 attached volumes...
WARN[0005] Failed to delete volume 'k3d-k3s-default-images' of cluster 'k3s-default': failed to find volume 'k3d-k3s-default-images': Error: No such volume: k3d-k3s-default-images -> Try to delete it manually
INFO[0005] Removing cluster details from default kubeconfig...
INFO[0005] Removing standalone kubeconfig file (if there is one)...
INFO[0005] Successfully deleted cluster k3s-default!
$ k3d cluster create --image rancher/k3s:v1.19.16-k3s1
INFO[0000] Prep: Network
INFO[0003] Created network 'k3d-k3s-default'
INFO[0003] Created image volume k3d-k3s-default-images
INFO[0003] Starting new tools node...
INFO[0003] Starting Node 'k3d-k3s-default-tools'
INFO[0006] Creating node 'k3d-k3s-default-server-0'
INFO[0009] Pulling image 'rancher/k3s:v1.19.16-k3s1'
INFO[0018] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0018] Using the k3d-tools node to gather environment information
INFO[0018] Starting new tools node...
INFO[0018] Starting Node 'k3d-k3s-default-tools'
INFO[0021] Starting cluster 'k3s-default'
INFO[0021] Starting servers...
INFO[0021] Starting Node 'k3d-k3s-default-server-0'
INFO[0028] All agents already running.
INFO[0028] Starting helpers...
INFO[0028] Starting Node 'k3d-k3s-default-serverlb'
INFO[0037] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap...
INFO[0039] Cluster 'k3s-default' created successfully!
INFO[0039] You can now use it like this:
kubectl cluster-info

We again confirm that the API server is running Kubernetes v1.19 with kubectl version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16+k3s1", GitCommit:"da16869555775cf17d4d97ffaf8a13b70bc738c2", GitTreeState:"clean", BuildDate:"2021-11-04T00:55:24Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.19) exceeds the supported minor version skew of +/-1

Now, let's apply our Ingress with the manifest specifying the deprecated API version again.

$ kubectl apply -f ingress-pre-1.22.yaml
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/minimal-ingress created

As we can see, we already get a helpful deprecation warning that recommends using networking.k8s.io/v1 instead of networking.k8s.io/v1beta1. For now, we assume, however, that our resources already have been deployed to a running cluster and we want to detect those resources (without applying them again).

This is where tools like pluto become handy to detect the use of deprecated resource API version.

Pluto is available on multiple platforms and via a variety of package managers. We'll use binenv here.

$ binenv install pluto
2022-07-21T19:47:18+02:00 WRN version for "pluto" not specified; using "5.8.0"
fetching pluto version 5.8.0 100% |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (11/11 MB, 9.578 MB/s)
2022-07-21T19:47:22+02:00 INF "pluto" (5.8.0) installed

There is a couple of ways we can hand over resource manifests to pluto in order to detect deprecated API versions: for example via directory scan or by direct input. Additionally, there is a convenient integration with Helm that checks our releases for deprecations.

Directory scan

Using a directory scan requires that we know where to find the Kubernetes manifests for our cluster. In our case, this is pretty simple since we only have two simple manifests and they are both in the current directory. The following command makes pluto run a directory scan and detect our deprecated api version.

$ pluto detect-files --directory . --target-versions k8s=v1.22.0
NAME              KIND      VERSION                     REPLACEMENT            REMOVED   DEPRECATED
minimal-ingress   Ingress   networking.k8s.io/v1beta1   networking.k8s.io/v1   true      true

Note that we can specify our target version(s) using the --target-versions option. If we were to target Kubernetes v1.15.0 instead, pluto would return the empty list, because our CRD API version was deprecated only later on in v1.16.0.

$ pluto detect-files --directory . --target-versions k8s=v1.15.0
No output to display

Direct input

Using direct input, we can just pipe a resource manifest directly into pluto. This is particularly useful if we want to scan the resources already deployed to the cluster (and going over the manifests is too complicated). Since our CRD is already deployed we can obtain the resource manifest with kubectl get and hand it to pluto.

$ kubectl get ingress minimal-ingress -o yaml | pluto detect -
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME              KIND      VERSION              REPLACEMENT            REMOVED   DEPRECATED
minimal-ingress   Ingress   extensions/v1beta1   networking.k8s.io/v1   true      true

ℹ️ You may have noticed that pluto reports the deprecated API version as extensions/v1beta1 instead of networking.k8s.io/v1beta1. This is the case because the API server in v1.19. apparently normalises networking.k8s.io/v1beta1 to extensions/v1beta1. The manifest of the deployed Ingress that is returned by kubectl get ingress minimal-ingress -o yaml therefore has API version extensions/v1beta1.

Helm releases

If we deploy our resources with Helm, Pluto also provides a detect-helm subcommand that checks our releases for deprecated API versions.

Summary

The Kubernetes API is constantly evolving. To keep our cluster up to date, we constantly have to watch for deprecated and soon-to-be-removed resource API versions. Manual checks are possible thanks to the Kubernetes deprecation guide, but they can become very tedious and error-prone. Tools like pluto allow us to automate API deprecation checks and simplify the effort involved in maintaining up-to-date resource API versions.

More on Kubernetes

Check out my blog for more posts on Kubernetes:

GitLab CI/CD Runner Clean-up with Pre-build Scripts

Friedrich Kurz — Mon, 23 May 2022 19:08:24 +0000

Introduction

GitLab CI/CD has a powerful but somewhat under-documented pre-build script feature that allows us to execute custom logic before builds are run on a GitLab runner. We’ll have a look at how to utilize the pre-build script to automate Docker system clean-up on GitLab runners.

GitLab runners and the problem of automated clean-up

Imagine we work on a project using GitLab CI/CD with a set of demanding criteria such as

high degree of automation,
need for high runner uptime,
fast build execution, and
frequent use of Docker.

Particularly due to the lack of downtime for clean-up and maintenance, we may eventually run into the well-known problem that GitLab runners will run out of disk space.

Unfortunately, finding a general solution for GitLab runner clean-up is not particularly easy as indicated by the existence of this, to date, unresolved issue on the GitLab runner issue tracker. For instance, if we simply clean up all Docker resources after each build, we won't likely run out of disk space. However, our build times would be much higher because Docker won't be able to leverage its build cache mechanism.

Meanwhile, GitLab's documentation recommends running the clear-docker-cache script once a week via cron as a workaround. Using the cron approach is also fairly simple and will moreover slow down our builds less frequently. On the flip side, however, we will now have to provide our runners with sufficient disk space for a full week (or, whatever interval the cron job is set to run on) which might be excessive and is moreover hard to guess correctly.

As noted in the unresolved issue that I mentioned earlier, the GitLab suggested way of managing disk space also has at least two more problems:

it only addresses images, and it is
indiscriminate meaning that it may end up cleaning up frequently used images as well.

This is problematic for a couple of reasons. First of all, other cached Docker resources—like volumes and containers—are not targeted by the clean-up script. Moreover, it may result in a build slow-down because frequently used images are being cleaned up and have to be rebuilt. Finally—since the script is run by cron—, there also is the intricate problem of race conditions between the clean-up script and build jobs: since cron runs asynchronous to build job executions, our clean-up job may inadvertently crash pipelines because it cleans up images some jobs depend upon. ^[For example if we build our images and push them to our registry in separate jobs.] This problem is of course mitigated by running the clean-up script only once a week but developers will have to keep in mind that pipelines may fail once in a while due to missing images.

Determining Docker disk usage and cleaning up Docker cache

So, what to do? First of all, let's look at what tools are available to determine Docker disk usage as well as trigger clean-up of resources that occupy disk space.

To get an indication of the current disk usage of Docker, we can run docker system df. Here's an example output from my local machine:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          2         2         216.6MB   0B (0%)
Containers      2         2         84.83kB   0B (0%)
Local Volumes   5         5         506.7MB   0B (0%)
Build Cache     0         0         0B        0B

As we can see, Docker helpfully lists the disk space taken up by each of its resource classes.

With some light BASH acrobatics, we can work this into a test that tells us if Docker disk space usage is below a given limit (see is_docker_disk_space_usage_above_limit.sh).

#!/bin/bash
# --
# Test if Docker daemon disk space usage is above a given limit in bytes.
#
# Example 1: Docker disk space usage is above limit 
#
#    $ source is_docker_disk_space_usage_above_limit.sh
#    $ is_docker_disk_space_usage_above_limit 1
#    Docker disk space usage is above limit (actual: 1050000005B, limit: 1B)
#    $ printf $?
#    0
#
# Example 2: Docker disk space usage is below or equal to limit
#
#    $ source is_docker_disk_space_usage_above_limit.sh
#    $ is_docker_disk_space_usage_above_limit 1000000000000
#    Docker disk space usage is below or equal to limit (actual: 1050000005B, limit: 1000000000000B)
#    $ printf $?
#    1
# 
# --
iec_string_to_bytes() {
  local iec_string=$1
      iec_format_pattern='([0-9.]+)\s*([kMGTP]?B)'

  if ! [[ "${iec_string}" =~ ${iec_format_pattern} ]]; then
    printf "Input string has invalid format (received: \"%s\", expected: \"%s\")." "$1" "${iec_format_pattern}"
    return 1
  fi 

  local number_value="${BASH_REMATCH[1]}"
    iec_unit="${BASH_REMATCH[2]}"
    factor=""

  case "${iec_unit}" in 
    B) factor=1;;
    kB) factor=1000;;
    MB) factor=1000000;;
    GB) factor=1000000000;;
    TB) factor=1000000000000;;
    PB) factor=1000000000000000;;
  esac

  # We use scale=0 here to drop the (redundant) decimal points.
  # This only works with division so we divide by one.
  printf "scale=0;%s * %s/1\n" "${number_value}" "${factor}" \
    | bc 
}

calculate_docker_total_disk_space_usage() {
  local bc_expression="0"
    disk_space_used="$(docker system df --format='{{.Size}}' | tr '\n' ' ')"

  for disk_space_used_by_resource in ${disk_space_used}; do 
    # shellcheck disable=SC2086
    disk_space_used_by_resource_bytes="$(iec_string_to_bytes \"${disk_space_used_by_resource}\")"
    bc_expression="${bc_expression} + ${disk_space_used_by_resource_bytes}"
  done 

  printf "%s\n" "${bc_expression}" \
    | bc
}

is_docker_disk_space_usage_above_limit() {
  local disk_space_limit=$1
    docker_disk_space_used="$(calculate_docker_total_disk_space_usage)"
    docker_disk_space_usage_is_above_limit="$(printf '%s > %s\n' ${docker_disk_space_used} ${disk_space_limit} | bc -l)"

  # Note that bc returns 1 if the comparison is true and 0 otherwise.
  if [ "${docker_disk_space_usage_is_above_limit}" -eq 1 ]; then 
    printf "Docker disk space usage is above limit (actual: %sB, limit: %sB)" "${docker_disk_space_used}" "${disk_space_limit}"
    return 0
  fi 

  printf "Docker disk space usage is below or equal to limit (actual: %sB, limit: %sB)" "${docker_disk_space_used}" "${disk_space_limit}"
  return 1
}

Schematically, we can run this script in BASH as follows to trigger our clean-up logic:

# Check if docker disk space usage is above a given limit and run clean-up logic if it is
if is_docker_disk_space_usage_above_limit "${docker_disk_space_usage_limit}"; then
  # run clean up
fi

If Docker uses too much disk space, we may then proceed to remove unused resources via the Docker CLI. The most simple way to do this is by running docker system prune -af --volumes. Pruning the system with these parameters will simply clean up all dangling and unused images, containers, networks, as well as volumes.

In case we need a more elaborate clean-up logic, Docker CLI also has individual commands to free the cache of unused images, containers, networks, and volumes which support filters. E.g. docker image prune can be used to only clean up images that are older than 24 hours by running

docker image prune -a --force --filter "until=24h"

Runner hooks to the rescue

Now that we have a way how to find out if Docker is running out of disk space and how to trigger clean-up, we can think about when to run our logic. Per the requirements of our GitLab CI/CD set-up—as stated earlier—, we want to run

custom clean up logic,
pre-emptively (to avoid runner failures due to lack of disk space), and
in sync with pipeline execution (to prevent randomly failing pipelines).

Luckily, GitLab CI/CD provides a couple of script hooks that let us execute code in various stages of pipeline execution. Specifically, pre-clone, post-clone, pre-build, and post-build (see The [[runners]] section in Advanced Configuration).

Both the pre-build as well as the post-build hooks make sense in our scenario as both

run synchronous with pipeline jobs (either before or after a job),
pre-emptively allow us to clean up resources (either before the current job or the next job), and
provide a mechanism to define custom clean-up logic.

We choose the pre-build hook here.

To register our pre-build script, we have to configure our GitLab runners using their configuration file like so:

# /etc/gitlab-runner/config.toml
# ...
[[runners]]
  # ...
  pre_build_script = '''
    # execute clean-up script
  '''

Having added the pre_build_script property, our GitLab runners will now execute our clean-up script before each job.

This is unfortunately not a perfect, general solution either—as will be discussed later in Prerequisites and limitations of the pre-build script approach—but let's look at how to implement the pre-build clean-up technique first.

A quick test drive

Installing Docker and gitlab-runner

To test our setup we will install and configure GitLab runner on an AWS EC2 instance. (Obviously, any other similar cloud infrastructure as a service solution would work too.) GitLab runner binaries are available for multiple platforms. We pick an Ubuntu 20.4 machine here to use the Linux binaries.

After logging into our EC2 instance

ssh -i "${key-pair-pem-file}" "${ec2-user}@${ec2-instance-address}"

we first want to install Docker. E.g. using the official convenience install script

curl -fsSL https://get.docker.com -o /home/ubuntu/get-docker.sh
sudo sh /home/ubuntu/get-docker.sh

⚠️ Note that using the convenience script is not recommended for production builds. Neither is it generally a good idea to execute a downloaded script file with sudo. But since we trust the source here and only run a test, it's not a big deal.

Verify Docker installation success by running e.g. docker --version

$ docker --version
Docker version 20.10.16, build aa7e414

Now let's execute the script shown below to install gitlab-runner. ^[The GitLab runner installation script is also available from Settings > CI/CD > Runners > Specific runners of a GitLab project for reference.]

sh <<EOF
  # Download the binary for your system
  sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

  # Give it permission to execute
  sudo chmod +x /usr/local/bin/gitlab-runner

  # Create a GitLab Runner user
  sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

  # Install and run as a service
  sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
  sudo gitlab-runner start
EOF

And then let's register our runner with GitLab by running

sudo gitlab-runner register

The gitlab-runner tool will guide us through the process and ask for some configuration details. We have to enter the URL of our GitLab instance (e.g. https://gitlab.com for the public GitLab) and a registration token (that can be copied from Settings > CI/CD > Runners > Specific runners). We moreover pick the Docker executor (docker) since it is—at least in my experience—the most common one as well as the docker:20.10.16 image to be able to run Docker builds within our pipeline jobs. ^[Full disclosure, I tried the SSH executor for simplicity's sake but was not able to make it work due to connection problems.] Also, when prompted for tags, we enter gl-cl to be able to run our pipeline jobs on exactly this machine.

At the end of the registration process, we should see the following confirmation that our runner has been registered.

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

As described in Advanced configuration, the configuration file is stored in /etc/gitlab-runner/config.toml on Unix systems. We will add the runners.pre_build_script and runners.docker.volumes properties shown below.

# /etc/gitlab-runner/config.toml
# ...
[[runners]]
  # ...
  pre_build_script = '''
    sh $CLEAN_UP_SCRIPT     
  '''
  # ...
  [runners.docker]
  # ...
  volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

The pre_build_script property uses a little trick and simply executes a reference to a script file path CLEAN_UP_SCRIPT which we will later add as a GitLab CI/CD variable of type file. By doing that, we can apply changes and test our clean-up script without having to connect to our runner. The volumes property mounts the host machine's Docker socket in our container so we clean up resources on the host machine rather than only in the container (Docker in Docker via Docker socket binding).

Configuring our GitLab CI/CD pipeline

The file variable CLEAN_UP_SCRIPT has to be defined in the Settings > CI/CD > Variables section of our project for this to work. As shown in the next section. Let's add the following clean-up script for now.

set -eo pipefail
apk update 
apk upgrade 
apk add bash curl
curl https://gist.githubusercontent.com/fkurz/d84e5117d31c2b37a69a2951561b846e/raw/a39d6adb1aaede5df2fc54c1882618bcea9f01e0/is_docker_disk_space_usage_above_limit.sh > /tmp/is_docker_disk_space_above_limit.sh
bash <<EOF || printf "\nClean-up failed."
  source /tmp/is_docker_disk_space_above_limit.sh
  if is_docker_disk_space_usage_above_limit 2000000000; then
    printf "\nRunning clean up...\n"
    docker system prune -af --volumes
  else 
    printf "\nSkipping clean up...\n"
  fi 
EOF

Make sure to select File as Type when defining the variable.

Note that we install bash and curl in the pre-build script for simplicity's sake. This implies that both tools are installed before every job that is processed on this runner. In a real scenario, we'd naturally want to provide a custom image that has all the required tools installed to speed up the pre-build script's execution.

Now let's add a sample gitlab-ci.yaml to our project which creates a large one GB image and will therefore eventually trigger clean-up. (Code is available on GitLab.)

stages:
  - build

build-job:
  stage: build
  tags:
    - gl-cl
  script:
    - echo "Generating random nonsense..."
    - ./scripts/generate-random-nonsense.sh
    - echo "Building random nonsense image..."
    - ./scripts/build-random-nonsense-image.sh

To limit our pipelines to our new runner, we use tag selectors and pick our previously registered runner via the gl-cl tag. Now we may finally run our pipeline a couple of times to see the effect of our pre-build script. Depending on the runner's disk size, we should see a couple of jobs without clean-up followed eventually by a run that contains a log similar to this one:

Docker disk space usage is above limit (actual: 2361821460B, limit: 2000000000B)
Running clean up...
Deleted Containers:
9a126aead174a15a4f76f2cb5744e36aff30741cc6ab0ac5044837aaee946496
2fa51dc3e7f5b1e3bec63a635c266552f9b02eb74015a9683f7cbf13418a12eb

Deleted Images:
untagged: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-febb2a09
untagged: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:edc1bf6ab9e1c7048d054b270f79919eabcbb9cf052b3e5d6f29c886c842bfed
deleted: sha256:c20c992e5d83348903a6f8d18b4005ed1db893c4f97a61e1cd7a8a06c2989c40
deleted: sha256:873201b44549097dfa61fa4ee55e5efe6e8a41bbc3db9c6c6a9bfad4cb18b4ea
untagged: random-nonsense-image-1653227274:latest
deleted: sha256:67fde47d8b24ee105be2ea3d5f04d6cd0982d9db2f1c934b3f5b3675eb7a626f
deleted: sha256:1a310f85590c46c1e885278d1cab269f07033fefdab8f581f06046787cd6156e
untagged: alpine:latest
untagged: alpine@sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
untagged: random-nonsense-image-1653226909:latest
deleted: sha256:b5923f3fb6dd2446d18d75d5fbdb4d35e5fca888bd88aef8174821c0edfcb87f
deleted: sha256:59150b0202d2d5f75ec54634b4d8b208572cbeec9c5519a9566d2e2e6f2c13f3
deleted: sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b

Total reclaimed space: 2.059GB

This output indicates that our pre-build script was executed when Docker's disk space usage reached a value above 2GB and clean-up was triggered successfully (freeing in this case roughly 2GB of disk space).

Prerequisites and limitations of the pre-build script approach

It's probably easy to see that our pre-build script approach fulfills the requirements we laid out for it. I.e. it

runs synchronously before pipeline jobs, it
pre-emptively cleans-up unused resources, and it
allows us to provide a custom clean-up logic.

Nonetheless, there are still a few limitations left to consider.

First of all, bash must be available during pre-build script execution if we want to use the is_docker_disk_space_usage_above_limit.sh script because it uses some BASHisms. Moreover, since we use the Docker executor, we need to use some kind of runner image that has the Docker CLI installed (such as the official Docker base image we used in our test earlier). Writing a custom image to use as the base image to run our pipelines takes care of and reduces the severity of this problem, but it's still something that has to be addressed.

Another thing to keep in mind is that using Docker disk space usage is only an approximation (i.e. lower than) the system disk space usage. Consequently, we have to find a good value for the limit that triggers our clean-up logic to not run out of disk space on the machine anyway.

Also, it may still be tricky to pick the right clean-up logic. For instance, if we just run the docker system prune -af --volumes as in our test, we may delete images that are required by subsequent jobs in more complex pipelines. Excluding certain images from clean-up—for instance, those built in the last 24 hours—may be able to alleviate this exact problem. However, in more complicated pipelines, we will likely need a more elaborate clean-up logic.

Lastly, there are still edge cases even with the pre-build clean-up script approach where our runners will run out of disk space. Off the top of my head, if the limit is too high, a runner might still end up running out of disk space because the job run produces more resources than available free space.

Summary

As we've seen, we can use GitLab CI/CD's pre-build script hook to clean up GitLab runners in sync with job execution, pre-emptively to avoid breaking pipelines, as well as using custom clean-up logic. That being said, the pre-build script clean-up approach is not perfect, because it cannot avoid all situations where a runner will run out of disk space. Nonetheless, I think it is still a more elegant way to handle clean-up of GitLab runners than maintenance downtimes or the cron job approach.

More on GitLab CI/CD

Check out my blog for more posts about GitLab CI/CD:

DEV Community: Friedrich Kurz

Extending AstroJS Markdown Processing With Remark and Rehype Plugins

AstroJS

Remark and Rehype

Extending AstroJS With Remark and Rehype

Example: Enabling rehype-mermaid in AstroJS

More on AstroJS

How to define and work with a Rust-like result type in NuShell

Motivation

Error handling in NuShell

Addendum

More on Nushell

Dev Containers on Kubernetes With DevSpace

Motivation

DevSpace

Basic Development Workflow with DevSpace

Discussion

More on DevSpace

Detecting Kubernetes API Deprecations with pluto

What does API deprecation mean in Kubernetes?

The problem with Kubernetes API deprecations

Deploying resources with removed API versions

Detecting API deprecations with pluto

Directory scan

Direct input

Helm releases

Summary

More on Kubernetes

GitLab CI/CD Runner Clean-up with Pre-build Scripts

Introduction

GitLab runners and the problem of automated clean-up

Determining Docker disk usage and cleaning up Docker cache

Runner hooks to the rescue

A quick test drive

Installing Docker and gitlab-runner

Configuring our GitLab CI/CD pipeline

Prerequisites and limitations of the pre-build script approach

Summary

More on GitLab CI/CD

Example: Enabling `rehype-mermaid` in AstroJS