DEV Community: Jean-Nicolas Moal

GitLab CI speeding up your pipeline with caching

Jean-Nicolas Moal — Wed, 12 Apr 2023 13:12:59 +0000

Introduction

GitLab CI comes with a cache system very useful when you want to speed up your pipeline.

In this article, we’ll see how we can use the cache system to speed up our pipeline.

📌 NOTE
Don’t use cache to store build results, artifacts are made for this.

What you should know about GitLab CI cache

The cache is stored where the gitlab runner is installed.
It can be uploaded if the distributed cache is enabled.
If you’re running your runners in Kubernetes, the distributed cache is a must-have.

You can clear the cache when necessary.

You can use a fallback key when your cache doesn’t exist. Useful to share cache between branch, while still keeping some isolation.

You can have up to 4 caches per pipeline.

You can tell GitLab whether you want to pull-push (download and update) the cache, or if you just want to pull it.

❗ IMPORTANT
For security reason, GitLab creates a different cache if the branch is protected or not. This feature is configurable, but active by default on your project. You can read more on this here

Let's play with the caching system

Say that you have a python project, for which you need to run linters, tests suites... And finally, you want to package your application.
And you have the following constraints:

Your final package shall only contain what’s necessary to run
Only download dependencies when they’ve changed
The cache from the main branch is the reference

In this example, we’ll set up the cache so that:

Each branch will have its own cache
When building on a new branch, the build will fall back to the main branch cache
The dependencies will only be downloaded and updated when necessary

For clarity, we’ll only use fake jobs in this example.

Defining rules to manage cache updates

.deps_update_rules:
  rules:
    - if: $CI_COMMIT_BRANCH != "main" ①
      changes:
        paths:
          - my-referent-file.lock ②
        compare_to: refs/heads/main ③
    - if: $CI_COMMIT_BRANCH == "main" ④
      when: always
    - if: $CI_COMMIT_TAG != null ⑤
      when: always

When not on the main branch, we only update dependency if necessary.
Put all the relevant files here
Compares changes with the main branch
On the main branch, always update dependencies
When tagging the repository, update the dependencies

This hidden field defines rules that will be reused by our job.
If you aren’t familiar with rule, check out this documentation.
Feel free to adapt those rules to your workflow.

Creating the cache

Before creating the jobs to update our caches, let’s first define some reusable stuff.
We’ll leverage some predefined variables to make this work properly.

variables:
  DEFAULT_CACHE_KEY_PREFIX: ${CI_DEFAULT_BRANCH} ①
  CACHE_KEY_PREFIX: ${CI_COMMIT_REF_SLUG} ②
  DEPS_FOLDER: ${CI_PROJECT_DIR}/.deps ③

.deps_dev_cache: ④
  cache: &deps_dev_cache
    key: ${CACHE_KEY_PREFIX}-deps-dev ⑤
    paths:
      - ${DEPS_FOLDER}
    policy: pull ⑥

.deps_run_cache: ⑦
  cache: &deps_run_cache
    key: ${CACHE_KEY_PREFIX}-deps-run
    paths:
      - ${DEPS_FOLDER}
    policy: pull

Use to define the fallback cache key in the job
Use to define the cache key to be used by the job
The location of the downloaded dependencies (the folder we want to keep)
The cache configuration for jobs that needs the dev dependencies
The key to use to find the cache with a suffix to identify dev dependencies
The default policy we want to use, most jobs only need to download the cache, not to update it.
The cache configuration for jobs that needs the run dependencies

📌 NOTE
The environment variables with the CI_ prefix are predefined variables.

Now that we have our predefined cache configuration, let’s use them in the jobs that need to update the dependencies.

dev-dependencies: ①
  stage: prepare
  script:
    - echo "I am a dependency" > ./deps.txt
    - echo "I am a dev dependency" >> ./deps.txt
  cache:
    <<: *deps_dev_cache
    policy: pull-push
  rules: !reference [.deps_update_rules, rules]

run-dependencies: ②
  stage: prepare
  script:
    - echo "I am a dependency" > ./deps.txt
  cache:
    <<: *deps_run_cache
    policy: pull-push
  rules: !reference [.deps_update_rules, rules]

This job can download all dependency groups as its cache will be used for linting, testing...
This job shall download only the run dependency (for example poetry install --only main).

Using the cache

Now, we can use those caches to test or package our application.

test-app:
  stage: test
  script:
    - cat ./deps.txt
  cache:
    <<: *deps_dev_cache

package-app:
  stage: build
  script:
    - cat ./deps.txt
  cache:
    <<: *deps_run_cache

📌 NOTE
Using this method, the pipeline must run at least once on the main branch before the other branches.

Building OCI images with Buildah: Take control on how you build your container

Jean-Nicolas Moal — Wed, 20 Oct 2021 07:35:43 +0000

Photo by Ian Taylor on Unsplash

Introduction

In the Linux world, container technology has evolved in a way that we can consider them as a standard way to package and run an application.
Because Docker is the most known tools to play with containers, Dockerfile is the most common way to build them.

But today I want to share with you my preferred tool: Buildah.

It's a powerful tool for building container image in a simple but yet efficient way.

In this article, we'll see how to build a basic image, how to optimize build time using cache, and how to install OS packages without having the OS installed within the container.

Prerequisites

To follow this article, you'll need:

buildah
task

And a Fedora based machine.

Building an image

First, we'll create a basic container image and install Ansible in it.

Create a file called build-unoptimized.sh with the following content:

#!/usr/bin/env bash

set -o errexit

# Creates a new container instance, and stores its name in CONTAINER_ID.
CONTAINER_ID=$(buildah from docker.io/fedora:34)

# Runs commands within the created container.
buildah run "${CONTAINER_ID}" /bin/bash -c "dnf install -y ansible-$ANSIBLE_VERSION"

# Save the container into an image.
buildah commit "${CONTAINER_ID}" "${IMAGE_NAME}:${IMAGE_TAG}"

Before running this script, let's add a Taskfile.yaml to easily configure and build the image:

version: "3"

env:
  IMAGE_TAG:
    sh: git log -n 1 --format=%h
  ANSIBLE_VERSION: "2.9.25"

tasks:
  build-unoptimized:
    env:
      IMAGE_NAME: "buildah-demo-unoptimized"
    desc: "Build the container image without any optimization"
    cmds:
      - time -p ./build-unoptimized.sh

Now, if you run task build-unoptimized, Buildah will create a container image based on docker.io/fedora:34.

It will install ansible using dnf.

On my computer, running this task took 123 seconds and the image size is 614 MB.
We could optimize the size of the image by cleaning up all the DNF cache, but there is a better alternative.

Optimizing by caching

We can kill two birds with one stone with caching, this will help on building time and on the image size.

Let's create a file called build-with-cache.sh with the following content:

#!/usr/bin/env bash

set -o errexit

DNF_CACHE_FOLDER=${HOME}/.dnfcache/
RUN_OPTION="-v ${DNF_CACHE_FOLDER}:/var/cache/dnf:Z"

mkdir -p "${DNF_CACHE_FOLDER}"

# Creates a new container instance, and stores its name in CONTAINER_ID.
CONTAINER_ID=$(buildah from docker.io/fedora:34)

# Runs commands within the created container.
# The keepcache=true in the dnf.conf file tells dnf not to delete successfully installed packages.
buildah run ${RUN_OPTION} "${CONTAINER_ID}" /bin/bash -c "echo 'keepcache=True' >> /etc/dnf/dnf.conf && dnf install -y --nodocs ansible-$ANSIBLE_VERSION"

# Save the container into an image.
buildah commit "${CONTAINER_ID}" "${IMAGE_NAME}:${IMAGE_TAG}"

This new script does the exact same installation as the previous one, but this time it shares a specific local folder with the container,

so that all files downloaded with dnf are kept locally for next runs.

Let's add the corresponding task within the Taskfile.yaml file:

  build-with-cache:
    env:
      IMAGE_NAME: "buildah-demo-with-cache"
    desc: "Build the container image using caching"
    cmds:
      - time -p ./build-with-cache.sh

In order to test this new script, we need to run it twice.

The first time to create the cache, and the second one to see the improvements.

On my computer this build took 35 seconds, and the image size is 365 MB.

That's a huge improvement, we reduced the time by almost 4, and the size by almost 2.

The size is reduced because with this new script the cache is outside the container, so it doesn't end within the container.

We could call it a day, but, in my opinion, containers shall only contains what is necessary to run.

So I don't want all the extra pre-installed packages that comes from the base image.

Let's try to install Ansible from a scratch container.

Installing OS package, without the OS

The idea is simple, we'll tell DNF to install all the package in a specific location.

If the package properly declares all its dependencies, then this should work fine.

But if it doesn't, then we'll need to find the missing packages and install them too.

Let's create a file called build-without-os.sh with the following content:

#!/usr/bin/env bash

set -o errexit
set -x
CONTAINER_ID=$(buildah from scratch)
MOUNT_POINT=$(buildah mount "${CONTAINER_ID}")
DNF_CACHE_FOLDER=${HOME}/.dnfcache

mkdir -p "${DNF_CACHE_FOLDER}"

dnf --installroot "${MOUNT_POINT}" --releasever 34 --nodocs --setopt install_weak_deps=false --setopt cachedir="${DNF_CACHE_FOLDER}" install -y ansible-$ANSIBLE_VERSION

# Save the container into an image.
buildah commit "${CONTAINER_ID}" "${IMAGE_NAME}:${IMAGE_TAG}"

In this script, I added the --installroot option, which tells dnf to install package in a specific location.

I also configured DNF with the keepcache=True on my workstation.

With this new script, we do not use command inside the container, but command available from the host.

This way, no need to have an OS installed or any other package manager inside the container.

Let's add the corresponding task in the Taskfile.yaml file:

  build-without-os:
    env:
      IMAGE_NAME: "buildah-demo-without-os"
    desc: "Build the container image from scratch"
    cmds:
      - time -p buildah unshare ./build-without-os.sh

Running task build-without-os took 115 seconds on my computer, and the image size is 284 MB.

We've lost the build time improvement, but won on the size.

Since we started from scratch, dnf needs to install more packages.

IMHO, it's an acceptable tradeoff since this new image only contains what's necessary.

But from now we only built images, but never tested it, and I can't say it's working if I don't test it.

Testing the image

I won't use a container registry just to test the image, so we're going to export the image and reload it from a container runtime.

Note that buildah builds OCI images, which might not work as-is for docker.

Don't worry, there is an option --format that you can use to build docker compliant images.

In my case, I'm going to use podman to test the image, hereunder a generic task to test images:

  test:
    vars:
      IMAGE_NAME: '{{ default "" .IMAGE_NAME }}'
    env:
      IMAGE_ID:
        sh: buildah images --filter=reference={{ .IMAGE_NAME }} --format='{{ printf "{{ .ID }}" }}'
    desc: "Test"
    cmds:
      - buildah push $IMAGE_ID oci-archive:${PWD}/{{ .IMAGE_NAME }}.tar.gz
      - podman import ./{{ .IMAGE_NAME }}.tar.gz
      - podman run -ti --rm localhost/{{ .IMAGE_NAME }}:$IMAGE_TAG /bin/ansible --version

And now the task to test the image built from scratch:

  test-without-os:
    vars:
      IMAGE_NAME: "buildah-demo-without-os"
    desc: "Load the image in podman and run a test on it"
    cmds:
      - task: test
        vars: { IMAGE_NAME: "{{ .IMAGE_NAME }}" }

If you run task test-without-os ansible version shall be printed.

This is it for now, thank you for reading this post.
Feel free to share it, and see you later for more tech.

Have a nice day!