Accelerating CI Pipelines with Docker Buildx and BuildKit Caching

Introduction

Efficient CI pipelines are critical to smooth development workflows, and caching plays a significant role in reducing build times. As CI workflows increasingly leverage Docker, understanding how Docker build caching works is essential for optimizing performance, especially in complex projects. In this post, we’ll dive into the fundamentals of Docker build caching, explore the components needed for effective caching with GitHub Actions, and walk through practical examples.

Docker’s v23 update introduced a significant shift in its build architecture, standardizing the use of Buildx as the build client and BuildKit as the backend. This change provides developers with the power to optimize their build processes, yet caching in CI still requires a deliberate setup, particularly when working with Docker Compose. Here, we’ll break down the basics, explain the necessary tooling, and demonstrate how to implement build caching in CI using GitHub Actions.

Docker Build in v23 and Beyond

As of Docker v23, which was released in February 2023, the traditional build system transitioned to a new architecture. The docker build command now defaults to using Buildx as the client and BuildKit as the backend for managing the build process.

This shift is important because BuildKit significantly enhances the performance of Docker builds by allowing parallel execution, efficient layer caching, and better resource utilization. To benefit from these improvements, it is crucial to understand how to use Buildx and configure caching effectively, particularly in CI environments.

Why Docker Compose Does Not Automatically Use BuildKit

While the standard docker build command has adopted Buildx and BuildKit, Docker Compose continues to use the traditional build system by default. When running docker compose build, neither Buildx nor BuildKit is utilized unless explicitly configured.

This is important because Docker Compose, a tool frequently used for defining and running multi-container Docker applications, does not take advantage of the caching optimizations offered by BuildKit out of the box. To leverage caching in a CI pipeline, you must configure Docker Compose to use Buildx and BuildKit manually.

Specifying Build Cache Inputs and Outputs

To enable caching, the build cache must be inputted and outputted to a specific location in the file system. Docker provides options for specifying external cache sources and destinations via --cache-from and --cache-to:

docker build --cache-from type=local,src=/path/to/dir --cache-to type=local,dest=/path/to/dir ...

• --cache-from: Specifies the source of the build cache.
• --cache-to: Defines where the cache should be exported after the build.

By using these flags, you can share and store cache layers between builds, ensuring faster rebuilds by skipping already cached layers. This setup is especially useful in CI environments where each build typically starts from scratch.

The Need for Buildx and docker-container Driver

However, when running the above command in a local environment, you might encounter the following error:

Cache export is not supported for the docker driver.
Switch to a different driver, or turn on the containerd image store.

This happens because the legacy Docker driver (docker) does not support BuildKit’s cache export features. To resolve this, you need to use the docker-container driver, which fully supports BuildKit’s advanced features. (docs)

To create a new builder that uses the docker-container driver, run:

docker buildx create --name mybuilder --driver docker-container
docker buildx use mybuilder

This sets up a Buildx builder that uses the docker-container driver. Now, you can use the --cache-from and --cache-to options to cache Docker layers effectively.

Example: Building with Buildx and Caching

Once the new builder is created, you can use the following command to build and cache your image:

docker buildx build --builder mybuilder \
  --cache-from=type=local,src=/path/to/dir \
  --cache-to=type=local,dest=/path/to/dir ...

Caching with Docker Compose

As mentioned earlier, Docker Compose does not automatically utilize BuildKit. To achieve caching with Docker Compose, you need to use the docker buildx bake command. Bake is a tool in Buildx that allows multi-step orchestration of Docker builds, including advanced caching options.

Here’s how you can set up Docker Compose to leverage BuildKit caching:

docker buildx bake --file docker-compose.yml \
  --builder mybuilder \
  --cache-from=type=local,src=/path/to/cache \
  --cache-to=type=local,dest=/path/to/cache,mode=max

This approach bypasses the default docker compose build command and uses Buildx’s bake to handle the build process with caching.

Implementing Caching in GitHub Actions

To use Docker build caching in GitHub Actions, you’ll need to set up caching for the build directory, configure Buildx, and run docker buildx bake as part of the CI workflow. Here’s a sample workflow file that illustrates this:

name: CI Build

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332

    - name: Setup Docker Buildx
      id: buildx
      uses: docker/setup-buildx-action@988b5a0280414f521da01fcc63a27aeeb4b104db

    - name: Cache Docker layers
      uses: actions/cache@v3
      with:
        path: /tmp/.buildx-cache
        key: docker-build-cache-${{ github.ref }}-${{ github.sha }}
        restore-keys: |
          docker-build-cache-${{ github.ref }}
          docker-build-cache-

    - name: Build with caching
      run: |
        docker buildx bake --file docker-compose.yml \
          --builder="${{ steps.buildx.outputs.name }}" \
          --cache-from=type=local,src=/tmp/.buildx-cache \
          --cache-to=type=local,dest=/tmp/.buildx-cache,mode=max

Verifying the Cache

Once the workflow completes, you can inspect the contents of the build cache stored in /tmp/.buildx-cache. The cache is typically stored in OCI (Open Container Initiative) image format, with each image layer cached in the blobs directory. The index.json file in the cache contains metadata about the cached layers.

Here’s a snippet of what the index.json file might look like:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:c6fc9d593e92059b058b0b147be8e51...",
      "size": 4193,
      "annotations": {
        "org.opencontainers.image.ref.name": "latest"
      }
    }
  ]
}

Conclusion

In this guide, we covered the essentials of setting up Docker build caching in CI, focusing on Buildx, BuildKit, and GitHub Actions. Docker Compose doesn’t use BuildKit by default, but by utilizing the buildx bake command, you can implement effective caching strategies. While caching might not always result in significant performance improvements depending on the use case, understanding and leveraging Docker’s build architecture will put you in a better position to optimize your CI pipelines.

By following these steps, you can make sure your builds are faster, more efficient, and consistent across environments.

Thank you for reading, and happy optimizing!

---

For more tips and insights on security and log analysis, follow me on Twitter @Siddhant_K_code and stay updated with the latest & detailed tech content like this.