Using BuildKit for Container Image Building in GitLab CI And Pushing To ECR

Intro

If you're still using Kaniko for building container images in your GitLab CI pipelines, it's time to make a change. Kaniko has been deprecated, and BuildKit has emerged as the modern, efficient alternative for building container images in CI/CD environments. BuildKit offers superior caching capabilities, better performance, and enhanced security features compared to its predecessor. While Gitlab Docs offer examples, I had to spend some time getting our pipelines authenticated for AWS and allowing ECR pushes to work flawlessly.

In this post, I'll walk you through migrating your GitLab pipelines from Kaniko to BuildKit, specifically focusing on building images, authenticating against AWS, and then pushing them to AWS Elastic Container Registry (ECR). The migration makes sure your pipelines stay up to date with industry standards. It should also bring improvements in build speed and reliability!

Step-by-Step Guide

1. Understanding the BuildKit Setup

BuildKit runs in rootless mode for enhanced security, which means it doesn't require privileged access to build container images. The key components of our setup include:

• BuildKit rootless image: moby/buildkit:rootless
• ECR credential helper: For seamless authentication with AWS ECR
• BuildKit daemon: Running in daemonless mode for GitLab CI compatibility

2. Setting Up the Base Job Configuration

First, let's create a reusable job template that contains all the common configuration.
You can reuse this later for different kind of build, e.g. production vs QA builds, or maybe E2E builds.

.build backend:
  stage: build
  tags:
    - medium-runner
  variables:
    GIT_STRATEGY: fetch
    AWS_DEFAULT_REGION: eu-central-1
    ECR_REPOSITORY: $ECR_REPOSITORY
    ECR_REGISTRY: $ECR_REGISTRY
    AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  image:
    name: moby/buildkit:rootless
    entrypoint: [""]

The BUILDKITD_FLAGS: --oci-worker-no-process-sandbox flag is necessary for running BuildKit in GitLab's shared runners without requiring additional privileges and something I had missing in my first jobs.

3. Installing and Configuring ECR Authentication

The before_script section handles the ECR authentication setup:

before_script:
  # Install ecr login helper
  - mkdir credentials-binaries
  - wget -O credentials-binaries/docker-credential-ecr-login https://amazon-ecr-credential-helper-releases.s3.us-east-2.amazonaws.com/0.10.0/linux-amd64/docker-credential-ecr-login
  - chmod +x credentials-binaries/docker-credential-ecr-login
  - chown 1000:1000 credentials-binaries/docker-credential-*
  - export PATH="${CI_PROJECT_DIR}/credentials-binaries:$PATH"

  # Create docker config for ECR authentication
  - mkdir -p ~/.docker
  - |
    cat << EOF > ~/.docker/config.json
    {
        "credHelpers": {
            "$ECR_REGISTRY": "ecr-login"
        }
    }
    EOF

This setup downloads the ECR credential helper and configures Docker to use it for authentication, using your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for that.
The credential helper also handles token refresh, so you don't need manual docker login commands.

4. Building and Pushing Images with BuildKit

The main build logic uses buildctl-daemonless.sh, which is perfect for CI environments
that don't have the docker service available:

script:
  # Build and push image using BuildKit
  - |
    buildctl-daemonless.sh build \
      --frontend dockerfile.v0 \
      --local context=$CI_PROJECT_DIR/backend \
      --local dockerfile=$CI_PROJECT_DIR/backend \
      --export-cache type=registry,ref=$ECR_REPOSITORY:buildcache \
      --import-cache type=registry,ref=$ECR_REPOSITORY:buildcache \
      --output type=image,name=$ECR_REPOSITORY:$IMAGE_TAG,push=true

The build uses registry-based caching for subsequent builds and builds+pushes to ECR in a single operation,
based on the backend/ folder in this example and the backend/Dockerfile.
Of course, this might need adaption in your usecase.

5. Creating Environment-Specific Jobs

Extend the base job for different environments:

build backend test:
  extends:
    - .build backend
  environment:
    name: test
  variables:
    IMAGE_TAG: backend-latest
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: on_success
    - when: never

This pattern allows you to create jobs for different environments (staging, production) or usecases by extending the base job and overriding necessary variables.

6. Required GitLab CI Variables

Ensure these variables are configured in your GitLab project settings:

• ECR_REGISTRY: ECR registry domain (e.g., 123456789.dkr.ecr.eu-central-1.amazonaws.com)
• ECR_REPOSITORY: Full ECR repository URI (e.g., 123456789.dkr.ecr.eu-central-1.amazonaws.com/my-app) (could reuse $ECR_REGISTRY lol)
• AWS_ACCESS_KEY_ID: AWS credentials with ECR push permissions
• AWS_SECRET_ACCESS_KEY: Corresponding AWS secret key

Conclusion

The setup shown here provides a solid foundation that you can customize for your specific needs. Migrating from Kaniko is pretty straightforward with this.
BuildKit brings your CI/CD pipeline up to date with current best practices. It also brings benefits like improved build speeds through better caching, enhanced security with rootless builds, and more reliable image building.
The ECR integration step is something that I needed to pierce together from different sources, but I'm very happy with the setup and the speed.
As you implement this in your own projects, you might want to explore additional BuildKit features like multi-platform builds or custom frontend implementations.

As always, feel free to share your experiences or ask questions about this migration approach down below :-)