Building Multi-Arch Images Inside Kubernetes Using BuildKit

Problem Statement

A SaaS company, when rolling out applications in a customer's setup, often deals with a mix of different architectures. To ensure we can support all these variations, we create Docker images in a multi-architecture format. In the current landscape, many companies leverage CI/CD in Kubernetes and utilize tools like Kaniko for their Docker-building processes. It is important to note that Kaniko ( on the day of writing ) itself lacks support for the multi-architecture build of Docker images.

---

Prerequisite

Before implementing multi-architecture builds using Kubernetes, it's essential to have a Kubernetes cluster with Jenkins Setup and helm installed in your local to set up the buildkit helm chart. At Aurva, we streamline our Jenkins configuration using a shared library, coupled with Jenkins Configuration as Code (JCasC), to set up and manage Jenkins jobs efficiently.

Introduction to BuildKit

Introducing BuildKit, a solution for optimizing container image construction. This tool enhances efficiency, security, and speed in the image-building process and is seamlessly integrated into Docker's latest release (v18.06). As an integral component of the Moby project.

Setting Up BuildKit

Assuming you already have a Jenkins setup in a Kubernetes namespace named Jenkins the next step is to install the BuildKitservice within that namespace.

Step 1: Helm Chart Magic

Start by deploying the BuildKit service using Helm. Execute the following command in your terminal:

helm repo add andrcuns https://andrcuns.github.io/charts
helm install buildkit andrcuns/buildkit-service --namespace jenkins

Step 2: Setting Up Docker Agent

Now, let's take the next stride by tweaking your Kubernetes agent pod in your Jenkins Groovy script. Use the below Dockerfile to create your docker image to be used for building agents

Dockerfile Example

`FROM --platform=${BUILDPLATFORM} docker:rc-dind-rootless as build
ARG TARGETARCH
WORKDIR /home/rootless
RUN wget -O docker-credential-ecr-login https://amazon-ecr-credential-helper-releases.s3.us-east-2.amazonaws.com/0.7.1/linux-${TARGETARCH}/docker-credential-ecr-login

FROM --platform=${TARGETPLATFORM} docker:rc-dind-rootless
COPY --from=build /home/rootless/docker-credential-ecr-login /home/rootless/.local/bin/docker-credential-ecr-login
RUN chmod +x /home/rootless/.local/bin/docker-credential-ecr-login
ENV PATH="/home/rootless/.local/bin:${PATH}"
USER rootless`

Jenkins Agent.yaml
`
def buildkit_agent() {
template = """
apiVersion: v1
kind: Pod
metadata:
name: buildkit-agent-pod
namespace: build
spec:
serviceAccount: "aurva-jenkins-worker-sa"
containers:

• name: buildkit-agent image: docker:rc-dind-rootless command:
  • /usr/bin/bash tty: true

"""
template
}`

Note: "In this blog, we won't delve into an ECR repo configuration. However, if you're keen on setting up an ECR repo, you can leverage AWS ECR Credential Helper to streamline pushing images to ECR."

Step 3: Create a Remote Driver & Execute the Docker Build

1. Remote Driver Setup:

BuildKit requires the creation of a remote driver that points to the BuildKit service. This command can be executed from the Docker docker-agent shell, and running the command will create a remote driver.

docker buildx create --use --driver=remote tcp://buildkit-buildkit-service.jenkins:1234

The above command will create a remote Docker drive and point to the BuildKit using a BuildKit service deployed in the Jenkins namespace.

2. Build Docker Images:

It's time for the final step, run the following command to build Docker images and push them to your Amazon ECR repository.

docker buildx build - push -fpwd/${dockerFilePath} -t ${imageFullName} - platform linux/arm64/v8,linux/amd64pwd`/${buildContextPath}

Output will be

docker buildx build --push -f /home/jenkins/agent/workspace/aurva-gateway/docker/dockerfiles/gateway.Dockerfile --build-arg ENV_TAG= -t .dkr.ecr.ap-south-1.amazonaws.com/: --platform linux/arm64/v8,linux/amd64 /home/jenkins/agent/workspace/aurva-gateway/`

In the aforementioned docker buildx command, there is a 'platform' parameter where you can define all the platforms for which you want to build the image

Conclusion

In a nutshell, this guide explores implementing multi-architecture Docker builds in Kubernetes using BuildKit. We addressed the challenge of a diverse docker arch, introduced BuildKit as a seamless solution integrated into Docker, and outlined key steps for setup. By adopting these practices, you'll boost your CI/CD pipelines for flexible and efficient deployment across various architectures, staying ahead in containerized application development.