Understanding Multi-arch Containers, Benefits and CI/CD Integration

Have you ever seen “exec /docker-entrypoint.sh: exec format error” error message on your server while running any docker image or Kubernetes pods? This is most probably because you are running some other CPU architecture container image on your server OR did you ever use --platform linux/x86_64 option on your Apple silicon M1, M2 MacBook? If yes, then you are not getting the native performance of Apple silicon and it may be draining your MacBook battery.
To avoid this kind of error and performance issue, we need to run the correct multi-arch container image or we may need to build our own image because all container public image does not have multi-arch image available.

In this blog post, we will learn what are multi-arch container images? How it works? How to build and promote them? and we will write a sample code for building a multi-arch image in the CI/CD pipeline.

What is a Multi-arch Container Image?

A multi-arch Docker image is a list of images that has references to binaries and libraries compiled for multiple CPU architectures. This type of image is useful when we need to run the same application on different CPU architectures (ARM, x86, RISC-V, etc) without creating separate images for each architecture.

Multi-arch Container Use Cases

Performance and cost optimization: Container multi-arch is used to optimize performance on different CPU architectures. By building and deploying images that are optimized for specific architecture, we can achieve better performance and reduce resource usage. Using Karpenter we can easily deploy our workload to arm64 and get the benefit of AWS Graviton’s performance and cost savings.

Cross-platform development: If you are developing an application that needs to run on multiple platforms, such as ARM and x86, you can use buildx to build multi-arch Docker images and test the application on different architectures.

IoT devices: Many IoT/Edge devices use ARM processors, which require different binaries and libraries than x86 processors. With multi-arch images, you can create an image that can run on ARM, x86, and RISCV devices, making it easier to deploy your application to a wide range of IoT devices.

Benefits of Using Multi-arch Container Image

Several advantages of using multi-arch container images are:

• Ability to run Docker image on multiple CPU architectures
• Enables us to choose eco-friendly CPU architecture
• Seamless migration from one architecture to another
• Better performance and cost saving using arm64
• Ability to support more cores per CPU using arm64

How to Build Multi-arch Container Image?

There are multiple ways to build a multi-arch container but we will be focusing on widely used and easy methods.

1. Traditional Docker build command
2. Using Docker buildx

Using Traditional Docker Build Command

In this tutorial, we will manually build both images on different CPU architecture machines and push them to the container registry (eg. Dockerhub) and then create the manifest file which has both image references. A manifest file is a simple JSON file containing the index of container images and its metadata like the size of image, sha256 digest, OS, etc. We will see more about manifest file later in this blog.

For eg. this is our basic Dockerfile.

FROM nginx
RUN echo “Hello multiarch” > /usr/share/nginx/html/index.html

########## on amd64 machine ##########
docker build -t username/custom-nginx:v1-amd64 .
docker push username/custom-nginx:v1-amd64

########## on arm64 machine ##########
docker build -t username/custom-nginx:v1-arm64 .
docker push username/custom-nginx:v1-arm64

########## Create a manifest index file ##########
docker manifest create \
    username/custom-nginx:v1 \
    username/custom-nginx:v1-amd64 \
    username/custom-nginx:v1-arm64

########## Push manifest index file ##########
docker manifest push username/custom-nginx:v1

Using Docker Buildx

With buildx, we just need to run one single command with parameterized architecture.

docker buildx build \
--push \
--platform linux/arm64,linux/amd64 \ 
-t username/custom-nginx:v1 .

In the background, the Docker buildx command uses buildkit so when we run the above command, it creates one container with moby/buildkitd image, which has QEMU binary for multiple CPU architectures which are responsible for the emulating CPU instruction sets. We can view these QEMU binaries by running ls /usr/bin/buildkit-qemu-* inside the running buildkit container.

In the above command, we passed --platform linux/arm64,linux/amd64 so it uses the /usr/bin/buildkit-qemu-aarch64 QEMU binary for building linux/arm64 image and linux/amd64 are natively built on the host machine. Once both images are built, then it uses the --push option to create the manifest file and pushes both images to the registry server with the manifest file.

By inspecting the manifest file we can see “Ref” contains the actual image link which will be fetched when platform[0].architecture matches the host system architecture.

$ docker manifest inspect -v nginx

[
        {
                "Ref": "docker.io/library/nginx:latest@sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",
                        "size": 1570,
                        "platform": {
                                "architecture": "amd64",
                                "os": "linux"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 7916,
                                "digest": "sha256:080ed0ed8312deca92e9a769b518cdfa20f5278359bd156f3469dd8fa532db6b"
                        },
….

        {
                "Ref": "docker.io/library/nginx:latest@sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",
                        "size": 1570,
                        "platform": {
                                "architecture": "arm64",
                                "os": "linux",
                                "variant": "v8"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 7932,
                                "digest": "sha256:f71a4866129b6332cfd0dddb38f2fec26a5a125ebb0adde99fbaa4cb87149ead"
                        }

We can also use buildx imagetools command to view the same output in a more human-readable format.

$ docker buildx imagetools inspect sonarqube:10.0.0-community

Name:      docker.io/library/sonarqube:10.0.0-community
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:51588fac6153b949af07660decfe20b5754da9fd12c82db5d95a0900b6024196

Manifests:
  Name:      docker.io/library/sonarqube:10.0.0-community@sha256:8b536568cd64faf15e1e5be916cf21506df70e2177061edfedfd22f255a7b1a0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/library/sonarqube:10.0.0-community@sha256:2163e9563bbba2eba30abef8c25e68da4eb20e6e0bb3e6ecc902a150321fae6b
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8

If you are having any issues building multi-arch images, you can run the following command to reset the /proc/sys/fs/binfmt_misc entries.

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

We can also build multi-arch container images using Buildah as well.

How do Multi-arch Container Images Work?

As we can see in the diagram, the host machine has x86/amd64 CPU architecture, and on top of that, we install operating systems which can be Windows or Linux. Windows requires WSL or LinuxKit to run Docker. It uses QEMU to emulate multiple CPU architectures and Dockerfile builds run inside this emulation.

When we run the docker pull or build command, it fetches the requested manifest file from the registry server. These manifest files are JSON file that can have one Docker image reference or contains more than one image list. It fetches the correct image depending on the host machine's CPU architecture.

How to Integrate Multi-arch Container Build with CI/CD?

If your workload runs on multiple machines with different CPU architectures, it is always better to build multi-arch Docker images for your application. Integrating multi-arch build into CI/CD streamlines the image build and scan process easier, adds only one Docker tag, and saves time. Below we have written Jenkins and GitHub CI sample code for building multi-arch images.

Jenkins Multi-arch CI

Currently, the Jenkins Docker plugin does not support multi-arch building so we can use buildx to build multi-arch images.

pipeline
{
    agent {
        label 'worker1'
    }

    options{
        timestamps()
        timeout(time: 30, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '10'))
    }
    environment {
        DOCKER_REGISTRY_PATH = "https://registry.example.com"
        DOCKER_TAG = "v1"
    }


    stages
    {
        stage('build-and-push')
        {
        steps{
          script{
            docker.withRegistry(DOCKER_REGISTRY_PATH, ecrcred_dev){
            sh '''
              ####### check multiarch env ###########
              export DOCKER_BUILDKIT=1
              if [[ $(docker buildx inspect --bootstrap | head -n 2 | grep Name | awk -F" " '{print $NF}') != "multiarch" ]]
              then
                docker buildx rm multiarch | exit 0
                docker buildx create --name multiarch --use
                docker buildx inspect --bootstrap
              fi
              ####### Push multiarch ###########
              docker buildx build --push --platform linux/arm64,linux/amd64 -t "$DOCKER_REGISTRY_PATH"/username/custom-nginx:"$DOCKER_TAG" .
            '''
            }
          }
        }
      }
    }
}

Otherwise, we can use the traditional Docker build command as shown above in Jenkins stages with different sets of Jenkins worker nodes.

GitHub CI Pipeline for Building Multi-arch Container Images

GitHub Actions also supports multi-arch container images. It also uses QEMU CPU emulation in the background.

name: docker-multi-arch-push
on:
  push:
    branches:
      - 'main'
jobs:
  docker-build-push:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3


      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2


      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2


      - name: Login to docker.io container registry
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USER }}
          password: ${{ secrets.DOCKER_TOKEN }}


      - name: Build and push
        id: docker_build
        uses: docker/build-push-action@v3
        with:
          context: .
          file: ./Dockerfile
          platforms: linux/amd64,linux/arm64
          push: true
          tags: username/custom-nginx:latest

How to Promote your Multi-arch Image to Higher Environments?

Promoting Docker multi-arch requires a few extra steps as the docker pull command only pulls a single image based on the host machine's CPU architecture. To promote multi-arch Docker images we need to pull all CPU architecture images one by one by using –plarform=linux/$ARCH and then create a manifest file and push them to the new registry server. To avoid these complex steps we can leverage the following tools.

Skopeo or Crane can be used to promote our multi-arch image from one account to another using just a single command. In the background, what these tools do is use Docker API to fetch all multi-arch images and then create manifest and push all images and manifest.

$ skopeo login --username $USER docker.io

$ skopeo copy -a docker://dev-account/custom-nginx:v1 docker://prod-account/custom-nginx:v1

What if you want to use only the Docker command to promote this image to a higher environments (Production)?

####### Pull DEV images ###########
docker pull --platform=amd64 "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG"
docker pull --platform=arm64 "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG"


####### Tag DEV image with STAGE ###########
docker tag "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG" "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64


docker tag "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG" "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64


####### Push amd64 and arm64 image to STAGE ###########
docker push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64
docker push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64


####### Create mainfest and push to STAGE ###########
docker manifest create \
    "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG" \
    --amend "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64 \
    --amend "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64


docker manifest push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"

How to Scan Multi-arch Images for Vulnerabilities?

We can use any tool like Trivy, Gryp, or Docker scan for image scanning but we have to pull multi-arch images one by one and then scan them because by default Docker pull command will only fetch the one image that matches with the Host CPU. We can leverage the Docker pull command with --platform={amd64, arm64} to pull different CPU arch images. Here is how we can do that:

####### Pull amd64 image and scan ###########
docker pull --platform=amd64 nginx:latest
trivy image nginx:latest


####### Pull arm64 image and scan ###########
docker pull --platform=arm64 nginx:latest
trivy image nginx:latest

Some Caveats of Using Multi-arch Container

There are prominent benefits of using multi-arch containers, but there are some caveats that you should certainly be aware of before taking the leap.

• It takes extra storage for storing other arch images.
• Takes time to build the multi-arch container image also while building the arm64 on QEMU emulation consumes lots of time and resources.
• Performance is significantly slower with emulation to run binaries on different CPUs compared to running the binaries natively.
• There are still some issues with buildx building arm64 images like the base image not being available in arm64 and also performing sudo level access or building cross-compile statically linked binary requires an extra step.
• Need container scanning for all images one by one.
• Buildx multi-arch builds are only supported on amd64 CPU architecture.

Conclusion

In this blog, we saw what are multi-arch containers and their use cases. We integrated multi-arch build with Jenkins and Github CI with sample code and provides you the several ways to promote and scan multi-arch container images, and finally, we learned the caveats of using multi-arch containers.

Using multi-arch images gives us the ability to build once and run everywhere. We can seamlessly migrate from one CPU arch to another CPU with ease. Also by deploying images that are optimized for specific architectures, we can achieve better performance and reduce resource costs.

Thanks for reading this post. We hope it was informative and engaging for you. We would love to hear your thoughts on this post, so do start a conversation on LinkedIn.

Looking for help with building your DevOps strategy or want to outsource DevOps to the experts? Learn why so many startups & enterprises consider us as one of the best DevOps consulting & services companies.

Refrences

• Multi-arch build and images, the simple way
• Building Multi-Architecture Containers with Buildah
• Building Multi-Architecture Docker Images With Buildx
• Multiarch qemu-user-static
• How to Build Multi-Arch Docker Images
• Building multi-platform images
• Building Multi-Arch Images for Arm and x86 with Docker Desktop