DEV Community

Cover image for Multi-architecture Kubernetes clusters on Amazon EKS
AWS Community Builders profile image Jason Andrews
Jason Andrews for AWS Community Builders

Posted on • Edited on • Originally published at learn.arm.com

Multi-architecture Kubernetes clusters on Amazon EKS

#kubernetes #aws #docker

AWS Graviton processors continue to deliver price performance for cloud workloads. Migrating Kubernetes applications can be tricky, but AWS and Docker provide tools to make the process easier.

You can experiment with multi-architecture clusters to better understand the price performance benefits, and select the best hardware for the task. Multi-architecture clusters also ease migration when some software components are not available for the Arm architecture.

Refer to the The Insider’s Guide to Building a Multi-Arch Infrastructure for more information on the benefits of multi-architecture Kubernetes clusters.

Multi-architecture Kubernetes clusters

A multi-architecture Kubernetes cluster runs workloads on multiple hardware architectures, typically arm64 and amd64 (using Docker terminology).

To learn more about multi-architecture Kubernetes you can create a cluster in Amazon EKS and gain some practical experience with arm64 and amd64 nodes. This will also help you understand multi-architecture container images.

Before you begin

You will need docker, eksctl and kubectl on your local computer to try out the steps below.

Your local computer can be Windows, Linux, or macOS. The instructions generally assume Linux.

Multipass from Canonical is a great tool for Linux virtual machines if you use macOS.

You can refer to the Kubectl install guide and the AWS EKS CLI install guide for installation details. The guides provide details for Arm Linux machines, but links to installation information about other platforms are also provided.

There are plenty of places to find Docker installation instructions. One option is the Docker install guide.

You will also need an AWS account.

To create an account, go to https://aws.amazon.com and click on Create an AWS Account in the top right corner. Follow the instructions to register. See the Creating an AWS account documentation for full instructions.

Make sure to configure your access key ID and secret access key, which are used to sign programmatic requests that you make to AWS. Refer to AWS Credentials for a quick summary of how to run aws configure.

Multi-architecture containers

Before you can try out a multi-architecture Kubernetes cluster, you need a multi-architecture container image.

Multi-architecture container images are the easiest way to deploy applications, and hide the underlying hardware architecture. Building multi-architecture images is slightly more complex compared to building single-architecture images.

Docker provides two ways to create multi-architecture images:

  • docker buildx builds both architectures at the same time
  • docker manifest builds each architecture separately and joins them together into a multi-architecture image

You can read more about how to create container images in the article Using Docker manifest to create multi-arch images on AWS Graviton processors

Below is a simple Go application you can use to learn about multi-architecture Kubernetes clusters.

Use your favorite text editor to create a file named hello.go with the contents below:

// Copyright 2022 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
    "fmt"
    "log"
    "net/http"
    "os"
    "runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello from image NODE:%s, POD:%s, CPU PLATFORM:%s/%s",
        os.Getenv("NODE_NAME"), os.Getenv("POD_NAME"), runtime.GOOS, runtime.GOARCH)
}

func main() {
    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}
Enter fullscreen mode Exit fullscreen mode

Next, create a file named go.mod with the following two lines:

module example.com/arm
go 1.21
Enter fullscreen mode Exit fullscreen mode

Create a third file named Dockerfile with the contents:

ARG T

#
# Build: 1st stage
#
FROM golang:1.21-alpine as builder 
ARG TARCH
WORKDIR /app
COPY go.mod .
COPY hello.go .
RUN GOARCH=${TARCH} go build -o /hello && \
    apk add --update --no-cache file && \
    file /hello   

#
# Release: 2nd stage
#
FROM ${T}alpine
WORKDIR /
COPY --from=builder /hello /hello
RUN apk add --update --no-cache file
CMD [ "/hello" ]
Enter fullscreen mode Exit fullscreen mode

Docker buildx

With the three text files created, you can build the container using docker buildx.

Before you start, log in to Docker Hub:

docker login
Enter fullscreen mode Exit fullscreen mode

Run the commands below to build the image, and make sure to substitute your own Docker ID instead of jasonrandrews.

docker buildx create --use --name builder
docker buildx build --platform linux/amd64,linux/arm64 -t jasonrandrews/go-arch-x:1.0 --push .
Enter fullscreen mode Exit fullscreen mode

When the commands are complete, you will see the new image in your Docker Hub account. Note, this is the go-arch-x repository.

Docker manifest

You can also use docker manifest to create a multi-architecture image from two single-architecture images. This is an alternative way to to build the multi-architecture image, but you can also use it to learn how to deploy a container image that only supports one architecture.

Substitute your Docker ID instead of jasonrandrews and run these commands on an amd64 machine:

# Build for amd64
docker build -t jasonrandrews/go-arch:amd64 --build-arg TARCH=amd64 --build-arg T=amd64/ .
docker push jasonrandrews/go-arch:amd64
Enter fullscreen mode Exit fullscreen mode

Substitute your Docker ID instead of jasonrandrews and run these commands on an arm64 machine:

# Build for arm64
docker build -t jasonrandrews/go-arch:arm64 --build-arg TARCH=arm64 --build-arg T=arm64v8/ .
docker push jasonrandrews/go-arch:arm64
Enter fullscreen mode Exit fullscreen mode

After building individual containers for each architecture, merge them into a single image by running the commands below on either architecture:

docker manifest create jasonrandrews/go-arch:1.0 \
--amend jasonrandrews/go-arch:arm64 \
--amend jasonrandrews/go-arch:amd64
docker manifest push --purge jasonrandrews/go-arch:1.0
Enter fullscreen mode Exit fullscreen mode

You now have two single-architecture images in your Docker Hub account and the combined multi-architecture image in the same repository. Note, this is the go-arch repository.

In the sections below, you will learn how to deploy the images in Amazon EKS.

Create a multi-architecture Kubernetes cluster

You can use eksctl to create a cluster with multiple architectures.

First, use a text editor to create a text file cluster.yaml with the contents below:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster1
  region: us-east-1

nodeGroups:
  - name: ng-1
    instanceType: t2.micro
    desiredCapacity: 2
    volumeSize: 80
  - name: ng-2
    instanceType: t4g.small
    desiredCapacity: 2
    volumeSize: 80
Enter fullscreen mode Exit fullscreen mode

The instance types above take advantage of free tier and the T4g free trial.

Run the eksctl command with the yaml file to create an EKS cluster:

eksctl create cluster -f cluster.yaml
Enter fullscreen mode Exit fullscreen mode

The cluster includes two amd64 nodes and two arm64 nodes.

It takes 15-20 minutes to launch the cluster so take a break while it is created.

When the cluster is ready check the nodes by running kubectl on your local machine:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

The output shows four nodes, but doesn't print the architecture.

NAME                             STATUS   ROLES    AGE    VERSION
ip-192-168-11-18.ec2.internal    Ready    <none>   106s   v1.27.5-eks-43840fb
ip-192-168-27-254.ec2.internal   Ready    <none>   71s    v1.27.5-eks-43840fb
ip-192-168-32-244.ec2.internal   Ready    <none>   106s   v1.27.5-eks-43840fb
ip-192-168-61-31.ec2.internal    Ready    <none>   72s    v1.27.5-eks-43840fb
Enter fullscreen mode Exit fullscreen mode

To get the architecture run:

kubectl get node -o jsonpath='{.items[*].status.nodeInfo.architecture}'
Enter fullscreen mode Exit fullscreen mode

The output shows four nodes, two of each architecture.

arm64 amd64 amd64 arm64
Enter fullscreen mode Exit fullscreen mode

The Kubernetes cluster is now ready to use, and you can deploy the Go app.

Create a service to deploy

To deploy the application create a text file named hello-service.yaml with the contents:

apiVersion: v1
kind: Service
metadata:
  name: hello-service
  labels:
    app: hello
    tier: web
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: hello
    tier: web
Enter fullscreen mode Exit fullscreen mode

Deploy the service by running:

kubectl apply -f hello-service.yaml
Enter fullscreen mode Exit fullscreen mode

The output is:

service/hello-service created
Enter fullscreen mode Exit fullscreen mode

Deploy the amd64 application

You can deploy the amd64 container first to demonstrate a single-architecture container.

Create a text file named amd64-deployment.yaml with the contents below. Make sure to use your Docker ID instead of jasonrandrews in the file. Notice that the container image is go-arch:amd64. The amd64 image will only run on amd64 nodes. The nodeSelector is used to make sure the continer is only scheduled on amd64.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: amd-deployment
  labels:
    app: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
      - name: hello
        image: jasonrandrews/go-arch:amd64
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        resources:
          requests:
            cpu: 300m
      nodeSelector:
        kubernetes.io/arch: amd64
Enter fullscreen mode Exit fullscreen mode

Deploy the amd64 container by running:

kubectl apply -f amd64-deployment.yaml
Enter fullscreen mode Exit fullscreen mode

You can see the running deployment.

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

The output shows 1 pod running:

NAME                              READY   STATUS    RESTARTS   AGE
amd-deployment-7d4d44889d-vzhpd   1/1     Running   0          9s
Enter fullscreen mode Exit fullscreen mode

To get the endpoint of the service run:

kubectl get services
Enter fullscreen mode Exit fullscreen mode

The output prints the IP address of the load balancer.

NAME            TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)        AGE
hello-service   LoadBalancer   10.100.188.181   a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com   80:30489/TCP   70s
kubernetes      ClusterIP      10.100.0.1       <none>                                                                    443/TCP        14m
Enter fullscreen mode Exit fullscreen mode

To confirm the amd64 application is running use curl and substitute the endpoint of your load balancer:

curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

The output displays the hello message from the application and confirms the architecture is amd64.

Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:amd-deployment-7d4d44889d-vzhpd, CPU PLATFORM:linux/amd64
Enter fullscreen mode Exit fullscreen mode

Save the address of your load balancer (the http address) and use it for the curl commands in the following sections.

Deploy the arm64 application

Create a text file named arm64-deployment.yaml with the contents below. Again, substitute your Docker ID and notice that the nodeSelector is now arm64.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arm-deployment
  labels:
    app: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
      - name: hello
        image: jasonrandrews/go-arch:arm64
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        resources:
          requests:
            cpu: 300m
      nodeSelector:
        kubernetes.io/arch: arm64
Enter fullscreen mode Exit fullscreen mode

Deploy the arm64 container by running:

kubectl apply -f arm64-deployment.yaml
Enter fullscreen mode Exit fullscreen mode

You can see the running deployment.

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Now, the output shows both deployments, amd64 and arm64.

NAME                              READY   STATUS    RESTARTS   AGE
amd-deployment-7d4d44889d-vzhpd   1/1     Running   0          3m1s
arm-deployment-5996f6b85d-wp8cv   1/1     Running   0          10s
Enter fullscreen mode Exit fullscreen mode

Run the curl command a few times and you will see a mix of amd64 and arm64 messages printed, depending on which node services the request.

curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Deploy the multi-architecture application

You can also deploy the multi-architecture image.

Create a text file named multi-arch-deployment.yaml with the contents below. Again, substitute your Docker ID. The image is the multi-architecture image created with docker buildx and 6 replicas are specified.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-arch-deployment
  labels:
    app: hello
spec:
  replicas: 6
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
      - name: hello
        image: jasonrandrews/go-arch-x:1.0
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        resources:
          requests:
            cpu: 300m
Enter fullscreen mode Exit fullscreen mode

Deploy the multi-architecture container by running:

kubectl apply -f multi-arch-deployment.yaml
Enter fullscreen mode Exit fullscreen mode

You can see the running deployment.

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Now, the output shows all three deployments:

NAME                                     READY   STATUS    RESTARTS   AGE
amd-deployment-7d4d44889d-vzhpd          1/1     Running   0          4m24s
arm-deployment-5996f6b85d-wp8cv          1/1     Running   0          93s
multi-arch-deployment-547684cd44-7w6vf   1/1     Running   0          7s
multi-arch-deployment-547684cd44-chcc8   1/1     Running   0          7s
multi-arch-deployment-547684cd44-kw966   1/1     Running   0          7s
multi-arch-deployment-547684cd44-r4789   1/1     Running   0          7s
multi-arch-deployment-547684cd44-vqwc7   1/1     Running   0          7s
multi-arch-deployment-547684cd44-w4x8d   1/1     Running   0          7s
Enter fullscreen mode Exit fullscreen mode

To test the appliation, run a loop of curl commands and see the output messages from the application.

for i in $(seq 1 10); do curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com; done
Enter fullscreen mode Exit fullscreen mode

The output will show a variety of amd64 and arm64 messages:

Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-27-254.ec2.internal, POD:multi-arch-deployment-547684cd44-kw966, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-chcc8, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-27-254.ec2.internal, POD:arm-deployment-5996f6b85d-wp8cv, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
Enter fullscreen mode Exit fullscreen mode

You have now deployed single-architecture images for amd64 and arm64 as well as six copies of a multi-architecture image.

You can use the same techniques to incrementally migrate applications from amd64 to arm64 and invesigate the price performance provided by AWS Graviton processors.

Delete the clusters

When you are done, make sure to delete all AWS resources.

You can clean up with a single eckctl command.

eksctl delete cluster -n cluster1
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

duncanista profile image

Reduce AWS Lambda Cold Starts in .NET

jordan gonzález -

ccmjga profile image

A brand new Java scaffolding has been born today for Make Java Great Again!

CCmjga -

alexandrammr profile image

Countdown to re:Invent is on: 5 days to go

Alex Radu -

franckpachot profile image

Aurora Limitless - SQL Limitations

Franck Pachot -