Provisioning a GPU Node Pool With Autoscaling on Amazon EKS

#ai #machinelearning #tutorial #llm

A GPU node group is easy to create and easy to create in a state where pods requesting nvidia.com/gpu stay pending forever. Two steps are responsible for nearly all of those cases, and neither is part of creating the node group.

Choosing the AMI type

EKS publishes accelerated variants of its optimised AMIs that carry the NVIDIA kernel driver and container toolkit, so you do not install drivers yourself. AWS documents the current Amazon Linux 2023 variants as AL2023_x86_64_NVIDIA for x86 GPU instances and AL2023_ARM_64_NVIDIA for the Arm ones, alongside Bottlerocket equivalents (AWS, EKS-optimized accelerated AMIs for GPU instances).

One difference between the two families decides how much work is left after node creation, and it is stated plainly in the same document: the EKS-optimized AL2023 NVIDIA AMIs do not include the NVIDIA Kubernetes device plugin, and it must be installed separately, whereas the Bottlerocket NVIDIA variants ship it. If you are on Bottlerocket you can skip a section below; if you are on AL2023 you cannot, and skipping it is the single most common reason a correctly-sized GPU node reports zero allocatable GPUs.

Driver versions bundled in these AMIs move with releases — AWS documents driver 580 in the current accelerated AL2023 AMIs at the time of writing, and notes that some newer instance families require a newer driver than the AMI carries. Check the AMI release notes against your intended instance type before you commit to a family.

Creating the node group

Create it with a minimum size of zero so an idle cluster costs nothing, and taint it so that ordinary workloads cannot land on expensive hardware and block a scheduling decision the autoscaler is waiting on.

aws eks create-nodegroup \
  --cluster-name inference \
  --nodegroup-name gpu-g5 \
  --node-role arn:aws:iam::123456789012:role/eks-gpu-node-role \
  --subnets subnet-0a1b2c3d4e5f67890 subnet-1a2b3c4d5e6f78901 \
  --instance-types g5.xlarge \
  --ami-type AL2023_x86_64_NVIDIA \
  --capacity-type ON_DEMAND \
  --disk-size 200 \
  --scaling-config minSize=0,maxSize=8,desiredSize=0 \
  --labels accelerator=nvidia,workload=inference \
  --taints 'key=nvidia.com/gpu,value=true,effect=NO_SCHEDULE'

Three of those arguments deserve a sentence each.

--disk-size 200 because model images are large. A CUDA base image plus a framework plus weights routinely exceeds the default root volume, and the failure mode is a node that joins, starts pulling, and evicts everything under disk pressure — which looks like a scheduling problem rather than a storage one.
--taints with a matching toleration on your inference pods. Without it, any pod at all can be scheduled onto a GPU node, and the autoscaler will keep a node alive for a workload that never needed a GPU.
Subnets in the availability zones that have the capacity. GPU instance families are not offered in every zone of every region, and a node group spanning a zone without capacity produces intermittent scale-up failures whose error text is about capacity, not configuration.

Managed node groups place these nodes in an EC2 Auto Scaling group they own. You scale the node group, never the ASG directly — a manual ASG edit is reconciled away, and the autoscaler and EKS end up disagreeing about the desired count.

The device plugin the AMI does not include

Kubernetes does not know what a GPU is. Extended resources such as nvidia.com/gpu are advertised to the kubelet by a device plugin running as a DaemonSet, which discovers the hardware and reports a count. Without it, kubectl describe node shows a node with the right instance type, working drivers, and no nvidia.com/gpu in its allocatable resources at all — so a pod requesting one is unschedulable, and the autoscaler sees no node shape that could ever satisfy it.

Install the NVIDIA device plugin via its Helm chart, pinning the chart version rather than tracking latest, and give it a toleration for the taint you just added or it will not schedule onto the nodes it is meant to serve:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update

helm upgrade --install nvidia-device-plugin nvdp/nvidia-device-plugin \
  --namespace kube-system \
  --version "<pin the current chart release>" \
  --set-json 'tolerations=[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}]' \
  --set nodeSelector.accelerator=nvidia

If you use the NVIDIA GPU Operator instead, AWS documents that you must disable its driver and toolkit installation on these AMIs, because both are already present and installing them twice breaks the node. Confirm the result on a running node before moving on:

kubectl get nodes -l accelerator=nvidia \
  -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

Making it scale from zero

Cluster Autoscaler decides whether to add a node by simulating whether a pending pod would fit on a node from that group. With at least one node running it inspects a real node. With zero nodes it has nothing to inspect and falls back to a node template, which it builds from the instance type and from tags on the Auto Scaling group. CPU and memory it can infer from the instance type; an extended resource like nvidia.com/gpu is not part of the EC2 instance description in a form it can rely on, which is why the tag exists.

The autoscaler also needs the discovery tags to consider the group at all, and needs the label and taint tags to know that a pod with a node selector and a toleration would actually land there.

ASG=$(aws eks describe-nodegroup \
  --cluster-name inference --nodegroup-name gpu-g5 \
  --query 'nodegroup.resources.autoScalingGroups[0].name' --output text)

aws autoscaling create-or-update-tags --tags \
  "ResourceId=$ASG,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/enabled,Value=true,PropagateAtLaunch=false" \
  "ResourceId=$ASG,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/inference,Value=owned,PropagateAtLaunch=false" \
  "ResourceId=$ASG,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/node-template/resources/nvidia.com/gpu,Value=1,PropagateAtLaunch=false" \
  "ResourceId=$ASG,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/node-template/label/accelerator,Value=nvidia,PropagateAtLaunch=false" \
  "ResourceId=$ASG,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/node-template/taint/nvidia.com/gpu,Value=true:NoSchedule,PropagateAtLaunch=false"

The resource value must match the GPU count of the instance type in that group, which is the argument for one node group per instance type rather than a mixed list. If a group offers both a one-GPU and a four-GPU instance type, the template is wrong for one of them, and the autoscaler either refuses a scale-up that would have worked or launches a node that cannot hold the pod. Where you do run several similar groups — one per availability zone, typically — run the autoscaler with --balance-similar-node-groups so it spreads rather than piling into one zone.

This is also the point where Karpenter is worth considering instead: it provisions instances directly against pod requirements and does not need a node template, which removes this entire class of tag mismatch. The general trade-off is on GPU autoscaling.

Verifying a scale-up end to end

Test with a pod that requests a GPU and does something that proves the device is really attached, from a cluster currently at zero GPU nodes.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  nodeSelector:
    accelerator: nvidia
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: smi
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

Watch the sequence rather than the end state, because each stage identifies a different failure. The pod should go Pending; the autoscaler log should record a scale-up decision naming your node group within a minute or so; a node should join; the device plugin should report an allocatable GPU on it; and only then should the pod schedule.

kubectl get events --sort-by=.lastTimestamp | tail -20
kubectl -n kube-system logs -l app.kubernetes.io/name=cluster-autoscaler --tail=50
kubectl logs gpu-smoke-test

If the pod stays Pending and the autoscaler logs say the node group would not help, the node-template tags are wrong. If a node joins and the pod still will not schedule, the device plugin is missing or has not tolerated the taint. If the node joins and never becomes ready, look at disk and at the driver: an instance family newer than the AMI’s bundled driver fails here, and the message is in the kubelet log rather than in any Kubernetes event.