DEV Community

Multigrid
Multigrid

Posted on • Originally published at multigrid.ai

Running a GPU Task on Amazon ECS

#ai #machinelearning #tutorial #llm

Reserving a GPU in an ECS task definition is four lines of JSON. Nearly every failure happens outside those four lines — in the AMI the container instance booted from, in one agent configuration variable, and in a placement decision you did not make explicitly.

The cluster has to be able to say yes

ECS does not install drivers. It schedules against capacity that already reports GPUs, which means the container instance must have been launched from the Amazon ECS GPU-optimized AMI — the variant that ships with the NVIDIA kernel drivers and the NVIDIA container runtime preinstalled. AWS publishes its ID in Systems Manager Parameter Store, so you never hard-code an AMI:

aws ssm get-parameters \
  --names /aws/service/ecs/optimized-ami/amazon-linux-2/gpu/recommended \
  --region us-east-1
Enter fullscreen mode Exit fullscreen mode

The second prerequisite is one line of agent configuration. AWS documents that ECS_ENABLE_GPU_SUPPORT must be set to true in /etc/ecs/ecs.config before the agent will advertise GPUs to the scheduler. Set it in the instance user data alongside the cluster name, because the file is read at agent start and editing it afterwards means restarting the agent:

#!/bin/bash
echo "ECS_CLUSTER=inference" >> /etc/ecs/ecs.config
echo "ECS_ENABLE_GPU_SUPPORT=true" >> /etc/ecs/ecs.config
Enter fullscreen mode Exit fullscreen mode

The instance itself has to be one of the accelerated families ECS supports. As documented by AWS at the time of writing that is the p3, p4d, p5, g3, g4, g5, g6, g6e and g6f families; p2 is supported only on GPU-optimized AMI versions earlier than 20230912, and GPUs are not supported on Windows containers at all. A cluster may hold a mix of GPU and non-GPU instances, which is normal — the scheduler filters.

Instance-family support is the part of this page most likely to age. AWS maintains the current list on the ECS task definitions for GPU workloads page, which also carries the per-family GPU and memory table.

The task definition

The GPU is requested per container, not per task, through resourceRequirements. When a container definition carries one, ECS switches that container’s runtime to the NVIDIA container runtime and pins specific physical devices to it:

{
  "family": "embed-worker",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["EC2"],
  "executionRoleArn": "arn:aws:iam::111122223333:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "server",
      "image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/embed:1.4.0",
      "essential": true,
      "cpu": 3584,
      "memory": 14000,
      "resourceRequirements": [
        { "type": "GPU", "value": "1" }
      ],
      "environment": [
        { "name": "NVIDIA_DRIVER_CAPABILITIES", "value": "utility,compute" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/embed-worker",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Two details in there are easy to skip and expensive to skip. The total GPU value across all containers in a task cannot exceed the GPUs on the instance the task lands on — ask for two on a g5.xlarge and the task is unplaceable rather than slow. And NVIDIA_DRIVER_CAPABILITIES is only set for you if your image is built on an NVIDIA CUDA base image. AWS documents that ECS sets NVIDIA_VISIBLE_DEVICES itself, to the device IDs it assigned, but leaves the other runtime variables to you. A slim Python image with a wheel that bundles CUDA will start cleanly and then report no devices, which reads exactly like a driver problem and is not one. Set it to utility,compute or all.

Placing it on the instance you meant

In a mixed cluster the scheduler will use any instance that satisfies the requirement, and “satisfies” is a low bar: one GPU is one GPU whether it is an A10G or an L4. If the model only fits in 24 GB or the throughput only works on a particular family, say so with a placement constraint rather than hoping:

aws ecs run-task \
  --cluster inference \
  --task-definition embed-worker \
  --placement-constraints \
      type=memberOf,expression="attribute:ecs.instance-type == g5.xlarge"
Enter fullscreen mode Exit fullscreen mode

The same expression syntax works on a service, and it composes — attribute:ecs.instance-type =~ g6.* pins a family rather than a size. This is also the honest way to keep a cheap sidecar off an expensive instance: constrain the GPU service in, rather than trying to constrain everything else out.

Fractional GPUs

Historically the value had to be a whole number, which made a small embedding model an expensive tenant of a whole card. AWS now documents fractional GPU scheduling on the G6f family, where the hardware exposes a partitioned slice of an NVIDIA L4 with its own dedicated memory. The value becomes a decimal:

"resourceRequirements": [
  { "type": "GPU", "value": "0.25" }
]
Enter fullscreen mode Exit fullscreen mode

AWS documents three valid fractions and the instance sizes that back them: 0.125 (1/8 of a GPU, 3 GB) on g6f.large and g6f.xlarge, 0.25 (1/4, 6 GB) on g6f.2xlarge, and 0.5 (1/2, 12 GB) on g6f.4xlarge and gr6f.4xlarge. Three constraints come with it. Only one container per task may request a fractional value, and if one does, no other container in that task definition may request a GPU at all. Integer values are still treated as whole-GPU requests and will only place on instances with a full card free. And Fargate and ECS Anywhere do not support fractional scheduling — this is EC2 and ECS Managed Instances only.

There is a separate, blunter way to share: remove resourceRequirements entirely, and set the NVIDIA runtime as the Docker default on the instance with --default-runtime nvidia in the daemon options. Every container then sees the GPUs and ECS reserves nothing. That works, and it means the scheduler has no idea how loaded the card is, so two tasks that each want 20 GB will happily co-locate and one will die on an out-of-memory error. Use it when you control what runs; do not use it under an autoscaler.

When the task never leaves PROVISIONING

The characteristic failure is a service that reports (service X) was unable to place a task because no container instance met all of its requirements, with the cluster visibly holding GPU instances. Work through it in this order:

  1. Describe a container instance and look at registeredResources. If there is no resource named GPU, the agent is not advertising any — the AMI or ECS_ENABLE_GPU_SUPPORT is wrong, and no amount of task definition editing will fix it.
  2. If GPU is registered, compare remainingResources to registeredResources. A stopped-but-not-reaped task, or a task that asked for ALL, will have consumed the devices.
  3. Check CPU and memory too. A GPU instance is still an EC2 instance, and a task asking for more memory than the instance has left is unplaceable for an entirely non-GPU reason that the error string does not distinguish.
  4. Check the placement constraint. An expression naming an instance type you no longer run makes the cluster look full when it is empty.

Once it places, the scaling question is separate and worth treating separately — see autoscaling an ECS service by queue depth for the signal to scale on, and GPU autoscaling for why utilisation is a poor one.

Related

Top comments (0)