Amazon has added GPU partitioning to SageMaker Inference, according to a post on the AWS Machine Learning Blog. It uses NVIDIA Multi-Instance GPU (MIG) to cut one physical GPU into partitions. Each partition gets a fixed share of compute and its own block of memory, and the isolation is enforced in hardware.
What changed
Until now the smallest unit you could request was one whole accelerator, so an instance with eight A100s held at most eight models. Teams that wanted real partitioning had to run their own Kubernetes stack. SageMaker now handles GPU configuration, placement and scaling once you choose a partition profile.
The feature adds two optional API fields and no new operations or IAM actions. A layout on the production variant in the endpoint configuration declares how every GPU is divided. Each inference component then names the partition type it runs on. Invocation is unchanged, and an endpoint that omits both fields behaves as before. AWS measured about 17 extra seconds of provisioning time for the partitioning step.
Where it applies
Partitioning is available on the P4d, P5, G7 and G7e families in seven Regions: US East (N. Virginia and Ohio), US West (Oregon), Singapore, Tokyo, Frankfurt and Ireland. A GPU yields at most seven partitions. G7 splits into two and G7e into four. The endpoint must use inference components. Single-model endpoints, serverless endpoints, heterogeneous instance pools and asynchronous invocation are out of scope.
The limits
- A component's partition type is fixed for its life, and changing the carve of an endpoint means a new endpoint and a traffic migration.
- A component that requests a profile missing from the layout is rejected, with no fallback to a whole GPU. Whole devices are still required by any model that calls for more than one GPU.
- You are billed for the full instance, so the saving comes from higher utilization and not from a lower hourly price.
- GPU isolation is not a full tenancy boundary, because models on one instance still share host CPU, memory, storage and networking.
- On P4d, one small profile needs a newer NVIDIA driver than the default image ships, and AWS says the failure does not name the profile.
What AWS measured
AWS compared one three-compute-unit, 40 GB partition of an H100 against a whole A100-40GB, both serving Llama-3.1-8B-Instruct on one replica each. The partition produced 53 to 89 percent more output tokens per second at the load where its cost benefit peaked, with lower time to first token in every case. It was 18 to 34 percent cheaper per million output tokens above a crossover concurrency. That crossover was 64 for 100-token prompts and 4 for 8,000-token prompts. Below it, the whole A100 was cheaper. At the top of the load range on long prompts, the A100 failed a third to a half of requests, while the partition failed none. This is one vendor test on one model, so treat it as a method to repeat on your own traffic.
AWS also notes that a partition must hold the weights, the KV cache and one to two gigabytes of runtime overhead. An 8B model at FP16 needs roughly 16 GB for weights alone.
Top comments (0)