DEV Community

Muskan Bandta
Muskan Bandta

Posted on

GPU Rightsizing Without Breaking Production: G5, G6, P4, P5 and the CUDA Check Nobody Mentions

CPU rightsizing is a solved, well-documented practice. GPU rightsizing is where the real money is now, and almost nobody writes about it, because GPU instances are expensive enough that people are scared to touch them and unsure how. Given how much a GPU box costs per hour, an over-provisioned one is the single most expensive rightsizing mistake in your account. Here is how to rightsize AWS GPU instances without breaking the workload, including the compatibility check that quietly bites people.

Know what each GPU family is for

Rightsizing starts with using the right family, not just the right size. On AWS:

  • G5 / G6 (NVIDIA A10G / L4): inference, graphics, smaller training. The workhorses for serving models and lighter ML. Cheaper per hour.
  • P4 / P5 (A100 / H100): large-scale training and heavy inference. The expensive tier, built for jobs that genuinely need the horsepower and interconnect.

The most common GPU waste is running a training-class P-family instance for an inference workload that a G-family instance would serve fine at a fraction of the cost. Wrong family is a bigger error than wrong size.

Rightsize on the binding resource, and it is usually not CPU

GPU workloads have several resources that can be the bottleneck, and CPU utilization, the thing you would check for a normal instance, is often the least relevant:

  • GPU utilization: is the GPU actually busy, or idle between requests? (CloudWatch does not report this by default; you need the CloudWatch agent with GPU metrics or nvidia-smi telemetry.)
  • GPU memory: many inference workloads are GPU-memory-bound, not compute-bound. A model that fits in less VRAM can move to a smaller GPU.
  • Host CPU and RAM: sometimes the GPU is fine but the instance is over-sized on host resources.

The rightsizing signal is a GPU sitting at low utilization or using a fraction of its VRAM over a sustained window (a 90-day-style baseline, same idea as CPU rightsizing). That is your candidate to move down a size or across to a cheaper family.

The CUDA check nobody mentions

Here is the gotcha that turns a clean rightsizing into an outage. Different GPU families use different NVIDIA architectures, which means different CUDA driver and library requirements. Move a workload from, say, an A100-based P4 to an L4-based G6, and the driver version, CUDA toolkit, and framework build that worked on one may not match the other.

If you rightsize the instance but do not verify CUDA compatibility, the box comes up and the workload fails to initialize the GPU, or silently falls back to CPU and runs at a crawl. Before any GPU family change:

  1. Confirm the target family's supported NVIDIA driver and CUDA version.
  2. Confirm your framework build (the CUDA-compiled wheels for your ML library) supports that target.
  3. Test on one instance before you roll the fleet.

This step is the difference between "rightsized and saving money" and "rightsized and paged at 2am because inference is down." It is also why teams avoid GPU rightsizing entirely, they got burned once and never went back. Do the compatibility check and it is safe.

The other GPU cost lever: do not run them idle

Rightsizing the instance matters, but the bigger GPU waste is often time, not size. GPU utilization on "always-on" fleets is routinely far below what people assume, and eval, dev, and training-experiment pools have no reason to run overnight or on weekends.

  • Schedule non-production GPU pools to scale down off-hours. A 50%-cheaper right-sized GPU still burns money at 3am doing nothing. (We schedule GPU node groups the same way as any other non-prod resource, which is part of what ZopNight does, but a scheduled scale-down gets you the crude version.)
  • Right-size and schedule together. They compound: the correct family and size, running only when needed, is dramatically cheaper than an oversized one running 24/7.

The take

GPU rightsizing is the highest-value, least-covered cost work in most AI-touched accounts, because the per-hour price makes every mistake expensive. Use the right family (G5/G6 for inference, P4/P5 for heavy training), rightsize on GPU utilization and VRAM rather than CPU, and always run the CUDA compatibility check before a family change so you do not trade savings for an outage. Then make sure the thing is not idling overnight, which is often the bigger win.

Have you rightsized GPU instances, and did the driver/CUDA compatibility catch you the way it caught a lot of us? That check is the step every GPU-cost article skips and every GPU incident includes.

Top comments (0)