Plan AI capacity per workload type (training, fine-tuning, batch inference, online inference, and development), and size online inference in replicas from measured throughput at your latency targets. Control cost by matching reserved, committed, on-demand, and spot capacity to each workload, keeping training next to its data, and tracking unit cost per team and model across every cloud.
GPU capacity planned as one shared pool ends up sized for the busiest workload and idle for the rest. When that capacity spans two or three clouds and several regions, new cost lines appear that single-cloud dashboards do not show. This guide gives a capacity plan by workload type, a worked sizing formula for inference, and the cost lines that grow when AI runs across clouds and regions.
Disclosure: Control Plane publishes this guide.
TL;DR
- Plan per workload type. Training, fine-tuning, batch inference, online inference, and development each need different placement, purchasing, and scaling rules.
- Size inference in replicas. Load-test your model, quantization, and serving runtime with your real prompt and response lengths, then multiply replicas by GPUs per replica.
- Match capacity to the workload. Reserved capacity or capacity blocks for large training, a committed baseline plus on-demand for online inference, and spot for fine-tuning and batch jobs that checkpoint.
- Keep training next to its data. Moving large datasets across clouds or regions adds egress cost and hours of transfer time.
- Cache model weights in every inference region. Pulling weights from a remote registry at startup adds cold-start time and egress charges.
- Track unit cost. Measure cost per 1,000 input and output tokens, cost per request, and GPU efficiency by team and model.
- On Control Plane, inference services run in Control Plane’s Global Virtual Cloud (GVC), a named set of locations that can span regions and clouds, with autoscaling and Day 2 operations handled by the platform on one cloud or several.
How do I manage costs and capacity planning for AI infrastructure running across multiple clouds and regions?
Plan each AI workload class on its own, because training, fine-tuning, batch inference, online inference, and development differ in latency needs, interruption tolerance, and unit economics.
| Workload | Where to place it | Capacity to buy | What drives scaling | Unit cost metric |
|---|---|---|---|---|
| Large model training | Single region and provider with high-bandwidth interconnect, near the training data | Reserved capacity or capacity blocks booked ahead of the run | Job schedule and checkpoint frequency | Cost per training run or per GPU-hour used |
| Fine-tuning | Any region with GPU availability | On-demand or spot with checkpointing | Job queue depth | Cost per fine-tuning run |
| Batch inference and embeddings | Cheapest region or cloud with available capacity, close to the source data | Spot or on-demand | Queue depth | Cost per 1,000 items or tokens |
| Online inference | Multiple regions near end users | Committed baseline plus on-demand burst | Requests or tokens per second and latency targets | Cost per 1,000 tokens or per request |
| Development and experiments | One region | On-demand with idle shutdown | Number of active developers and their schedules | Cost per developer per month |
How many GPUs do I need for inference?
Size each inference region in replicas from peak demand and measured throughput, then convert replicas to GPUs.
Replicas per region = ceil(peak output tokens per second in that region × (1 + headroom) / measured output tokens per second per replica at your latency targets). GPUs per region = replicas × GPUs per replica, plus a failure reserve.
The formula has five inputs.
- Peak output tokens per second comes from traffic forecasts. Use the peak of your busiest hour, because a daily average undersizes the fleet. For a new product, estimate from beta traffic or comparable endpoints.
- Measured output tokens per second per replica must come from a load test with your model, quantization, serving runtime (vLLM, TensorRT-LLM, NVIDIA Dynamo-Triton, or similar), and your production mix of prompt and response lengths. Long prompts load the prefill phase and raise time to first token. Output tokens are generated in the decode phase, which is limited by memory bandwidth. The KV cache fills GPU memory and caps how many sequences a replica can serve at once. Find the highest concurrency that still meets your time-to-first-token and inter-token latency targets, and read throughput at that concurrency.
- GPUs per replica depends on model size. A small model fits on one GPU. A 70-billion-parameter model at 16-bit precision needs about 140 GB for weights alone, so on 80 GB GPUs such as the H100 it is split across two or more GPUs with tensor parallelism.
- Headroom of 20 to 30 percent is a common planning range. It covers forecast error, deploy surges, and the minutes new replicas need to load weights before they pass readiness checks.
- Failure reserve depends on the failure you plan for. One spare replica covers a node or GPU failure. Surviving a zone loss requires the remaining zones to carry peak, so 5 replicas across 3 zones becomes 3 per zone, or 9 in total.
Worked example, with illustrative numbers for an 8-billion-parameter model on one GPU per replica, measured at 1,500 output tokens per second per replica within its latency targets:
- Region A: peak 6,000 output tokens per second. 6,000 × 1.25 / 1,500 = 5 replicas. Add 1 spare replica for node failure = 6 GPUs.
- Region B: peak 3,000 output tokens per second. 3,000 × 1.25 / 1,500 = 2.5, rounded up to 3 replicas. Add 1 spare = 4 GPUs.
- Failover: if Region A fails while both regions are at peak, Region B must serve 9,000 output tokens per second. 9,000 × 1.25 / 1,500 = 7.5, rounded up to 8 replicas, plus 1 spare = 9 GPUs. Regions in different time zones often peak at different hours, which lowers this number.
There are three ways to cover failover: keep the extra GPUs running, autoscale on demand, or accept slower responses during the failover. Autoscaling depends on the cloud having GPUs free at that moment, and new replicas take minutes to load weights, so it does not cover the first minutes of a failover. Size surviving regions to your SLA and budget.
Which capacity should you commit, rent on demand, or run on spot?
Match the purchasing model to how predictable and how interruptible each workload is.
Committed use discounts and reservations
AWS offers Savings Plans and Reserved Instances. Google Cloud offers committed use discounts in resource-based and flexible (spend-based) forms. Azure offers Reserved VM Instances and Azure savings plan for compute. Each trades a one- or three-year commitment for a lower hourly rate, and commitments generally cannot be reduced once bought. Savings Plans and spend-based commitments lower the price without guaranteeing GPU availability, so pair them with capacity reservations where availability matters. A three-year commitment on a GPU family that a newer generation may replace within that term carries obsolescence risk, so one-year terms are often safer for GPUs.
Capacity reservations for training
GPU capacity for large training runs is hard to get on demand. AWS EC2 Capacity Blocks for ML reserve GPU or Trainium instances for 1 to 182 days, starting up to eight weeks ahead, with no cancellations. Google Cloud’s Dynamic Workload Scheduler offers Flex-start for jobs that can wait for capacity and Calendar mode, which reserves accelerator capacity, including GPUs, for 1 to 90 days. Azure on-demand capacity reservations hold VM capacity in a region or zone with no term commitment, billed at pay-as-you-go rates, but they cover only some GPU series and not the ND-series or NCads H100 v5 machines used for large training, which teams usually arrange with Microsoft directly. Booking ahead secures GPUs for the run date.
Spot GPUs
Spot Instances on AWS (two-minute interruption notice), Spot VMs on Google Cloud (the successor to preemptible VMs, with about 30 seconds of notice), and Spot VMs on Azure (about 30 seconds of notice) can be reclaimed whenever the provider needs the capacity. They suit fine-tuning, small single-node training, and batch inference that save checkpoints often enough that the work lost to a preemption is acceptable. Large multi-node training on spot stops the whole job on any single preemption and rolls back to the last checkpoint, so book reserved capacity or capacity blocks for those runs. Keep the online inference baseline off spot. Burst replicas on spot need to drain gracefully within the reclaim notice.
GPU neoclouds
GPU-focused cloud providers can offer training and batch capacity at competitive prices and sometimes earlier access to new GPU models. Evaluate them when hyperscaler capacity is short. The neocloud vs. hyperscaler evaluation guide walks through the comparison.
Review commitment utilization
Check commitment utilization monthly. Buy new commitments in smaller tranches with staggered end dates, so a model change or a new GPU generation does not leave a large commitment idle. The build vs. rent GPUs break-even model helps decide when owned hardware makes sense.
Which costs grow when AI runs across clouds and regions?
Multi-cloud, multi-region AI adds cost lines that single-cloud deployments do not have.
| Cost line | Why it grows with more clouds or regions | How to control it |
|---|---|---|
| Idle GPU capacity | Reserved or warm GPUs in each region wait for traffic that may be concentrated elsewhere at a given hour | Size committed capacity per region with the inference sizing formula; autoscale on-demand capacity for burst |
| Data egress for training datasets | Moving training data to another cloud or region incurs per-GB egress fees on datasets that can reach tens of terabytes | Keep training in the same region and cloud as the data; if you must move data, move it once and keep a copy |
| Model weights and container images in every region | Each region needs a local copy of model weights (tens to hundreds of gigabytes per model) and the serving image | Cache weights in an object store or registry in every inference region; ship LoRA adapters instead of full weights where possible |
| Cross-region inference traffic | Cross-region retrieval for RAG and large payloads such as images and audio routed to a non-local region incur egress fees | Route users to the nearest region with a healthy deployment; keep retrieval data in the serving region |
| Duplicated vector stores and caches | Vector stores and response caches are replicated in every serving region, and prefix caches warm separately in each region | Accept duplication where latency requires it; compress and deduplicate where possible |
| Checkpoints and dataset copies | Checkpoints and training data stored in more than one cloud or region multiply storage cost | Set retention policies for checkpoints; keep one primary copy of each dataset |
| Commitments split across providers | Smaller committed spend per provider means less leverage for negotiated enterprise discounts and more commitments to track | Concentrate commitments for steady inference on one or two providers; use spot or on-demand elsewhere |
| Observability and logging per cloud | Each cloud’s monitoring and logging bill separately, and shipping telemetry to one platform adds egress | Centralize on one platform; filter and sample at the source to limit cross-cloud telemetry egress |
| Support plans and managed AI services | Support plans and managed endpoints, model hosting, or vector databases each bill per provider | Compare self-managed serving on GPUs with managed endpoints at your scale; review support tiers per provider |
The total cost of multi-cloud AI is these lines plus the engineering time to run across providers. Tracking only GPU-hours misses egress, duplication, and tooling.
How do you track AI cost across clouds?
Track AI cost with consistent tags, a common billing schema, and unit metrics tied to tokens and requests.
Tagging and allocation
Tag every GPU instance, storage volume, and container with team, model name, environment, workload type, and cloud provider. Shared GPU nodes run pods from several teams, so split their cost by namespace or label with OpenCost or the provider’s Kubernetes cost allocation, such as AWS split cost allocation data or GKE cost allocation. Allocate costs that cannot be tagged, such as egress and support, by a documented rule.
A common billing schema
The FinOps Open Cost and Usage Specification (FOCUS), published by the FinOps Foundation, defines a standard schema for billing data from AWS, Google Cloud, Azure, Oracle Cloud, and other providers. Normalizing bills to FOCUS lets you compare GPU-hour cost across clouds and total spending by team or model.
Unit cost metrics
Track cost per 1,000 tokens for generative models and cost per request for classification and embedding endpoints. Count input and output tokens separately, or weight them, because output tokens take more GPU time than input tokens. For GPU efficiency, use streaming multiprocessor (SM) activity and memory bandwidth utilization from NVIDIA DCGM profiling metrics, and model FLOPS utilization (MFU) for training. The nvidia-smi utilization figure reports the share of time any kernel was running and overstates efficiency.
Budgets and anomaly alerts
Set budgets per team and per model from forecast traffic and unit cost. Configure anomaly alerts that fire when spending departs from the forecast by a set threshold. A runaway fine-tuning job or a misconfigured autoscaler can burn through a GPU budget in hours.
Showback and chargeback
Publish cost reports to the teams that use AI infrastructure. Showback shows each team its costs. Chargeback bills each team for its usage. The cloud cost optimization tools roundup covers platforms that support both across clouds.
How Control Plane fits
Control Plane is the unbreakable platform for running production workloads. It runs serverless, standard, stateful, and cron workloads natively on AWS, GCP, Azure, and Oracle Cloud Infrastructure (OCI), VM workloads on locations joined through Control Plane’s Managed Kubernetes, and workloads on Kubernetes clusters you already run or your own machines, including on premises, active-active across regions and clouds.
Model-serving workloads deploy to a GVC, and Control Plane runs each workload in every location of that GVC. The workload’s canonical endpoint uses DNS to send users to the nearest location where the workload passes its readiness probes, and routing tiers and latency offsets shape where traffic goes. Workloads can attach NVIDIA GPUs, and the pricing page lists models from T4 to B200, with “Per-second billing on NVIDIA accelerators. Subject to availability.” You pay for the CPU and memory a workload reserves, plus GPUs, egress, storage, and the other items on the pricing page.
Workloads autoscale on requests per second, CPU, memory, latency, concurrency (serverless only), multi-metric rules, or KEDA. Capacity AI, Control Plane’s automatic CPU and memory right-sizing, does not apply to GPU workloads, so GPU sizing follows the inference sizing formula in this guide.
Control Plane’s GPU Ocean lets you pull GPU capacity from any provider, which you can do today with Control Plane’s Managed Kubernetes by joining GPU servers, from neoclouds or your own hardware, as locations. The guide on running AI inference across multiple GPU clouds explains the architecture.
Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security. The same Day 2 operations apply when a model runs in a single region on one cloud. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant. Teams that want to run AI and ML containers on GPUs without vendor lock-in can keep the same serving container on one cloud, several clouds, or their own GPU servers.
FAQ
Short answers on AI capacity planning, GPU sizing, and multi-cloud AI cost.
How do I manage costs and capacity planning for AI infrastructure running across multiple clouds and regions?
Treat training, fine-tuning, batch inference, online inference, and development as separate planning units. Book training capacity ahead in the region that holds the data, run online inference near users on a committed baseline with on-demand burst, and send interruptible work to spot. Load-test each model to get replicas per region, then add headroom and a failure reserve. Tag every resource, normalize bills across providers, and review cost per 1,000 tokens and commitment utilization every month.
How many GPUs do I need for LLM inference?
Divide peak output tokens per second, plus 20 to 30 percent headroom, by the output tokens per second one replica sustains within your latency targets, then multiply by the GPUs each replica needs. Measure that throughput with your own prompt and response lengths. Add one spare replica for node failure, and if the service must survive a zone loss, size each zone so the remaining zones can carry peak.
Should AI training and inference run in the same cloud?
Not necessarily. Training runs where the data and high-bandwidth interconnect are, and inference runs near users, which is often a different region and sometimes a different cloud. Plan them separately, with capacity blocks or reservations for training runs and a committed baseline for steady inference.
Is it cheaper to run AI on spot GPUs?
Yes, for interruptible work. Spot GPUs cost less and suit fine-tuning, small training jobs, and batch inference that checkpoint often. Large multi-node training and the online inference baseline need reserved or on-demand capacity, because a single reclaim stops a synchronized job or drops live requests.
How do I reduce egress costs for AI workloads?
Run training in the same region and cloud as its data. Store model weights and serving images in each inference region so new replicas do not pull them across clouds. Keep retrieval data in the serving region, ship LoRA adapters instead of full model copies where possible, and sample telemetry before it leaves each cloud.
What is cost per token and how do I measure it?
Cost per token is the infrastructure cost of processing or generating a token, usually stated per 1,000 tokens. Divide the total GPU, egress, storage, and platform cost of an inference service over a period by the tokens it processed in that period. Report input and output tokens separately or weight output tokens more heavily, because each output token takes more GPU time. Compare it across models, quantization levels, and hardware.
What is FinOps for AI?
FinOps for AI applies cloud financial management to GPU and model costs, with unit metrics such as cost per token, GPU efficiency tracking, and purchasing matched to each workload. The FinOps Foundation publishes FinOps for AI guidance and working-group papers.
How does Control Plane help with AI infrastructure across clouds?
Control Plane deploys model-serving workloads the same way to its managed regions on AWS, GCP, Azure, and Oracle Cloud and to GPU servers you join as locations through Control Plane’s Managed Kubernetes. Workloads scale on request, latency, or other signals, and users reach the nearest location where the service is ready, with Day 2 operations handled by the platform.
Can I mix hyperscaler regions and neocloud GPUs on Control Plane?
Yes. One Control Plane organization can run several GVCs, such as one with Control Plane-managed regions on AWS, GCP, Azure, and Oracle Cloud and one with GPU servers from neoclouds or your own hardware, joined as locations through Control Plane’s Managed Kubernetes. Each workload runs in every location of its GVC, so you choose where a GPU model runs by choosing the GVC you deploy it to.
Run AI inference across regions and clouds with GPUs where you need them. Talk to the Control Plane team or sign up free.
Originally published on the Control Plane blog.
Top comments (0)