Choose AI infrastructure by workload type first, because training, inference, and AI agents need different things: training needs large, tightly connected GPU clusters for a limited time, inference needs GPUs close to users with autoscaling and high availability, and agents need isolated sandboxes, secure access to tools and data, and audit trails. Most growing companies spend most of their effort on inference and agents, not training, so the decisions that matter most are where models are served, how capacity is sourced, and how agent actions are contained and recorded.
Disclosure: Control Plane publishes this guide. Other providers are described from their own pages, linked inline and checked in October 2026.
TL;DR
- Training is a limited-time job that needs many GPUs with fast interconnects. Rent it from a hyperscaler or GPU cloud, or buy dedicated hardware if utilization is steady.
- Inference is a 24/7 service. It needs autoscaling, readiness checks, multi-region placement, and a cost-per-request view.
- Agents execute generated code and call tools. They need sandboxes, least-privilege access, private connectivity to internal data, rate limits, and audit logs.
- Hyperscalers, GPU clouds, and your own hardware are not exclusive. Many teams use two or three, which makes portability a requirement.
- Avoid lock-in to one GPU provider by running your serving layer across providers, so a capacity shortage or outage in one place does not stop traffic.
- Control Plane runs inference services and agents active-active across any cloud, neocloud, or your own GPU servers in one Global Virtual Cloud. The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations.
What AI-ready infrastructure actually means
AI-ready infrastructure is the set of compute, network, storage, orchestration, observability, and security capabilities that lets you train, serve, and operate models and agents in production without rebuilding the stack for each new workload.
- Compute (GPUs and accelerators). GPU memory decides which models fit on one device; the number of GPUs and their interconnect decide how large a model you can split across devices. NVIDIA lists 24 GB for the L4, a 72W low-profile PCIe card, and 80 GB for the H100 SXM: small cards for lighter serving, larger parts for bigger models and training.
- Networking. Training across many GPUs depends on fast links between GPUs and nodes, which is why Lambda and Nebius describe InfiniBand-connected clusters. Inference needs a different kind of networking: global routing to a healthy location, private paths to internal data, and encrypted traffic between services.
- Storage and data access. Model weights are large files that must load quickly onto new replicas, and agents and retrieval pipelines need governed access to business data.
- Orchestration and scaling. Something has to place workloads on GPUs, scale replicas, hold traffic until a replica is ready, and roll out new model versions safely. Kubernetes is the common base, and many teams want a layer above it so they don’t operate clusters themselves.
- Observability. GPU workloads fail in ways CPU services do not, such as memory exhaustion, queue buildup, and slow model loads, so you need metrics, logs, and traces across the request path and the GPU, plus cost per model and per request.
- Security and isolation. Models touch sensitive data, and agents run code no one reviewed, so you need tenant isolation, secrets management, short-lived credentials, mutual TLS between services, and an audit trail.
Training vs inference vs agents
| Dimension | Training | Inference | AI agents |
|---|---|---|---|
| What the workload does | Builds or fine-tunes a model from data | Serves a trained model to users and applications | Plans, calls tools, and runs code to complete tasks |
| Compute pattern | Long batch jobs that run for hours to weeks | Always-on service with variable request traffic | Bursty, many short-lived sessions plus calls to models and tools |
| GPU profile | Many GPUs per job with high-bandwidth interconnects, such as multi-GPU HGX-class servers or rack-scale systems | One to several GPUs per replica, from small cards such as the L4 up to H100-class and newer parts | Often no local GPU when agents call hosted models; GPUs apply if you self-host the model |
| Scaling pattern | Scale out for the run, then release | Scale replicas with request rate or latency, across regions | Scale sandboxes and workers with active sessions |
| Availability needs | Tolerates interruption with checkpointing | High; outages are visible to users | High for the control layer; individual sandboxes are disposable |
| Where it usually runs | Hyperscaler or GPU cloud clusters, or dedicated hardware | Near users, across clouds, GPU clouds, and your own hardware | Next to the data and tools it needs, isolated from everything else |
| Main cost driver | GPU-hours across the run, plus idle time between runs | GPU-hours kept warm for latency targets, divided by requests served | Sandbox time, model calls, and the cost of over-permissioned access |
Google Cloud’s AI Hypercomputer guidance draws a similar split, treating pre-training, fine-tuning, inference or serving, and small or medium models as separate cases: reservations for pre-training and fine-tuning, and reservations, on-demand, or Spot capacity for inference.
Where to run it: hyperscaler, GPU cloud, or your own hardware
Use a hyperscaler when you want AI next to existing cloud services, a GPU cloud when GPU access and cluster performance come first, and your own or colocated hardware when utilization is steady and you want control. The order below is not a ranking.
Hyperscalers: AWS, Google Cloud, Azure
Strengths. GPUs alongside the rest of a cloud: identity, networking, storage, databases, and managed services. AWS describes AI infrastructure for training and inference with Amazon SageMaker AI, Amazon EKS, and EC2 Capacity Blocks for reserving GPU instances in EC2 UltraClusters, and lists P5 instances for training and HPC. Google Cloud lists NVIDIA GPU machine types from L4 up to B200 and GB300 systems, including the A4, A4X, and A4X Max series, and its AI Hypercomputer integrates with Google Kubernetes Engine and Slurm. Azure offers GPU virtual machines within its HPC solution, with storage, Azure Batch, and Foundry Tools alongside.
Trade-offs. GPU access is governed by quotas, reservations, and regional availability, and each provider’s AI services have their own APIs and identity model, so moving a serving stack between hyperscalers takes work unless the layer above them is portable.
Best fit. Teams whose data, identity, and compliance controls already live in one hyperscaler.
GPU clouds and neoclouds: CoreWeave, Lambda, Nebius
Strengths. CoreWeave describes an AI cloud with GPU compute, bare-metal servers, managed Kubernetes, and model training and serving. Lambda lists self-serve GPU instances, 1-Click Clusters, and single-tenant Superclusters on NVIDIA GPUs, and states SOC 2 Type II certification. Nebius describes non-virtualized GPUs with InfiniBand, built-in MLOps tooling, and serverless and managed inference.
Trade-offs. A neocloud covers GPUs and the services around them, not a hyperscaler’s full catalog, so you connect it to your identity, networking, and data; see how to add a neocloud without running a second stack.
Best fit. Teams that need GPU capacity, cluster performance, or newer GPU generations, with a serving layer on top.
Your own or colocated hardware: Dell, HPE, Supermicro
Strengths. Control over GPU configuration, data location, and long-run unit cost at steady utilization. Dell lists PowerEdge XE-series servers for AI training, large-scale inference, and HPC, including configurations with NVIDIA HGX B300. HPE lists ProLiant Compute XD685 and Compute XD690 servers for AI training, tuning, and inference with NVIDIA B200 and B300 options. Supermicro lists air-cooled and liquid-cooled HGX B300 and B200 systems, GB300 NVL72 racks, and rack-level integration.
Trade-offs. Procurement lead time, power and cooling, hardware lifecycle, and an operations team; utilization decides whether it pays off.
Best fit. Teams with data residency or sovereignty requirements, steady high utilization, or custom GPU configurations.
Core infrastructure vs AI-specific infrastructure
| Layer | Examples from provider pages | What it gives you |
|---|---|---|
| Core infrastructure | AWS EC2, Google Compute Engine, Azure Virtual Machines | General compute, networking, and storage that GPUs attach to |
| Managed AI/ML platform | Amazon SageMaker AI, Microsoft Foundry and Azure Machine Learning | Pre-integrated tooling for building, training, and deploying models |
| Clusters created on demand | Lambda 1-Click Clusters, Nebius clusters, EC2 Capacity Blocks | GPU capacity launched or reserved when you need it |
| Pre-built hardware | Dell PowerEdge XE, HPE ProLiant Compute and Compute XD, Supermicro SuperCluster designs | Validated servers and racks for your own or a colocated facility |
| Serving and agent layer | Control Plane | A layer that runs inference services and agents across all of the above |
Inference in production: the part most teams underestimate
Training runs end; inference runs continuously as a service with users, latency targets, and an uptime expectation, and it is often where most GPU spend ends up for companies adding AI features. The inference layer guide covers the full stack, including GPU sizing and serving metrics. The decisions that matter most:
- Autoscale on request rate, latency, or queue depth, not CPU. Queue depth or in-flight requests per replica usually reacts faster than latency.
- Decide what stays warm. Keep user-facing models warm, and scale internal or low-traffic models down when a slower first request is acceptable.
- Gate traffic on readiness, since a started replica may not have loaded the model yet.
- Run in several regions or providers. Any region or provider can fail or run out of capacity, so deploy the same service to multiple locations and route each request to a healthy one; see multi-cloud failover solutions.
- Source GPU capacity across providers as prices and availability change, keeping the serving layer independent of any one; see how to combine GPU capacity across providers.
- Track cost per request: GPU replica cost per hour divided by requests served per hour at your latency target.
Agents in production
Agents differ from inference services because they take actions: they run generated code, call APIs, and read internal data. Four controls matter most:
- Isolated sandboxes for untrusted or generated code, with no default reach into the host, other tenants, or your network. MicroVM-based isolation is a common approach, and sandboxes should be ephemeral so state does not persist between tasks.
- Short-lived credentials instead of stored keys, because long-lived cloud keys leak through code, logs, and images; see workload identity federation.
- Least-privilege access to internal data over private networking to your VPCs and on-premises systems.
- Rate limits and audit trails, so a loop or prompt injection cannot run up cost or damage, and every action is recorded.
Observability and operations for 24/7 model serving
Instrument the request path and the GPU, alert on user-visible symptoms first, and make rollouts reversible:
- Metrics: latency percentiles (including time to first token), throughput, GPU utilization and memory, queue depth, error rate by type, and cost per request.
- Logs and traces: request IDs, model version, input and output sizes, and outcome, with traces across gateway, retrieval, model, and tools. Keep prompts and outputs out of logs unless your data policy allows it.
- Alerting: page on error rate, latency against target, and queue depth; route cost and utilization signals to a ticket queue.
- Rollouts: treat every model version as a deploy, release gradually, gate traffic on readiness, and keep the previous version ready for fast rollback.
- On-call: set SLOs for latency and availability, write runbooks for GPU out-of-memory, capacity shortage, slow model load, and provider outage, and rehearse failover.
Decision checklist for picking AI infrastructure
- Which workloads do you run: training, inference, agents, or a mix? Choose per workload.
- Is training a limited-time run you can rent, or steady demand that justifies owned hardware?
- Where are your users, what are your latency targets, and can one region or provider meet your availability needs?
- Can you source GPU capacity from more than one provider without changing your deployment?
- Do residency, sovereignty, or compliance rules force specific locations or your own hardware?
- Does the platform autoscale on request rate, latency, or custom metrics, and gate traffic on readiness?
- Can agents run in isolated sandboxes with short-lived credentials, private data access, and audit logs?
- Do you get unified logs, metrics, and traces, cost per request, and safe rollouts and rollbacks?
How Control Plane fits
Control Plane is the platform for running unbreakable apps and agents. Train on whichever cluster fits the run (a hyperscaler, a GPU cloud, or your own hardware); Control Plane is where the trained model goes into production, serving inference and running agents across all of that compute in one Global Virtual Cloud (GVC).
- Inference that stays up. A workload deployed to a GVC runs active-active in every location of that GVC, and a global endpoint routes each request to the nearest healthy location, so you get active-active failover without paying for idle standby capacity. The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations. During the October 2025 AWS us-east-1 outage, 20% of Control Plane customers had us-east-1 in their Global Virtual Clouds. Their workloads were active and running in healthy locations within 10 seconds, with no customer downtime, according to Control Plane internal incident telemetry.
- GPU capacity from any provider. Workloads can attach NVIDIA GPUs, and the pricing page lists models from T4 to B200. GPU Ocean lets you pull GPU capacity from any provider, and Managed Kubernetes and Bring Your Own Kubernetes (BYOK) let GPU servers, neocloud capacity, or on-premises clusters join a GVC as locations.
- Containers and VMs where you already run: serverless, standard, stateful, cron, and VM workloads on AWS, GCP, Azure, your own clusters, or on premises.
- Autoscaling and readiness. Workloads autoscale natively on requests per second or latency, and on custom metrics such as queue depth through KEDA, and traffic only reaches replicas that pass readiness checks. Serverless workloads scale to zero natively, and standard and stateful workloads scale to zero through KEDA.
- Agents. Sandboxes for AI agents are ephemeral environments built from templates, with dedicated TLS endpoints, that sleep when idle and spin back up in under one second, and they run in dedicated Kata Containers microVMs. Production agents run as regular workloads, including on your own clusters and on premises. Universal Cloud Identity gives workloads short-lived, least-privilege access to 600+ AWS, GCP, and Azure services, with no long-lived cloud keys in your code, and Cloud Wormhole connects workloads to private VPCs and on-premises networks so agents and models reach internal data without exposing it.
- Operations built in: unified logs, metrics, and traces, secrets management, mutual TLS between workloads, readiness-gated rollouts, automatic TLS certificates, and a tamper-proof audit trail, with no Kubernetes cluster you have to operate. The Control Plane MCP server lets AI assistants deploy and operate workloads.
- Cost and compliance. Customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.
FAQ
How do I select the right infrastructure for new AI workloads?
Classify the workload first: training needs large GPU clusters for a limited time, inference needs GPUs close to users with autoscaling and high availability, and agents need isolated sandboxes, secure access, and audit trails. Then choose where to run it, size capacity with a load test, and confirm you can move between providers.
What is AI-ready infrastructure?
GPU compute, fast networking, storage and data access, orchestration and scaling, observability, and security and isolation, combined so models and agents run in production without rebuilding the stack for each workload.
What are the best AI infrastructure platforms for training, inference, and distributed workloads?
It depends on the workload. For distributed training, evaluate Google’s A4, A4X, and A4X Max series, AWS UltraClusters through EC2 Capacity Blocks, Lambda Superclusters and 1-Click Clusters, CoreWeave and Nebius, or dedicated hardware from Dell, HPE, and Supermicro. For inference and agents, the serving layer matters most, and Control Plane is built for it: one Global Virtual Cloud that serves models and runs agents across hyperscalers, GPU clouds, and your own hardware.
Should I run AI workloads on a hyperscaler, a GPU cloud, or my own hardware?
Use a hyperscaler to stay close to existing cloud services, a GPU cloud when GPU access and cluster performance matter most, and your own hardware when utilization is steady or data location is restricted. Many teams combine them and run the serving layer across all of them on a platform such as Control Plane.
How much GPU capacity do I need for inference?
Estimate weight memory from parameter count and precision (a 70-billion-parameter model at 16-bit needs about 140 GB for weights), add KV cache for your context length and concurrency, then load test to find requests per second per replica at your latency target. Divide peak demand by that number and add headroom plus a spare replica per location.
Which cloud platforms offer secure AI model deployment at the edge?
NVIDIA positions its L4 GPU for servers from the edge to the data center to the cloud, and Dell lists edge-oriented PowerEdge servers such as the XR8720t for network edge use. On Control Plane, a workload runs in every location of its GVC, including your own servers joined as locations, and each request goes to the nearest healthy one.
What are the best options for on-premises hardware with custom GPU configurations?
Dell PowerEdge XE-series, HPE ProLiant Compute and Compute XD, and Supermicro systems all list configurable NVIDIA GPU options, and Dell’s AI Solutions Explorer shows small, medium, large, and extra-large reference configurations. Dell’s AI server guidance recommends PCIe GPUs for inference and mixed workloads and SXM modules with NVLink for large training.
Where can I get AI-ready infrastructure for high performance computing?
Azure HPC, Google Cloud’s accelerator-optimized A-series machine types, AWS P5 instances, Lambda and CoreWeave clusters, and HPC-oriented servers from Dell. Control Plane then serves the resulting models and runs agents on whichever of that compute you choose, across providers in one Global Virtual Cloud.
What is the most resilient way to run inference and AI agents?
Run them active-active in at least two locations, ideally on different providers, behind a global endpoint that routes to a healthy location. On Control Plane, the 99.999% availability SLA applies to workloads running at least two replicas in at least two locations, and those locations can be on different providers.
Next step: run inference and agents that stay up when a region or provider goes down. Talk to the Control Plane team or sign up free.
Originally published on the Control Plane blog.
Top comments (0)