How to Run AI Inference Across Multiple GPU Clouds: Failover, Burst, and Portability in 2026

Disclosure: Control Plane publishes this guide.

TL;DR

To keep inference online and portable, package the model server as a container, run it in at least two GPU locations on different providers, and route requests to the nearest healthy one. Control Plane runs one inference workload across cloud regions and your own GPU servers, and routes each request to the nearest healthy location.

• Why single-provider inference breaks: GPU capacity, quotas, regional outages, price changes, and model API outages all sit behind one vendor relationship.
• The three patterns: active-active across GPU providers, a primary provider with a failover provider, and your own GPUs as the base with cloud burst.
• How to choose providers: match each provider to your latency target, GPU memory needs, data residency rules, and exit cost. Pair a hyperscaler with a neocloud instead of ranking them.
• How Control Plane fits: one workload deploys to Control Plane’s Global Virtual Cloud (GVC), a set of locations that can be cloud regions or GPU servers. A global endpoint routes each request to the nearest healthy location.
• First steps: containerize the model server, pin versions, pre-stage weights in two locations, health check with a real inference request, and drill the failover.

Why does single-provider inference break?

Inference is a production service, but most teams run it behind one GPU cloud or one model API. Four things go wrong with that setup.

Capacity shortages and quota limits. Specific GPU models run short, and accounts carry quotas per GPU type and region. A launch or traffic spike can hit a quota while the model server is healthy.

Regional and provider outages. Major cloud providers have multi-hour outages. The Control Plane cloud outage tracker lists 62 major outages at AWS, Microsoft Azure, Google Cloud, and Cloudflare since 2020, with a median duration of 6h 08m. By provider, it shows AWS with 14 incidents (median 7h 34m), Azure with 20 (median 8h 17m), and Google Cloud with 12 (median 6h 08m). The tracker notes these counts reflect what each provider documents publicly and are not a reliability ranking. Its longest completed outage is Azure East US 2 in January 2025, at 50h 13m. As of its October 21, 2026 update, it also lists two AWS Middle East regions as disrupted since March 2026. The tracker covers cloud outages in general, not GPU services specifically. An inference service pinned to one region still inherits that region’s worst day.

Price changes. GPU prices and terms change. If your model server only runs on one provider’s services, you pay what that provider asks.

Model-provider API outages. If your product calls a hosted model API, its outage is your outage, whatever your own infrastructure looks like.

What are common challenges in AI model inference and how can I overcome them?

Challenge	Why it happens	Fix
GPU availability	Specific GPU models are scarce, and quotas cap what one account can use.	Qualify at least two providers for the same model server. Request quota in advance.
Cold starts and model load time	Weights are large and must be copied to a new node and loaded into GPU memory before the first request.	Pre-stage weights near every location. Keep a minimum number of warm replicas.
Latency for global users	Requests travel to one region, then back.	Run in several regions and route each request to the nearest healthy location.
Cost of idle GPUs	Standby capacity sits unused until a failure or spike.	Serve traffic from every location instead of holding idle standby. Own steady capacity and rent peaks.
Model and driver version drift	Locations are built at different times with different CUDA, driver, or server versions.	Pin versions in the container image. Deploy one image everywhere.
Observability	Each provider has its own metrics, logs, and dashboards.	Export metrics from the model server in one format. Track latency, queue depth, and error rate per location.
Data residency	Prompts and outputs may contain regulated data that must stay in a region.	Choose locations by jurisdiction. Route regulated traffic only to approved locations.

What are the three multi-provider patterns for AI inference?

All three patterns share one requirement: the same container image and the same model weights must be available in every location that might serve traffic.

Active-active across GPU providers

How it works. The model server runs in two or more locations on different providers. A global endpoint or load balancer sends each request to a healthy location, usually the nearest.

When to use it. Use it for user-facing inference where latency matters and an outage cannot wait for a manual switch.

Idle capacity cost. Low, because every location serves traffic. You still need enough total capacity that the remaining locations can absorb a lost location’s share. See the cost comparison in active-active vs active-passive.

What breaks during failover. Surviving locations take extra load and may queue requests or scale up. Scale-up needs warm-up time, so set a minimum replica count that covers a lost location. Version drift between locations can produce inconsistent answers.

Primary plus failover provider

How it works. One provider serves all traffic. A second provider holds the same image and weights and takes over when health checks fail.

When to use it. Use it for internal tools, batch inference, or products where a short brownout is acceptable and a second always-on fleet is hard to justify.

Idle capacity cost. Depends on how warm the failover is. A cold standby costs little and starts slowly. A warm standby starts fast and costs close to a second fleet.

What breaks during failover. Cold start is the main risk: weights must already be present and loaded. DNS-based switching adds propagation delay. Standby configurations also drift, because nobody tests them daily.

Own GPUs as the base with cloud burst

How it works. Steady traffic runs on GPU servers you own or reserve. Peaks go to cloud GPUs running the same image.

When to use it. Use it when baseline demand is predictable and high enough that owned hardware is cheaper than renting, and when peaks are brief.

Idle capacity cost. Owned servers carry fixed cost whether busy or not. Cloud burst capacity costs money only while it runs, but only if you can get the GPUs when the spike arrives.

What breaks during failover. Weights must be in the cloud location before the spike, not copied during it. The network path between your data center and the cloud adds latency and transfer cost. If the two environments run different driver or server versions, outputs can differ between them.

How do I fail over AI inference between GPU providers when one has an outage?

Follow these six steps.

1. Containerize the model server. Use a serving engine that ships as a container. vLLM describes itself as a fast and easy-to-use library for LLM inference and serving, and its docs cover deployment with Docker and Kubernetes. NVIDIA Dynamo-Triton, formerly Triton Inference Server, describes itself as open source inference serving software that supports models from multiple frameworks across cloud, data center, and edge. Hugging Face Text Generation Inference serves LLMs with Prometheus metrics and OpenTelemetry tracing, and its docs now state it is in maintenance mode and recommend vLLM, SGLang, llama.cpp, or MLX going forward.
2. Pin model and driver versions. Fix the model revision, serving engine version, CUDA version, and driver version in the image and in your infrastructure code. Upgrade all locations together.
3. Pre-stage weights in each location. Replicate weights to object storage or local disk near every GPU location. Verify checksums. A failover that has to download 100 GB of weights is a slow failover.
4. Use health checks that run a real inference request. A process that answers on a port can still fail to generate. Send a short prompt, check for a valid response within a time limit, and fail the location if it does not.
5. Route with a global endpoint or load balancer you control. The router must sit outside any one provider’s failure domain, and it must remove unhealthy locations automatically.
6. Drill the failover. Take a location out of rotation on a schedule. Measure how long recovery takes and how much traffic errors out. Fix what the drill finds.

How do I keep an AI product online when my model provider or GPU cloud has an outage?

For a GPU cloud outage, use the steps above. For a hosted model API outage, add a fallback chain:

• Primary: the hosted model API you use today.
• Second choice: a second API provider, with prompts tested against both.
• Last resort: a self-hosted open-weight model running in two GPU locations.

Test prompts and output parsing against every model in the chain, because models differ in format and tone. Decide in advance whether a lower-quality answer is better than an error. Set timeouts short enough that the fallback triggers before users give up. Agent workloads need the same discipline. See where to run agents in production.

How do I run LLM inference on my own GPUs and burst to the cloud when demand spikes?

Keep steady traffic on owned or reserved GPU servers. Send peaks to cloud GPUs. Four points decide whether this works.

Size the base to the floor, not the peak. Owned GPUs earn their cost when they are busy. Set the base to the load you carry most hours, and let the cloud cover the rest.

Keep both environments identical. Use the same container image, the same pinned versions, and the same configuration from one infrastructure-as-code repository. Differences between the owned and cloud environments show up as inconsistent answers during a spike.

Pre-stage weights in the cloud. Copy weights to cloud storage near the burst location before you need it. Keep a small number of warm replicas so the first burst requests do not wait for model load.

Account for latency and data transfer. Requests that cross between your data center and a cloud region add round-trip time. Moving data or weights between them can add transfer charges. Place the burst location in a region close to your users and your data, and keep data that must stay on premises out of the burst path.

How do I move between GPU providers as prices and availability change?

Use this portability checklist:

• Containers, not provider-specific services. Package the model server in a container image that any Kubernetes-compatible environment can run.
• Infrastructure as code. Define locations, networking, and workloads in code, such as Terraform or Pulumi, so a new provider is a new target.
• Open model formats where you can. Avoid formats that only one provider’s runtime can load. Keep a copy of the original weights.
• Object storage replication for weights. Replicate weights across providers on a schedule, so any location can start serving without a long copy.
• A routing layer you control. If your provider owns the router, your provider controls your exit. Run routing outside any one GPU cloud.
• Identity without long-lived keys. Provider-specific keys embedded in code make moves harder. Use short-lived credentials instead.

Test the checklist by moving a small share of non-critical traffic to a second provider each quarter.

How do the main GPU clouds compare?

CoreWeave vs Lambda vs AWS for GPU workloads

All three offer NVIDIA GPUs, and they differ in type and positioning. AWS is a hyperscaler that sells GPU instances next to a broad set of other services. CoreWeave and Lambda are neoclouds that focus on AI workloads. The table below states only what each provider’s own pages say, as of October 30, 2026. Check each page for current prices, GPU availability, and regions, which change often.

Best GPU cloud providers for inference

The right provider depends on your workload, so this table does not rank them.

Provider	Type	Best for (per its own pages)	Notes
AWS	Hyperscaler	Deep learning and HPC applications	The EC2 P5 page describes instances powered by NVIDIA H100 GPUs, and P5e and P5en instances powered by NVIDIA H200 GPUs.
Google Cloud	Hyperscaler	Machine learning, data processing, and graphics-intensive workloads	The Compute Engine docs list NVIDIA GPU models and the accelerator-optimized machine series they attach to.
Microsoft Azure	Hyperscaler	Compute-intensive, graphics-intensive, and visualization workloads (GPU sizes)	The VM sizes overview groups sizes into families, including the GPU accelerated NC, ND, NG, and NV families.
CoreWeave	Neocloud	Complex AI workloads	Describes itself as the AI cloud purpose-built for training and serving models at scale.
Lambda	Neocloud	Training and inference at scale	Lists NVIDIA Vera Rubin NVL72, GB300 NVL72, HGX B300, HGX B200, H200, and H100 GPUs, offered as Instances, 1-Click Clusters, and Superclusters.
Runpod	Neocloud	Training, inference, and batch workloads	Offers on-demand GPUs and serverless compute, with Pods, Serverless, and Clusters.
Nebius	Neocloud	AI developers	Describes built-in MLOps tooling, serverless inference, and managed inference.

For a fuller roundup, see The Best GPU Cloud Providers for AI Inference and Training in 2026.

The strongest setup uses more than one of these. A hyperscaler gives you broad services and compliance coverage. A neocloud may give you GPU capacity or terms the hyperscaler cannot. Running the same container in both means an outage, a quota limit, or a price change at one provider moves traffic to the other.

How do I choose an AI model inference service for my application?

Work through this checklist in order:

• Latency target. Set a time-to-first-token and total-response target for your users. Measure from where users sit, not from the provider’s region.
• Throughput. Estimate peak requests per second and tokens per request. Plan capacity for the peak, then decide what runs on base capacity and what bursts.
• Model size and GPU memory. The weights plus the KV cache, which grows with batch size and context length, must fit in the memory of one GPU or be split across several GPUs. This decides which GPUs qualify and how many you need.
• Data residency and compliance. List the jurisdictions and certifications your data requires. Remove providers and regions that do not fit.
• Managed API or self-hosted. A managed API is faster to start and ties you to that provider’s model and uptime. Self-hosting gives control over versions, cost, and location, and adds operational work.
• Exit cost. Estimate what it takes to leave: egress charges, weight migration, retraining of prompts, and rework of provider-specific code.

How do I combine GPU capacity from multiple providers into one environment?

On Control Plane, you deploy one inference workload to a Global Virtual Cloud (GVC). A GVC combines any cloud, neocloud, and bare metal into one environment. Its locations can be AWS, GCP, or Azure regions, and GPU servers from other providers or your own data center. You join GPU servers as managed locations with Control Plane Managed Kubernetes (MK8s), or connect clusters you already run with Control Plane’s Bring Your Own Kubernetes (BYOK).

The workload runs in every location of the GVC. A global endpoint routes each request to the nearest healthy location, so you get failover without paying for idle standby capacity. A workload’s GPU request applies in every location of its GVC, so choose locations that offer the same GPU model.

On Control Plane, a readiness probe (HTTP, gRPC, TCP, or a command) decides when each replica can take traffic. A command probe, or an HTTP health endpoint that runs a short inference request, makes that check real. When a location has no ready replicas, the workload’s global endpoint routes requests to another healthy location. Metrics, logs, and traces for every location appear in one place, so you do not stitch together each provider’s dashboards.

The approach has held up in a real outage. During the October 2025 AWS us-east-1 outage, which lasted about 15 hours, Control Plane customers with us-east-1 in their GVC had no downtime, and their workloads were “active and running in healthy locations within 10 seconds,” according to the company’s internal incident telemetry. See designing systems that survive the inevitable.

Workloads can attach NVIDIA GPUs, and the pricing page lists models from T4 to B200. Control Plane’s GPU Ocean lets you pull GPU capacity from any provider by joining GPU servers as locations with MK8s.

Control Plane’s location routing options shift traffic between locations by setting priority tiers and latency offsets per location.

Control Plane backs this with a 99.999% availability SLA, which applies to workloads running at least two replicas in at least two locations. Those locations can be two regions of one cloud or locations on different providers.

A few more facts matter for inference teams:

• Where it runs. Control Plane runs serverless, standard, stateful, cron, and VM workloads on AWS, GCP, and Azure, on clusters you already run through BYOK, on your own machines (bare metal or VMs) turned into managed locations with MK8s, and on premises. On Control Plane-provided compute, there is no Kubernetes cluster you have to operate. Many teams run it in one cloud, so you can adopt it before you add a second provider.
• Cost. Customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure.
• Billing. On Control Plane-provided compute, you pay for the CPU and memory you reserve, by the millicore and the megabyte, with no contracts or minimums. Other resources such as egress, storage, load balancers, and GPUs are also billed.
• Identity. Control Plane’s Universal Cloud Identity gives workloads short-lived, least-privilege access to 600+ AWS, GCP, and Azure services, with no long-lived cloud keys in your code.
• Compliance. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.
• Tooling. The CLI, console, Terraform and Pulumi providers, and the Control Plane MCP server all work through the same Control Plane API. Because Control Plane ships its own providers, it complements the infrastructure as code you already run.

For the layers an inference stack needs, read what infrastructure you need to run an inference layer.

How do I run inference on my own GPUs alongside my cloud apps?

On Control Plane, join your GPU servers as a managed location with MK8s, or attach an existing cluster with BYOK, and add that location to the same GVC as your cloud locations. Your inference workload and your cloud apps then share one environment, one API, and one global endpoint. To prefer your own GPUs, set Control Plane’s location routing options. Put the owned location in a higher-priority routing tier than the cloud locations, and the cloud locations, which run the same workload, take traffic only when the owned location becomes unavailable. To lean traffic toward your own GPUs while cloud locations still serve users near them, give the cloud locations a positive latency offset instead. With MK8s, Control Plane operates the Kubernetes layer on the GPU servers you join, and the hardware stays yours.

FAQ

How do I fail over AI inference between GPU providers when one has an outage?

Run the same container image in at least two GPU locations on different providers, with weights pre-staged in each. Use health checks that run a real inference request, and route through a global endpoint outside any one provider. Drill the failover on a schedule.

How do I keep an AI product online when my model provider or GPU cloud has an outage?

For a GPU cloud outage, run in two providers. For a hosted model API outage, set a fallback chain: a second API provider, then a self-hosted open-weight model. Test prompts against every model in the chain.

How do I move between GPU providers as prices and availability change?

Package the model server as a container, define infrastructure as code, replicate weights across providers, and run routing outside any one provider. Then a move is a routing change plus a new location, not a rebuild.

How do I run inference on my own GPUs alongside my cloud apps?

Put your GPU servers and your cloud locations in one environment that shares a deployment API and a routing layer. On Control Plane, join the servers as Managed Kubernetes (MK8s) locations in the same Global Virtual Cloud (GVC) as your cloud locations.

How do I run LLM inference on my own GPUs and burst to the cloud when demand spikes?

Size owned GPUs for your steady load. Run the same image and pinned versions in a cloud location with weights already staged, and route peaks there. Plan for latency and transfer cost between the two.

How do I combine GPU capacity from multiple providers into one environment?

Use a platform that treats each provider’s GPUs as a location of one workload. On Control Plane, one workload runs in every location of a Global Virtual Cloud (GVC), and a global endpoint routes requests to the nearest healthy one.

CoreWeave vs Lambda vs AWS for GPU workloads: which should I use?

Treat them as parts of one setup. AWS is a hyperscaler with a broad service catalog, while CoreWeave and Lambda are neoclouds that focus on AI workloads. Compare current prices, availability, and regions on each provider’s own pages, and consider using more than one.

What are the best GPU cloud providers for inference?

The answer depends on your latency target, model size, residency rules, and exit cost. This guide’s comparison table covers AWS, Google Cloud, Microsoft Azure, CoreWeave, Lambda, Runpod, and Nebius, and the fuller roundup goes deeper.

How do I choose an AI model inference service for my application?

Set a latency target, estimate throughput, check that the model fits GPU memory, list your data residency needs, choose managed API or self-hosted, and estimate exit cost. Then qualify two providers.

What are common challenges in AI model inference and how can I overcome them?

The main ones are GPU availability, cold starts, global latency, idle GPU cost, version drift, observability, and data residency. Pre-staged weights, pinned versions, multi-location routing, and consistent metrics address most of them.

Next steps

Talk to the Control Plane team or sign up free.

---

Originally published on the Control Plane blog.