Serverless GPUs on Container Apps give you an A100 or a T4 that scales to zero and bills per second. The constraints are narrow and specific, and two of them will invalidate an architecture you have already drawn.
What serverless GPU is and is not
Microsoft documents serverless GPU support for NVIDIA A100 and NVIDIA T4, with per-second billing, scale to zero, and data governance in the sense that your data never leaves the container boundary. It is supported only for Consumption workload profiles and is not supported for Consumption-only environments — those are two different things, and the second is the legacy environment type. If your environment is Consumption-only, no amount of CLI produces a GPU; you need a new workload profiles environment. Microsoft, Using serverless GPUs in Azure Container Apps.
The documented support limitations that change designs:
- Only one container in an app can use the GPU, and it is the first container listed. A sidecar-plus-model-server layout works, but only if the model server is first — and nothing warns you when it is not; the second container simply sees no device.
- Multi and fractional GPU replicas are not supported. One replica, one GPU. A model that needs two A100s does not fit here at all.
- Multiple apps can share the same GPU workload profile, but each requires its own replica.
- Consumption GPUs use one IP address per replica when integrated with your own virtual network — which is a subnet-sizing input, not a footnote.
The GPU software stack is platform-managed. At the time of writing Microsoft documents the current NVIDIA driver as 570 with CUDA 12.x, and an announced transition to driver 580 with CUDA 13.x. Versions are documented at the major or minor family level, and patch versions may change without notice. If your image pins a CUDA base image you are insulated; if you rely on the platform-provided runtime, that transition is a compatibility event you must test for.
Quota comes first
You need serverless GPU quota before any of this works, and it is requested through a support case rather than a self-service form. Microsoft notes that customers with enterprise agreements and pay-as-you-go customers have A100 and T4 quota enabled by default — so check what you already have before filing anything.
- Open a new support request with issue type Service and subscription limits (quotas).
- Set quota type to Container Apps.
- In the details pane, choose the quota by its exact name: Managed Environment Consumption NCA100 Gpus or Managed Environment Consumption T4 Gpus. Those two strings are the whole reason this section exists — a request for “GPU quota” without them gets routed somewhere else.
Region availability is uneven and asymmetric: at the time of writing the T4 list is considerably longer than the A100 list, and several regions offer T4 only. Check the current table before choosing a region, because picking one for latency and discovering it has no A100 is a full redeploy.
Adding the workload profile
A consumption GPU profile is managed like any other workload profile:
az containerapp env workload-profile list-supported \
--location westus -o table
az containerapp env workload-profile add \
--name env-inference \
--resource-group rg-model \
--workload-profile-name gpu-t4 \
--workload-profile-type Consumption-GPU-NC8as-T4
Run list-supported first and take the profile type string from its output rather than from any article, including this one. Profile type names are region-dependent and change as hardware is added; the list command is the authoritative source for your subscription in your region.
Deploying onto it
az containerapp create \
--name llm-server \
--resource-group rg-model \
--environment env-inference \
--workload-profile-name gpu-t4 \
--image myregistry.azurecr.io/vllm-server:0.6.3 \
--target-port 8000 --ingress internal \
--min-replicas 0 --max-replicas 2
For models published in the Azure ML registry as MLFLOW type, Microsoft documents a path that skips image building entirely:
az containerapp up \
--name llm-server \
--location westus \
--resource-group rg-model \
--model-registry azureml \
--model-name <MODEL_NAME> \
--model-version <MODEL_VERSION>
Microsoft notes that using --model-registry, --model-name and --model-version sets the environment variables that optimise cold start for you — which matters more than it sounds, for the reason below.
Two defaults that do not apply here
GPU workload profiles are carved out of two platform behaviours that hold everywhere else in Container Apps, and both carve-outs cost money or availability if you do not know about them.
No default health probes. When you enable ingress, Container Apps normally adds default Startup, Liveness and Readiness probes to the main app container. Microsoft documents an exception for GPU workload profiles, both dedicated and consumption — no probes are added. On a CPU app that default is a safety net you rarely think about; on a GPU app there is no net, and the failure it would have caught is exactly the one a GPU app has. A model server binds its port long before the weights are resident on the device, so without a readiness probe the platform routes the first request into a replica that is still loading. Write one, and make it check that the model is actually loaded rather than that the process is alive:
"probes": [
{
"type": "Readiness",
"httpGet": { "path": "/health/ready", "port": 8000 },
"initialDelaySeconds": 10,
"periodSeconds": 10,
"timeoutSeconds": 5,
"failureThreshold": 60,
"successThreshold": 1
}
]
A generous failureThreshold is the point: at a ten-second period, 60 failures is ten minutes of grace, which a multi-gigabyte weights load can genuinely need. Too small a threshold and the platform restarts the replica just before it would have become ready, forever.
No idle billing rate. Container Apps normally bills a replica held at the minimum replica count at a reduced idle rate when it is doing nothing. Microsoft documents that idle usage charges do not apply to serverless GPU apps — they are always billed for active usage. Set --min-replicas 1 on a GPU app to avoid cold starts and you are paying the full GPU rate every second of every day, whether or not a request arrives. Microsoft, Billing in Azure Container Apps.
That single rule is what decides whether serverless GPU is the right shape for your workload. If traffic is bursty and you can tolerate the first-request latency, scale to zero and the per-second billing is the whole value proposition. If you cannot tolerate it and pin a replica, you are paying a dedicated GPU price through a serverless product, and a dedicated workload profile is likely both cheaper and more predictable. Decide that consciously rather than by adding a minimum replica during an incident.
Cold start is the real problem
Scale to zero on a GPU app means the next request waits for an image pull and a weights load. Weights are gigabytes. This is the difference between a serverless GPU that is a good idea and one that is a permanent apology, and Microsoft documents two levers:
- Artifact streaming in Azure Container Registry, which starts the container before the whole image has landed. It requires a premium registry.
- Storage mounts. Keep large files — model weights above all — in an Azure storage account mounted into the app rather than baked into the image. The image stays small and the weights are read rather than pulled.
If neither gets the first-request latency into an acceptable range, the honest conclusion is that this workload wants a replica floor of one and the per-second billing story does not apply to it. That is a legitimate outcome; discovering it after the architecture review is not.
Driver and CUDA versions, region availability and workload profile type names all change. Every figure here is Microsoft’s documented state at the time of writing — treat the serverless GPU overview and workload-profile list-supported as the current answer.
Top comments (0)