A GPU on Cloud Run is one flag. The reason a first attempt fails is never the flag; it is one of four constraints that are documented on four different pages.
Four constraints, checked before anything
Google’s Cloud Run GPU configuration documentation, read on 11 August 2026, states each of these. Check all four against your intended deployment before writing any code.
- GPU type. Two are offered: NVIDIA L4 with 24 GB of VRAM, and NVIDIA RTX PRO 6000 Blackwell with 96 GB. The gcloud identifiers are
nvidia-l4andnvidia-rtx-pro-6000. VRAM is the hard gate — a model that does not fit does not run, and quantisation is a decision to make before you pick the machine rather than after. - Region. Google lists L4 availability in asia-southeast1, asia-south1, europe-west1, europe-west4, us-central1 and us-east4; RTX PRO 6000 in asia-southeast1, asia-south2, europe-west4 and us-central1. A GPU deploy into any other region fails on the flag, not on capacity.
- CPU and memory minimums. An L4 service requires at least 4 vCPU and 16 GiB; an RTX PRO 6000 service at least 20 vCPU and 80 GiB. These are floors, not defaults, so a deploy that does not name them is rejected rather than adjusted.
- Quota. The metrics are
run.googleapis.com/nvidia_l4_gpu_allocationandrun.googleapis.com/nvidia_l4_gpu_allocation_no_zonal_redundancy, with equivalents for the RTX family. Google documents an initial allocation of 3 GPUs per region for L4. That is your real--max-instancesceiling regardless of what you set, because one instance takes one GPU: Google documents a maximum of one GPU per instance.
The zonal-redundancy split in the quota names is the one to understand rather than skim. Zonal redundancy reserves capacity in a second zone so a zonal failure does not take the service down, and it is charged and quota-counted separately. Turning it off with --no-gpu-zonal-redundancy is cheaper and is drawn from a different allowance, which occasionally means a deploy that fails on quota succeeds immediately with that one flag added.
The deploy command
gcloud run deploy inference \
--image=us-central1-docker.pkg.dev/PROJECT/serving/vllm:1.4 \
--region=us-central1 \
--gpu=1 \
--gpu-type=nvidia-l4 \
--cpu=4 \
--memory=16Gi \
--no-gpu-zonal-redundancy \
--max-instances=3 \
--concurrency=4 \
--timeout=600 \
--port=8080 \
--no-allow-unauthenticated
--concurrency is on that command deliberately. The default is 80, which is wrong for almost every GPU workload and is the subject of its own page. --timeout is the per-request ceiling and needs to accommodate your slowest generation, not your median one. --no-allow-unauthenticated is the right default for an inference service that costs dollars per hour to run; a public GPU endpoint is a billing incident waiting for a crawler.
One flag you cannot use: Google documents GPU services as instance-based billing only, so the request-based billing mode is unavailable. That has consequences further down this page.
Getting the weights in
Google’s GPU best-practices guidance draws the line at roughly 10 GB. Below that, storing model weights inside the container image is recommended and is the simplest thing that works: the image is pulled once per instance start and there is no runtime dependency on another service. Above it, the guidance is to download from Cloud Storage during container startup using concurrent transfers — gcloud storage cp or the Cloud Storage API — because building and storing multi-copy images of very large models is slow and expensive.
The tradeoff is not really about speed. An image with weights baked in is immutable and reproducible: the revision you deployed is exactly the bytes that will run, forever. Weights fetched at startup can change underneath a running revision, which is convenient right up to the moment a rollback fails to roll back because the bucket moved on. If you take the download path, pin the object generation rather than the object name.
Whichever path you take, the download is paid on every cold start, not once. Cloud Run instances do not share a persistent disk and there is no local cache surviving between them, so an instance that starts at three in the morning fetches the full model again. That is the term that dominates cold-start latency on a GPU service and it is the reason weights and the container should live in the same region as the service — a cross-region fetch of a ten-gigabyte checkpoint is a cost paid per instance, per start, forever.
The startup probe is not optional
Cloud Run considers an instance ready when it accepts a connection on the port. A model server that binds its port before loading weights is therefore declared ready while it is still loading, and the first requests routed to it fail or hang. Google’s own best-practices page raises exactly this case, noting that Ollama can open a TCP port before a model has loaded, and recommends preloading.
Configure a startup probe with a failure threshold generous enough to cover the whole load — pulling several gigabytes of weights and initialising CUDA can take minutes on a cold instance — and have the probe hit an endpoint that only returns success after a real inference has completed once. A probe on a route that returns 200 unconditionally is worse than no probe, because it converts a slow start into a silent error.
What it costs while idle
This is the part to internalise before the first deploy. Because GPU services bill per instance rather than per request, an instance that exists is an instance you pay for, whether or not it is serving. Cloud Run will scale a GPU service to zero when traffic stops, which is the main reason to run inference here rather than on a Vertex AI endpoint — but the cold start you get back is the full weight-load you just spent a page avoiding.
So the two configurations that make sense are the extremes, and the middle is a trap. Minimum instances set to zero gives you genuinely bursty economics and multi-minute cold starts, which is right for batch, evaluation and internal tools. Minimum instances set to one or more gives you warm latency and a bill that accrues every hour of every day, which is right for user-facing traffic and is straightforwardly comparable to just renting the GPU elsewhere. Picking a value in between because it feels safe gets you both costs and neither benefit.
GPU types, regional availability, resource minimums and initial quota all change, and the RTX PRO 6000 option is recent enough that its regions are still expanding. Verify against Google’s Cloud Run GPU documentation before committing a region choice to infrastructure code.
Top comments (0)