Cloud Run’s default of 80 concurrent requests per instance was chosen for web applications that spend most of their time waiting. A model doing a forward pass is not waiting, and 80 is off by an order of magnitude in the expensive direction.
What the default actually is
The setting is --concurrency on the gcloud command and containerConcurrency in the service YAML. Google’s documentation gives the default as 80 and the maximum as 1,000 requests per instance, and notes that the default applies when a service is created rather than on every subsequent deploy — so a service created years ago keeps whatever it was created with, and reading the current value is more reliable than assuming it.
The name is worth reading literally. It is the maximum number of requests one instance may be handling at once, not a rate and not a queue depth. Eighty means the platform is willing to hand a single container eighty simultaneous requests before it starts another instance.
Two workloads, opposite answers
“Model inference on Cloud Run” covers two things that behave nothing alike, and the right concurrency for one is badly wrong for the other.
A service that proxies an upstream provider — your container receives a request, calls Gemini or another hosted API, and streams the answer back — spends essentially all of its time on a network wait. It uses almost no CPU per request. Here 80 is reasonable and sometimes low: the constraint is not your instance, it is the upstream rate limit, and adding instances does not raise that. Raising concurrency on this shape saves money; lowering it burns money multiplying idle containers.
A service that runs the model locally is the opposite. A forward pass saturates the accelerator or the CPU for its duration. Two concurrent requests on one GPU do not run in half the time each; they interleave, and both get slower while the instance looks busy rather than overloaded. Set concurrency to 80 here and eighty users share one GPU while Cloud Run sees no reason to scale, because from the platform’s point of view the instance is nowhere near its limit. Latency collapses and the autoscaler never reacts.
Google’s GPU best-practices documentation offers a formula for the second case: the number of model instances multiplied by the parallel queries each can handle, plus the number of model instances multiplied by the ideal batch size. In practice, for a single model instance on a single GPU with no request batching, that lands in the low single digits — which is very far from 80.
Python adds a third consideration to the local case. A synchronous single-threaded server holds the interpreter lock through the forward pass, so concurrency above one does not overlap anything useful; it only adds queueing inside your process where the platform cannot see it.
Concurrency is an autoscaling input
The reason a wrong value is so damaging is that concurrency is not a safety cap. It is the denominator the autoscaler divides by. Cloud Run adds instances when observed concurrency approaches the configured maximum, so the setting is a statement about how much one instance can hold, and the platform trusts it.
Overstate it and you get one overloaded instance and no scaling. Understate it and you get an instance per request. Google’s own concurrency documentation works through a case where setting concurrency to 1 required roughly 400 instances for a load that needed far fewer at the default, with the cost following the instance count directly.
This also interacts with cold starts. A low concurrency means traffic growth creates instances constantly, and every new instance on a GPU service pays a weight-load cold start. A concurrency of 1 on a service with a two-minute startup is a service that is permanently starting up.
Memory is the constraint people forget in the proxy regime, and it fails abruptly rather than gradually. Each in-flight request holds its request body, its response buffer and whatever the framework allocates around them, so peak memory is roughly per-request footprint multiplied by concurrency. Raise concurrency to 200 on a service handling large payloads and the instance is killed for exceeding its memory limit, which surfaces as requests failing with no application-level error at all. The arithmetic is worth doing once: divide the instance memory by a conservative per-request estimate and treat that as the ceiling on concurrency, regardless of what the CPU could manage.
Choosing a number
- Establish which regime you are in. If the container calls out to a provider, you are in the proxy regime; if it holds weights, you are in the local regime. A service that does both should be split into two services, because no single number is right for both.
- For the local regime, start from how many requests the model can genuinely have in flight — one per model replica, plus whatever your batching layer merges — and set concurrency to that.
- For the proxy regime, start at the default and raise it until either memory per instance or the upstream rate limit becomes the binding constraint. Those are the two real ceilings.
- Set
--max-instancesdeliberately in both cases. Concurrency decides when to scale; max instances decides where to stop, and on a GPU service it is capped by your accelerator quota anyway. - Watch tail latency rather than average latency as you change it. The symptom of concurrency being too high is a p99 that separates from the p50 while the instance count stays flat.
What happens at the limit
When every instance is at its concurrency maximum and the service is at --max-instances, additional requests are queued briefly and then rejected by the platform with a 429 before your container ever sees them. That is a different 429 from a provider rate limit, and confusing the two sends people to the wrong dashboard: this one is solved by raising max instances or concurrency, not by backing off against an upstream.
Streaming complicates the accounting in a way worth flagging. A streamed response occupies its concurrency slot for the entire duration of the stream, not just for the time to first token, so a service returning thirty-second generations holds each slot thirty seconds. Effective throughput is concurrency divided by response duration, and a service that looks comfortable at a hundred requests a minute of short answers will be saturated by ten long ones. Streaming from Cloud Run covers the mechanics.
Top comments (0)