A Kubernetes Constraint for a novice architect

Your Business Scales on Connections. Your Cluster Scales on CPU.

Why the most common Kubernetes autoscaling setup measures the wrong thing — and what it actually costs to fix.

The dial the business turns

Nobody in a business review has ever asked for more CPU.

What they ask for is this: "Marketing is running the campaign on the 14th. Can the platform take it?" And when they describe "it," they describe it in units they can defend — orders per minute, concurrent shoppers, quotes per second, how long a page takes before someone gives up.

Shopify is a good public example of doing this well. Ahead of Black Friday–Cyber Monday 2025, its engineering team ran five scale tests between April and October against forecast traffic, reaching 146 million requests per minute and over 80,000 checkouts per minute in the fourth test, then pushing a p99 scenario at 200 million requests per minute. The weekend itself peaked at 489 million requests per minute at the edge and $5.1 million in sales per minute. Note the units on both sides of that sentence: requests per minute and dollars per minute. That's the translation layer working.

The cautionary version is Ticketmaster's 2022 Eras Tour presale. The business asked the platform owner, explicitly and repeatedly, whether it could take the load. Taylor Swift's own statement afterwards was that "we asked them, multiple times, if they could handle this kind of demand" — and were assured it could. On the day, 3.5 billion system requests arrived, about four times the previous peak, queues were throttled to keep the platform standing, and the general sale was cancelled.

So business stakeholders do give us a number. Usually several: request rate, response time, concurrent sessions, connections held open. The question is what our autoscaler does with them.

The dial Kubernetes reads

The default answer, in the overwhelming majority of clusters, is a HorizontalPodAutoscaler pointed at CPU:

metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Every 15 seconds the controller reads average utilization across the pods and computes desiredReplicas = ceil(currentReplicas × currentValue / targetValue), ignoring changes inside a ±10% tolerance band, scaling up immediately but holding a five-minute stabilization window before scaling down.

This is a fine mechanism. It rests on one assumption: that CPU is a faithful proxy for load. For a CPU-bound service — compression, rendering, model inference, heavy serialization — it is. For the typical business service that accepts a request, calls a database and two internal APIs, and assembles a response, it isn't. Those services spend most of their wall-clock time waiting, and waiting is close to free in CPU terms.

Where the two dials come apart

The unit that actually runs out first is concurrency — a worker thread, a slot in an HTTP client pool, a connection borrowed from the database pool.

The arithmetic is Little's Law, and it's worth putting in front of stakeholders because it needs no Kubernetes knowledge at all:

in-flight requests = arrival rate × time to serve a request

At 2,000 requests per second and 50 ms of service time, roughly 100 requests are in flight at any moment. Now a downstream dependency degrades and service time goes to 250 ms. Traffic hasn't changed. But in-flight requests jump to 500 — five times the concurrency, five times the threads held, five times the connections checked out. The work per request is identical, so CPU barely moves. Per-request CPU may even fall, because threads are parked on socket reads instead of executing.

This produces two failure modes, and most organisations have quietly lived through both.

Failure mode one: saturation the autoscaler cannot see. The pool is empty, threads are blocked waiting to borrow a connection, latency is climbing exponentially, requests are timing out — and CPU is sitting at 30%, so nothing scales. This is such a well-worn pattern that it has a nickname in the database community: connection pool exhaustion as the silent killer, where the pool saturates while compute metrics barely register. Worse, the pods often look unhealthy enough for liveness probes to fail, so Kubernetes restarts them, and the fresh pods reconnect into the same wall.

Failure mode two: scaling that makes things worse, expensively. Each replica carries its own connection pool. Twenty pods with a pool of 20 demand 400 connections from a database that may only accept a couple of hundred. So when CPU-based scaling does fire during a slowdown, it adds contenders for a fixed downstream resource rather than capacity. One publicly written-up incident describes exactly this: CPU stayed under 60%, the HPA added pods, the extra pods competed for the same saturated read-replica pool, and p99 latency went from 120 ms to over 900 ms with no improvement at the database. PostHog's public postmortems from October 2025 show the same entanglement from the other direction — CPU pressure driving retries, which drove connection pool exhaustion, which cascaded.

And the cost tail: the five-minute scale-down window means every transient CPU blip you do catch is paid for in pod-minutes long after the blip is gone. Overprovisioned and under-protected at the same time.

So can you just scale on connections?

Yes. It is not one line of YAML.

Kubernetes has no native notion of "open connections." Getting there means building a metrics path:

1. Instrument the application. Export pool state as a gauge — active connections, idle connections, threads waiting to borrow, in-flight requests. Most frameworks already have this (HikariCP, most HTTP servers, most language runtimes); it just isn't scraped.
2. Collect it. Prometheus, or your platform's equivalent.
3. Make it visible to the autoscaler. Either deploy the Prometheus Adapter to serve the Kubernetes custom/external metrics API — an extra component to configure, secure and keep alive — or use KEDA, which wraps the HPA and does the adapter work for you via scalers.
4. Choose a target. This is the hard part, and it's an architecture decision, not a config value.

The KEDA version is compact enough to review in a design meeting. Scale on saturation, not on a raw count:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    name: orders-api
  minReplicaCount: 4
  maxReplicaCount: 25          # ceiling set by the DB, not by ambition
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: |
          avg(hikaricp_connections_active{service="orders-api"})
          / avg(hikaricp_connections_max{service="orders-api"})
        threshold: "0.7"       # scale when pools are 70% checked out
    - type: cpu                # keep CPU as a secondary guard
      metricType: Utilization
      metadata:
        value: "75"

If the signal you care about is genuinely concurrent HTTP requests rather than downstream connections, KEDA's HTTP add-on counts in-flight requests directly by putting an interceptor proxy in the request path and scaling on concurrency or request rate. It is honest about the trade: still beta, community-maintained, and every request takes an extra hop. Its API also changed recently (InterceptorRoute replacing HTTPScaledObject), which tells you something about maturity.

The part that isn't a YAML problem

Three things to say out loud before anyone builds this.

A ratio scales; a count doesn't. The HPA divides by replica count. "600 open connections" is meaningless as a target because the right number depends on how many pods exist. active / max per pod is scale-invariant. Get this wrong and you build an oscillator.

Pool size and replica count are one decision, not two. maxReplicas × poolSize must stay under what the database will accept, with headroom for migrations, admin sessions and failover. If that product exceeds the ceiling, autoscaling is a mechanism for causing outages faster. This is where a connection pooler like PgBouncer or ProxySQL earns its keep — multiplexing many client connections onto fewer backend ones so the two decisions can be decoupled.

Sometimes the right answer is not more pods. If the bottleneck is a fixed-capacity dependency, scaling the stateless tier cannot help; the correct responses are load shedding, admission control, queueing the work asynchronously, or raising the ceiling downstream. Autoscaling on connections tells you the truth about saturation — it does not create capacity that doesn't exist.

What to take into the next planning conversation

The translation is short enough to fit on one slide:

Business asks about	Technical signal	Does CPU track it?
Shoppers on site at once	in-flight requests, open connections	No — waiting is cheap
Orders per minute	request rate	Loosely, and only if latency is stable
"The site feels slow"	p99 latency, pool wait time	No — often inversely
"Will we survive the campaign?"	concurrency budget vs. downstream ceiling	Not at all

Default CPU autoscaling is a reasonable starting position and a poor finishing position. The mature version is: pick the signal from the SLO the business actually cares about, work out the concurrency budget per pod, scale on how full that budget is, keep CPU as a secondary guard, cap replicas at what your dependencies can survive, and prove it with a load test before the campaign date rather than during it.

The complexity isn't in the tooling. KEDA plus one Prometheus query is a day of work. The complexity is in knowing which number matters — and that is an architect's job, not the autoscaler's.