Why CPU-Based Autoscaling Makes Traffic Spikes Worse Before It Makes Them Better

The problem

A traffic spike hits. The Horizontal Pod Autoscaler (HPA) is watching average CPU utilization, target 70%. Requests per second triples in under a minute. Instead of smoothly adding capacity, the system does the opposite of what you'd expect: latency climbs, then error rate climbs, and pods keep getting added anyway — but too late, and too many at once. By the time things stabilize, you've paged three people and burned twenty minutes at 4x normal latency.

Nobody misconfigured anything. The autoscaler is working exactly as designed. The design is the problem.

Why it happens

Break down what HPA actually measures, and when.

1. Metric collection lag. metrics-server scrapes kubelets on an interval (commonly 15-60s), and HPA itself evaluates on its own sync period (15s by default). Your "current CPU" is already tens of seconds old by the time a scaling decision gets made on it. During a spike that doubles load in 90 seconds, that lag alone means every decision is made against traffic that no longer exists.

2. Average CPU is a saturating signal, not a demand signal. Once every existing pod is pegged near 100%, average CPU plateaus near 100% whether you're 10% over capacity or 300% over capacity. The metric that's supposed to tell HPA "how much more do I need" stops carrying that information exactly when you need it most — it can tell you you're maxed, not by how much.

3. New pods aren't instant capacity. Scheduling, image pull, readiness-probe delay, and — for anything with a warm cache or JIT — real warmup time before a pod is actually absorbing its share of load. A pod can show Running and Ready for a full minute before it's doing useful work. Meanwhile HPA's stabilization window can let it pile on more pods before the first batch has ramped up, overshooting the correction.

4. Thundering herd at the next layer down. Every new pod opens its own DB connection pool on boot. Scale from 10 pods to 40 in one HPA decision and you've just asked your database for 4x the connections in seconds — often the actual cause of the outage, not the original traffic spike. The layer you scaled to protect (compute) just attacked the layer you didn't (the database's max_connections).

What to do about it

• Scale on a leading indicator, not a lagging one. Request queue depth, in-flight request count, or requests-per-second-per-pod predicts saturation before CPU does, because it moves before compute exhausts. Custom metrics via Prometheus Adapter or KEDA let HPA target these instead of, or alongside, CPU.
• Don't try to out-tune the lag. Shortening the metrics window helps marginally but you're fighting collection lag with more collection, which has its own noise and cost tradeoff. Treat it as a mitigation, not a fix.
• Pre-provision for known patterns. If your spike is a marketing send or a cron-triggered batch job, scheduled scaling (a KEDA cron scaler, or a plain scheduled kubectl scale) beats reactive scaling every time — you're not waiting on a metric at all.
• Cap max replicas at what your downstream can actually absorb, and enforce that cap explicitly instead of discovering it as an outage. Pair it with connection pooling (PgBouncer, RDS Proxy) so a burst of new pods doesn't equal a burst of raw DB connections.
• Gate readiness on real warmup, not process start. If your workload is cache- or JIT-sensitive, a readiness probe that only checks "the process is up" will route production traffic to a pod that isn't actually ready to serve it well.

Key takeaways

• Reactive CPU-based autoscaling has lag from three compounding sources — metric collection interval, HPA evaluation interval, and pod startup/warmup time — and they stack, not average out.
• Average CPU stops being informative exactly when you need it most, because it saturates instead of scaling with demand.
• Autoscaling that isn't capacity-aware of its downstream dependencies doesn't prevent outages — it relocates them, usually straight into your database's connection pool.
• The fix isn't "scale faster." It's "scale on a signal that doesn't lag, and cap scaling at what downstream can survive."