---
title: "Kubernetes HPA vs KEDA: Taming Spiky Mobile Backend Traffic"
published: true
description: "Why CPU-based HPA fails during push notification campaigns, and how KEDA's event-driven autoscaling cuts the cold-start penalty that wrecks p99 latency."
tags: kubernetes, mobile, devops, performance
canonical_url: https://blog.mvpfactory.co/kubernetes-hpa-vs-keda-mobile-backend
---
## What We Are Building (and Why It Matters)
By the end of this workshop, you will understand exactly why HPA's CPU signal lags behind real mobile traffic bursts, how to configure KEDA to react to queue depth instead of resource exhaustion, and the three tuning parameters that prevent the oscillation pattern that kills p99 latency after a push notification campaign.
This is not theoretical. I have watched a 2-million-notification campaign take a production backend from 120ms p99 to 4 seconds in under a minute — entirely because of a default autoscaler configuration that was never revisited.
---
## Prerequisites
- A running Kubernetes cluster (1.24+)
- KEDA installed (`helm repo add kedacore https://kedacore.github.io/charts`)
- A message queue (RabbitMQ, SQS, or Kafka — examples use RabbitMQ)
- Basic familiarity with `kubectl` and YAML manifests
---
## The Core Problem: HPA Is a Lagging Indicator
Here is the gotcha that will save you hours.
Mobile traffic does not ramp — it steps. Push notification campaigns, App Store featuring events, and viral moments share one characteristic: they go from zero to thousands of RPS in seconds, not minutes.
HPA's default signal chain looks like this:
Traffic spikes → CPU/memory climbs → HPA observes metric →
Scaling decision made → Pod scheduled → Container starts → Ready
That chain takes **60 to 180 seconds** in a typical cluster. CPU-based HPA scrapes on a 15-second interval with a 30-second evaluation window. For a campaign delivering 2 million notifications in under 30 seconds, your backend has already absorbed the full burst before the first new pod is ready to serve traffic.
You are always reacting to what already happened.
---
## Step 1: Install KEDA and Understand the Inversion
KEDA inverts the model. Instead of reacting to resource exhaustion, it scales on **leading indicators** — queue depth, message lag, incoming RPS — before your pods are under stress.
bash
helm install keda kedacore/keda --namespace keda --create-namespace
Verify it is running:
bash
kubectl get pods -n keda
---
## Step 2: Replace Your HPA with a ScaledObject
Here is the minimal setup to get this working. Drop your CPU-based HPA and replace it with this `ScaledObject`:
yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: mobile-notification-processor
spec:
scaleTargetRef:
name: notification-service
minReplicaCount: 3
maxReplicaCount: 50
triggers:
- type: rabbitmq
metadata:
queueName: push-notifications
value: "100" # target messages per replica
advanced:
stabilizationWindowSeconds: 30
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Pods
value: 10
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60
The asymmetry between `scaleUp` and `scaleDown` is intentional. Let me show you a pattern I use in every project: scale up aggressively, scale down conservatively.
---
## Step 3: Tune the Three Parameters That Prevent p99 Degradation
| Parameter | HPA Default | KEDA Recommendation | Why It Matters |
|---|---|---|---|
| `scaleUp.stabilizationWindowSeconds` | 0 (but CPU lag adds ~60s) | 0 explicit | React immediately to queue growth |
| `scaleDown.stabilizationWindowSeconds` | 300s | 300–600s | Prevent oscillation after burst |
| `minReplicaCount` | 0 or 1 | 3+ | Eliminate cold-start for warm floor |
| `scaleUp policy (Pods)` | 4 pods/60s | 10 pods/15s | Match burst velocity of push campaigns |
| `scaleDown policy (Percent)` | — | 10%/60s | Gradual drain prevents rebound spikes |
**The replica floor is non-negotiable.** Most teams treat `minReplicaCount: 0` as a cost optimization, then wonder why p99 spikes from 120ms to 4 seconds during the first minute of a campaign. That is the cold-start penalty, and it is entirely self-inflicted.
A floor of 3–5 pods, sized to handle baseline traffic comfortably, eliminates it for the initial burst wave.
---
## Gotchas
**Aggressive scale-down causes rebound spikes.** App Store featuring events drive long tails of organic discovery traffic for hours after the initial peak. If you scale down too fast after the first burst and a secondary wave hits, you are right back in cold-start territory. The 10%/60s policy in the manifest above exists specifically to prevent this.
**`minReplicaCount: 0` is not a cost optimization — it is a latency time bomb.** The incremental cost of 3 warm pods is trivial compared to the user-facing impact of a 1-star review wave from a botched featuring event.
**The docs do not mention this, but** KEDA makes scaling decisions within 5 seconds of queue growth. If your trigger's `pollingInterval` is set too high (default is 30s), you lose that advantage. Set `pollingInterval: 5` for push-triggered workloads.
**Delete your existing HPA before applying a ScaledObject.** KEDA creates its own HPA under the hood. Having both will cause conflicts and unpredictable behavior.
bash
kubectl delete hpa notification-service
kubectl apply -f scaledobject.yaml
---
## Conclusion
Replace CPU-based HPA with KEDA for any mobile backend that receives push-triggered traffic. Queue depth and consumer lag are leading indicators; CPU is a lagging one.
Set `minReplicaCount` to at least 3 and size those pods for your P50 load. Use asymmetric stabilization windows: zero on scale-up, 300+ seconds on scale-down. The oscillation pattern that wrecks p99 latency in the minutes after a burst comes from aggressive scale-down followed by a rebound — conservative scale-down is the fix.
The 60–90 second lag in CPU-based scaling is not a Kubernetes limitation. It is a configuration choice. KEDA gives you the tooling to make a better one.
**Further reading:**
- [KEDA documentation](https://keda.sh/docs/)
- [KEDA ScaledObject spec reference](https://keda.sh/docs/2.13/concepts/scaling-deployments/)
- [Kubernetes HPA behavior configuration](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior)
Top comments (0)