Four pods at 180% CPU, target 60%. The formula says 12 replicas. The HPA goes to 8.
Nothing is broken. Three rules sit between the formula and the replica count, and they explain most "why did the HPA do that" threads. I'd written them up from the docs. Then I did what I should have done first: I put a load generator in front of a real HPA (kind, Kubernetes 1.37, metrics-server at 15-second resolution) and watched it with a stopwatch.
Two of the three rules behaved exactly like the docs. The third surprised me, and so did a fourth one I hadn't written about at all.
The formula
desired = ceil(current × currentMetric / target)
= ceil(4 × 180 / 60) = 12
Utilization is measured against the pod's request, not its limit. A pod without a CPU request can't be scaled on CPU at all.
The lab setup: the classic hpa-example image (a PHP page that burns CPU), 200m request, 4 replicas, HPA target 60%, min 1, max 20, all controller defaults. I aimed the load at 180%. Real load doesn't aim well: the HPA read 152% at the first evaluation, so in my run the formula said ceil(4 × 152 / 60) = 11.
Rule 1: the default scale-up policy
Without a behavior section, scale-up is limited to the larger of +100% or +4 pods every 15 seconds. From 4 pods that's 8, whatever the formula says.
In the lab it held to the letter, and faster than I expected. I created the HPA at 15:03:29. Three seconds later:
15:03:32 cpu: 152%/60% current 4 desired 8
Normal SuccessfulRescale New size: 8; reason: cpu resource utilization
(percentage of request) above target
The first evaluation doesn't wait for a 15-second tick. The HPA wakes up hungry.
Rule 2: new pods count as zero
This is the part my original post got wrong. I'd written the next step as "8 pods share the load at 90%, and the HPA goes to 12." Here's what actually happened 15 seconds later:
15:03:47 cpu: 156%/60% current 8 desired 11
Eleven, not twelve. The four new pods were Ready within one to three seconds, but metrics-server hadn't scraped them yet. When scaling up, the HPA treats pods without metrics as using 0% of their request. So it averaged four pods at 156% with four pods at 0%: 78%, and ceil(8 × 78 / 60) = 11.
That's deliberate, and it's in the docs: it stops the HPA from overshooting while new pods warm up. It also means the replica count you get is shaped by how fast your metrics pipeline sees new pods, not just by the load.
On 11 pods the load came out at 55%, a ratio of 0.92, which is inside the tolerance. The HPA stopped there and never went to 12. The formula's answer was never the destination; it was the first guess.
Rule 3: the 10% tolerance (and how sharp its edge is)
If currentMetric / target is between 0.9 and 1.1, the HPA does nothing. Four pods at 65% with a 60% target: the ratio is 1.08, so you stay at 4, even though the formula says 5.
I tested the edge with a single replica:
15:16:55 cpu: 66%/60% current 1 desired 1 (ratio 1.10, no action)
15:17:00 cpu: 68%/60% → New size: 2 (ratio 1.13)
Two percentage points of CPU, one extra pod. The band is real, and it's razor-sharp.
What I didn't expect: near the target, the HPA isn't asleep, it's jumpy. Trying to hold 2 pods around 65%, the 15-second metric swung from 34% to 71% between samples. One noisy 71% sample (ratio 1.18) scaled 2 → 3, and then the scale-down window kept that third pod for five minutes at 42–49%. Near the target, metric noise beats the tolerance band.
Rule 4: the 5-minute scale-down window replays the past
Scale-down uses a 5-minute stabilization window: the HPA picks the highest recommendation from the last five minutes. I removed the load at 15:06:02 with 11 replicas running. The timeline:
15:06:02 load removed 11 replicas
15:07:33 metric reaches 0% (90 s of lag) 11
15:11:32 New size: 10 ← 5 m 30 s after the load stopped
15:11:48 New size: 5
15:12:03 New size: 1
That 10 → 5 → 1 staircase is the HPA's own recommendations from 15:06:48, 15:07:03 and 15:07:18, each replayed exactly five minutes later. The window doesn't wait five minutes and then decide. It waits five minutes and then repeats what it would have done.
Add the metric lag, and the real delay from "load gone" to "pods gone" was 5.5 minutes, not 5.
Same load, with a behavior section
Then I gave the same HPA a custom behavior: scale up by at most 2 pods per 15 seconds, and scale down with a 60-second window and at most 50% per 15 seconds.
behavior:
scaleUp:
policies:
- type: Pods
value: 2
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 60
policies:
- type: Percent
value: 50
periodSeconds: 15
From kubectl get hpa -w:
15:24:53 145%/60% 4 replicas
15:25:06 147%/60% 6
15:25:21 102%/60% 8
15:25:36 70%/60% 10
... 10 (settled at ~61%)
15:27:40 load removed
15:28:51 0%/60% 10
15:29:36 0%/60% 5
15:29:51 0%/60% 2
15:30:06 0%/60% 1
A gentle staircase up, and pods gone about 1 minute 40 seconds after the load, instead of 5.5 minutes. If your pods are expensive and your traffic is spiky, this is the section to change. If your traffic comes back every three minutes, the default five-minute window is protecting you from yourself.
Things that went wrong in my lab (so they don't in yours)
- I started
kubectl get hpa -wtwo seconds before creating the HPA. It died instantly withError from server (NotFound)and didn't retry. Create first, watch second. A dumb 5-second polling loop saved the run. - I stored
kubectl --context …in a shell variable, and zsh happily ran it as a single command name:command not found: kubectl --context kind-nodesize-lab. A three-line wrapper script fixed it. - Busybox
shdoesn't supporttime ( … ). I found out fromsyntax error: unexpected word, which is busybox's way of saying "this isn't bash".
See the steps for your numbers
I built a small calculator that applies the formula, the tolerance, min/max and the default scale-up policy step by step, then tells you where the replica count settles and whether you hit maxReplicas: HPA calculator. It shows the shape, not the timing: as the lab shows, metric lag and new pods counted at zero bend the real curve.
Your turn
What's in your behavior section? Do you keep the defaults, or did a spike teach you otherwise?
Top comments (0)