HPA does not keep the replicas you set with kubectl scale

Sometimes you want more Pods even though CPU usage is low: when the workers in each Pod are all tied up waiting on a downstream response, or when you want capacity in place before an event. In those cases, people sometimes raise replicas with kubectl scale, even on a Deployment that has an HPA (HorizontalPodAutoscaler). HPA is a controller that decides the number of Pods from metrics such as CPU usage and rewrites the Deployment's replicas.

Right after the scale-up, READY in kubectl get deploy matches the new count. HPA does not revert it right away either, so the count looks like it will hold.

A few minutes later, it is back to the original count. No errors or warnings appear. If HPA reverted it, it is not clear why it took a few minutes to do so.

I reproduced this state and checked whether HPA was really what reverted it, and why the revert happens a few minutes later.

Reproduction

I reproduced the state above on a Kubernetes v1.37.0 cluster created with kind.¹

I attached an HPA to a Deployment that runs 2 Pods using almost no CPU. The HPA targets 50% CPU utilization, with minReplicas 2 and maxReplicas 5. HPA decides the count again every 15 seconds. Below, each of these runs is called a sync.

$ kubectl get hpa c1-1
NAME   REFERENCE         TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
c1-1   Deployment/c1-1   cpu: 0%/50%   2         5         2          81s

I scaled this Deployment from 2 to 4 with kubectl scale.

$ kubectl scale deploy/c1-1 --replicas=4
deployment.apps/c1-1 scaled

Right after the scale-up, READY reached 4/4.

$ kubectl get deploy c1-1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
c1-1   4/4     4            4           107s

About 5 minutes later, it went back to 2.

$ kubectl get deploy c1-1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
c1-1   2/2     2            2           7m17s

Narrowing it down

Deployment replicas can be written back by kubectl apply run by another person or by CI, by GitOps tools, and by controllers such as HPA.

Kubernetes objects carry managedFields, a record of which manager (writer) wrote each field of the object.² Viewing the Deployment after it went back to 2 with kubectl get deploy c1-1 -o yaml --show-managed-fields showed that the manager of spec.replicas was kube-controller-manager.

  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:replicas: {}
    manager: kube-controller-manager
    operation: Update
    subresource: scale

kube-controller-manager is the in-cluster process that runs HPA and the other controllers. If kubectl apply or a GitOps tool had written the field, the manager would be some name other than kube-controller-manager.

Events show which controller wrote it. An Event is a record of something that happened, which Kubernetes keeps per object. kubectl events --for hpa/c1-1 showed a SuccessfulRescale at the time of the revert.³ New size: 2 means HPA rewrote replicas to 2. HPA was what reverted it.

$ kubectl events --for hpa/c1-1
LAST SEEN               TYPE      REASON                         OBJECT                         MESSAGE
16s                     Normal    SuccessfulRescale              HorizontalPodAutoscaler/c1-1   New size: 2; reason: All metrics below target

What remains of the opening question is why HPA takes about 5 minutes to revert.

What is happening

What a sync does

On each sync, HPA does the following.

1. Reads the Pods' CPU utilization
2. Divides the average utilization by the target to get a ratio, and computes a Pod count from the ratio. Below, this count is called the computed count. If the ratio is between 0.9 and 1.1, the computed count is the current count (the Deployment's spec.replicas). Outside that range, it is the current number of Pods multiplied by the ratio, rounded up.⁴
3. When scaling down, does not go below the maximum computed count over the past 5 minutes. It then clamps the result to the range from minReplicas to maxReplicas.⁵
4. If the count decided in step 3 differs from the Deployment's spec.replicas on the cluster, rewrites spec.replicas to that count

The "5 minutes" in step 3 is the value used when the HPA has no behavior, and it comes from the default of kube-controller-manager's --horizontal-pod-autoscaler-downscale-stabilization.⁶ It can be changed per HPA with behavior.scaleDown.stabilizationWindowSeconds.

From the scale-up to the revert

The table below shows the syncs from the scale-up to the revert. Seconds are the time elapsed since running kubectl scale. The computed counts were taken from kube-controller-manager logs in the reproduction environment, with --v=4 (a setting that makes the logs more detailed).

Since the operation	Metrics for the 2 new Pods	Computed count	replicas
8 s	none	4	4
23 s	none	4	4
38 s	present (0%)	0	4
(same every 15 s)
323 s	present (0%)	0	2

At the 8 s and 23 s syncs, the 2 new Pods are running but have no metrics yet. metrics-server is the component that collects Pod CPU and memory usage from each node and passes it to HPA. Values for a Pod that has just started do not appear right away.

When the ratio computed from only the Pods that have metrics is below 1, HPA recomputes it, treating the Pods without metrics as 100% utilized.⁷ Here the 2 existing Pods are at 0% and the 2 new ones at 100%, so the average is 50%. The ratio to the 50% target is 1.0, so in step 2 the computed count is the current count, 4.

From the 38 s sync on, metrics for the 2 new Pods are available and the computed count becomes 0. In step 3, though, HPA does not go below 4, the maximum computed count over the past 5 minutes, so replicas stays at 4.

At the 323 s sync, the computed count of 4 from 23 s is now more than 5 minutes old, and the maximum computed count over the past 5 minutes becomes 0. HPA clamps this to minReplicas, 2, and rewrites spec.replicas to 2.

To HPA this is an ordinary scale-down, so the only Event is a Normal SuccessfulRescale, with no errors or warnings.

How long the kubectl scale count lasts

HPA takes the replicas a person wrote with kubectl scale into its calculation as the current count. When scaling down, HPA preserves only the computed counts from the past 5 minutes.

The count a person wrote lasts only until 5 minutes after the computed count last matched it. In the reproduction, the 4 Pods lasted about 5 minutes because the computed count was 4 at the syncs before the 2 new Pods had metrics, and that 4 stayed as the maximum computed count over the past 5 minutes.

If the computed count never matches that number, the first sync reverts it. In the same setup, scaling from 2 to 3 went back to 2 at the sync about 10 seconds later. Even with the 1 Pod without metrics treated as 100% utilized, the average across 3 Pods is about 33%. The ratio to the 50% target is below 0.9, and the computed count is 2.

Whether the revert comes a few minutes later or right away, HPA is what does it.

Remedies

The two places to change are the HPA's lower bound (minReplicas) and its scale-down rules (behavior.scaleDown).

Raise minReplicas

Raise the HPA's minReplicas to the count you want. replicas goes up immediately. Only the lower bound goes up, so HPA can still add more Pods if load increases.

$ kubectl patch hpa <hpa> --type=merge -p '{"spec":{"minReplicas":4}}'
horizontalpodautoscaler.autoscaling/<hpa> patched

If the count you want is above maxReplicas, raise maxReplicas in the same patch.

Disable scale-down, then scale up

Set the HPA's behavior.scaleDown.selectPolicy to Disabled, then scale up with kubectl scale. HPA no longer reduces replicas.⁸

$ kubectl patch hpa <hpa> --type=merge -p '{"spec":{"behavior":{"scaleDown":{"selectPolicy":"Disabled"}}}}'
horizontalpodautoscaler.autoscaling/<hpa> patched
$ kubectl scale deploy/<deployment> --replicas=4
deployment.apps/<deployment> scaled

Which to choose

I choose to raise minReplicas. The only thing it stops is scaling down below the lower bound. HPA keeps adding Pods as load rises and keeps scaling down as far as the lower bound. Disabling scale-down also stops HPA from removing the Pods it added in response to load.

When you no longer need the extra Pods, set minReplicas back to its original value.

---

1. The results are here, and the code is in show-your-work. ↩

2. Field management in the Kubernetes documentation. ↩

3. Events are deleted after 1 hour by default, because the default of kube-apiserver's --event-ttl is 1h0m0s. ↩

4. Algorithm details in the Kubernetes documentation. ↩

5. normalizeDesiredReplicas in kube-controller-manager. ↩

6. kube-controller-manager's defaults. ↩

7. replica_calculator.go in kube-controller-manager. ↩

8. Example: disable scale down in the Kubernetes documentation. ↩