On the first Monday of January, when our trade customers restock after the holidays, the catalogue API slowed to four seconds a page by half past eight. It was running four pods. Its HorizontalPodAutoscaler allowed up to twenty, and had not moved from four since the middle of November. kubectl get hpa showed the target as unknown out of seventy percent.
In November we moved the catalogue to a managed database and added the vendor's connection proxy to the pod as a sidecar, copied from their example manifest. The example had no resources block. The proxy worked perfectly.
The autoscaler scales on CPU utilisation, and utilisation is usage as a percentage of the request, added up over every container in the pod. The Kubernetes documentation is direct about the consequence: if any container in the pod has no request for that resource, utilisation for the pod is not defined and the autoscaler takes no action for that metric. The HPA said so in its status, a ScalingActive condition set to False with the reason FailedGetResourceMetric and the message missing request for cpu, and as a warning event every few minutes. An autoscaler that cannot measure does not fail. It keeps the size it had. Through November and December four pods had been enough, so it looked like an autoscaler with nothing to do.
We gave the proxy a CPU and memory request, the pods rolled, and within five minutes the HPA had scaled to thirteen. Pages were back under three hundred milliseconds by nine fifteen.
Every namespace now has a LimitRange with default requests, so a container nobody configured still gets one. An admission policy rejects containers with no CPU or memory request outside those defaults. Deployments with sidecars scale on a ContainerResource metric, the CPU of the application container alone, so a proxy cannot change how the service scales. And we alert on any HPA whose ScalingActive condition has been false for ten minutes, read from kube-state-metrics, as well as on any HPA sitting at its maximum.
An autoscaler is a calculation. Leave one number out of it and it stops calculating, quietly, at whatever size you happened to have.
– Sergey Shinder
Top comments (0)