Every quarter we review CPU requests against what our services actually use. The numbers come from a Grafana dashboard that sums container_cpu_usage_seconds_total by pod and then by service. In the July review forty one services were using close to twice their requests at peak. We raised their requests to match, the cluster autoscaler added twelve nodes over the following week, and the compute bill for the cluster went up by about a third.
In September an engineer sizing a new service put the dashboard next to kubectl top pods. kubectl top said his pod used three hundred millicores. The dashboard said six hundred.
The metric comes from cAdvisor in the kubelet, and it does not only describe containers. Alongside a series for each container, it exposes one for the pod's own cgroup, with the same pod label and an empty container label, and that cgroup already contains every container in the pod. Summing every series for a pod adds up the containers and then adds their total again. Our dashboard query had no matcher on container at all, so every service had been shown at about twice what it really used, and the review had faithfully asked for twice the CPU. The memory panels beside it made the same mistake with container_memory_working_set_bytes.
Nobody doubted it, because the numbers were plausible. These were busy services, the peak season was coming, and the dashboard had already been used for two reviews. The autoscaler was not wrong either. It adds nodes for requests, not for usage, and the requests we had set really did need twelve more nodes.
Every query on cAdvisor metrics now filters on container!="" and image!="", and the dashboards read from recording rules defined once in the platform repository, instead of each panel writing its own sum. A nightly job compares those rules with metrics-server for a sample of pods and posts any pod where the two disagree by more than ten percent. We brought the requests back down over two weeks, and the autoscaler removed nine of the twelve nodes.
The dashboard added up everything the kubelet reported about a pod, and the kubelet reports a pod both in parts and as a whole.
– Sergey Shinder
Top comments (0)