Our cluster was full at nineteen percent CPU

A routine deploy sat in Pending at five past nine on a Monday. The event on the replica set said zero of thirty eight nodes are available, insufficient cpu. On the same screen, the cluster's CPU utilisation for the past hour was nineteen percent and memory was thirty four.

Both were true. The scheduler does not look at what a container is using. It looks at what a container has asked for, and a request is a reservation held for as long as the pod exists, whether or not a single cycle is ever spent against it. Our Helm chart's default request was one CPU and two gibibytes. That number came from the first service we containerised in 2023, which genuinely needed it, and it had been copied into every chart since by people doing the sensible thing and starting from what already worked. Two hundred and forty pods, two hundred and forty reserved CPUs, and a median actual usage of about sixty millicores per container. Ninety two percent of allocatable CPU was spoken for and almost none of it was being used.

The cluster autoscaler had been doing its job throughout, adding nodes to satisfy claims, which is how thirty eight machines ended up idling.

We took thirty days of per container usage and set each request from the ninety fifth percentile with headroom, and deliberately left limits alone, because limits are a different argument and I did not want the two mixed up in one change. A weekly job now posts a recommendation per deployment from the same data. A policy rejects any pod requesting more than five hundred millicores without an annotation giving a reason. And the chart that matters most is new: reserved against used, per node pool, which is the only view in which any of this was ever visible. Node count went to fifteen. An ingestion service that had been starved for weeks got scheduled without anyone asking for more capacity.

Utilisation is what you are spending. Allocation is what you have promised. The scheduler reads the promises, and a cluster fills up with promises long before it fills up with work.

– Sergey Shinder