Our workers were idle and the queue was an hour deep

The document ingestion workers had a horizontal pod autoscaler targeting seventy percent CPU, between four and thirty replicas. On the morning a customer uploaded a large batch, the queue went from a few hundred messages to fifty thousand, the oldest message age climbed past an hour, and the autoscaler sat at four replicas the entire time. It was not broken. CPU across those pods was eleven percent.

The work these pods do is call a document conversion service and wait. They are blocked on a socket for almost their whole lifetime. CPU is the one resource they will never run out of, which makes it the one signal that will never ask for more of them. We had chosen the metric because it is the default, and defaults are how most of these decisions get made.

There were two problems underneath, and only one of them was the autoscaler. Each pod processed one message at a time, so a single replica was one concurrent conversion, and thirty replicas would still have been thirty. Raising the per-pod concurrency to sixteen, which for an I/O bound worker costs almost nothing, changed the shape of the problem more than any scaling change did. That went in first and drained the backlog that morning.

The autoscaler now scales on a custom metric served through the Prometheus adapter: messages ready in the queue divided by ready replicas, with a target of thirty. That is a number I can explain to somebody. It says each worker should have about thirty items in front of it, and if it has more, add workers. We kept a CPU target as a secondary metric so a genuinely hot pod still counts, and set the scale-down stabilisation window to five minutes because the first version flapped between six and nineteen replicas every couple of minutes and we had to watch it do that for a day before believing it.

The alert changed too. We used to alert on consumer CPU and on pod restarts. Now we alert on oldest message age, because that is the thing the customer feels, and it is the only number in this story that was telling the truth the whole time.

An autoscaler is only as good as your guess about what runs out first.

– Sergey Shinder