A television advert went out at twenty to eight on a Sunday evening. Traffic to the storefront went up about four times in two minutes, which is roughly what marketing had told us, and which we had load tested happily a fortnight earlier. We served errors for six minutes anyway.
Everything did what it was configured to do. The horizontal autoscaler asked for pods, the cluster had no room, the cluster autoscaler added nodes, and a node joined and reported ready seventy seconds after the decision. Then nothing happened for another three minutes, because our image is 1.9 gigabytes and every one of those nodes had to pull it. Twelve nodes pulling at once from the same registry made it worse than that; the slowest pull we measured that evening was four minutes and twenty seconds. By the time capacity existed, the spike had done its damage and was easing off on its own.
The image was that size for the dullest reason available. It was a single stage Dockerfile: the JDK, Maven, the full dependency cache and a Node toolchain used only to build the front end assets, all present in the runtime image, all in one enormous layer that shared nothing with anything else we ran.
A multi stage build took it to 210 megabytes, which is most of the story. Every service now builds on the same small runtime base, so a new node pulls that base once and the per service layer is tens of megabytes. A small DaemonSet pre pulls the current and previous image of the ten busiest services onto any node that joins, so the pull usually happens before the scheduler needs it. And we moved the scaling threshold down, because the correct threshold depends on how long you take to arrive, not on how loaded you are when you decide.
The number we now watch is the one nobody had: time from the scaling decision to the new pod serving its first request. It was four minutes and forty seconds. It is one minute and ten.
An autoscaler decides in seconds and is judged on minutes it does not control. Elasticity is the sum of every step between the decision and the first request served, and almost all of those steps belong to you.
– Sergey Shinder
Top comments (0)