On the first morning of our spring sale the web tier went from twenty two instances to six in about four minutes, at ten past nine, with the queue of shoppers at its longest. Page loads went past ten seconds and the load balancer started returning 503s. The scaling policy noticed and put the instances back over the next quarter of an hour.
Nobody had touched the web tier. A pull request adding one IAM permission for the image resizing service, approved the day before, was merged and applied that morning so it would be in place for the sale. It lived in the same Terraform root module as the web tier. The plan at apply time had two changes: the policy, and desired_capacity on the web autoscaling group, from 22 to 6, update in place. The engineer applying it read the second line as leftover drift. The reviewer had seen the plan the previous afternoon, when the group was at six and the line did not exist.
When the group was created, somebody set desired_capacity to the minimum, because the module examples set it. From then on the scaling policy owned that number, and Terraform had no way of knowing. Every apply compared the running group with the code, found the policy's latest decision, and corrected it. On a quiet day the correction was invisible because the two numbers matched. On a busy day it was a scale in.
desired_capacity is no longer set on any group that has a scaling policy. Where a module needs a value at creation, it goes in ignore_changes, one of the very few places we still allow that after the firewall episode I wrote about earlier. The plan check that reads plans as JSON before review now treats every capacity attribute as disruptive: desired counts on groups, node counts on pools, replica counts on managed services. Those need a named approver and cannot be applied from a merge. And the apply job regenerates the plan and compares it with the one that was reviewed, stopping if a new line has appeared overnight.
Some numbers in a system belong to a controller that changes them for a living. Writing them into your code does not make them yours. It turns every apply into a vote to put them back where they were on the day you wrote the file.
– Sergey Shinder
Top comments (0)