Deleting one subnet from a list rebuilt six of them

The change was one line. We removed a spare subnet from a list of five in the staging account, the second entry, which had never had anything scheduled into it. The plan was long, as our plans are, and the summary at the bottom said four to change, five to destroy, six to add. Somebody read that summary at nine in the morning and reasonably concluded it was subnet sized.

Twenty five minutes after the apply the internal load balancer in staging was gone, the database could not be reached from the application tier, and three teams were asking whether staging was down.

We declared those subnets with count over the list, which gives each one an address ending in a number: private[0] through private[4]. That number is a position, not a name. Removing the second element does not remove the second subnet. It renumbers the list, so the resource at index one is now asked to hold the address block that used to belong to index two, index two takes three, and index four has nothing left to correspond to. A subnet cannot change its address block in place, so each of those became a destroy and a create, and everything attached to them, the route table associations, the load balancer's network interfaces, the database subnet group, went with them.

Terraform did exactly what we told it. The plan said all of this in full, resource by resource, above the summary that we read instead.

We moved the whole set to for_each keyed by a name, so the address is private["app_b"] and identity stops depending on order. The migration was a batch of state move commands followed by a plan that proposed nothing at all, which was the only evidence worth having. A check in CI now fails any plan file where a stateful resource uses count over a variable length list. And any plan containing a destroy prints the resource addresses of what it is destroying at the top, above the counts, because the counts are the part of a plan that cannot hurt you.

An index is a position in a list that somebody will one day edit in the middle. A name is a name.

– Sergey Shinder