Our suppliers send us stock files over SFTP, to one EC2 instance with an Elastic IP. In March a pull request added a backup tag to every instance in that Terraform root module. The next morning eleven of our fourteen suppliers could not deliver anything, and their clients all said the same thing: the host key has changed, someone could be doing something nasty.
The instance's AMI came from a data source: the most recent Amazon Linux 2023 image for our architecture, with most_recent set to true. It was written that way so the server would always be built on a patched image. A data source is read again on every plan, Amazon publishes new images of that family every few weeks, and on an aws_instance a change of ami forces replacement. So any plan of that module made after a new image came out proposed destroying the SFTP server and building another. The tag change was the module's first apply in seven months.
The plan said so, in one line among forty two: ami, forces replacement, between tag updates on fourteen instances. The reviewer read a tag change. The new instance came up in four minutes, took over the Elastic IP, and generated fresh SSH host keys at first boot, as every new instance does. Our suppliers' clients had stored the old keys and refused the new ones, which is exactly their job. The three whose clients accept any key carried on, which is a separate conversation.
We restored the old host keys from the previous night's volume snapshot that afternoon, and the files queued at the suppliers arrived by evening, seven hours late.
The AMI is no longer looked up at plan time. A scheduled job finds the newest image and opens a pull request that changes a pinned ID in a variables file, on its own, labelled as a replacement, so a new image arrives on purpose. The host keys live in Secrets Manager and the boot script writes them, so a rebuilt server keeps the identity of the old one. And the plan check I have described here before, which flags restarts and failovers, now fails any plan that replaces a resource unless the pull request was opened for that replacement.
A data source is a question asked again on every plan. Ours asked for the latest image, and the answer kept changing whether or not anyone was listening.
– Sergey Shinder
Top comments (0)