Our integration hub is one EC2 instance in a private subnet. It runs the small services that exchange files and API calls with our carriers, and Terraform builds it with a user data script that installs packages, fetches certificates and writes the configuration for each service.
In September one carrier moved its API behind a certificate issued by a new private authority and sent us the root to trust. The engineer handling it did the reasonable thing for a server defined in code. He added two lines to the user data script, one to download the new root and one to run update-ca-certificates, and opened a pull request. The plan showed a single change, user_data updated in place. It was approved and applied at eleven on a Tuesday morning.
Updated in place did not mean what either of us read it to mean. AWS only lets you change an instance's user data while the instance is stopped, so for this change the AWS provider stops the instance, writes the new user data and starts it again. Every carrier integration on the hub was down for about four minutes, in the middle of the morning collection run.
Then the instance came back and nothing in the new script ran. Cloud-init runs user data scripts once per instance, and it recognises an instance by its ID. A stop and start keeps the ID, so cloud-init saw a machine it had already set up and skipped the scripts entirely. The new root was never installed. Calls to that carrier kept failing for two more days, while the ticket said fixed, until someone ran the two lines by hand.
So the change caused an outage, fixed nothing, and was described by the plan as a minor update.
We decided what user data is for. On the hub it now does only what has to happen at first boot, and everything after that, certificates included, is applied by Ansible, which we can run again at any time and which reports what it changed. Instances we treat as immutable set user_data_replace_on_change to true, so a change builds a new instance that really does run the script. Any plan that touches user_data on anything else needs a second reviewer.
User data runs when a machine is first built. Changing it afterwards costs a restart and does nothing to the machine you already have.
– Sergey Shinder
Top comments (0)