In May I merged a pull request that moved our tagging into a shared locals block. Twelve files, no resources added or removed, a plan reviewed by two people. Forty minutes after the apply, the order service began failing to open new database connections with password authentication failed. Existing connections were fine, which is why it took forty minutes.
The master password for that Postgres instance was generated by a random_password resource. Somebody had given it a keepers map years ago, containing the environment name and the cost centre tag, so that a password would be regenerated if the database were ever moved to a different environment. My refactor built the tags from the shared block, which spelled the cost centre in lowercase. The keepers changed, so Terraform replaced the random_password, which changed the instance's master password, which updated the secret our services read. The plan said all of that. It said it in three lines among a hundred and forty lines of tag changes, and the replacement of a random string looked like the most harmless thing in the plan.
The services read the secret once at startup. Their pools kept working with old connections until those connections reached their maximum lifetime and were recycled, at which point every new connection used a password the database no longer accepted. Pods that restarted for other reasons came back with the new password and worked, so the failure moved around the fleet and looked like a network problem for a while.
We put the old password back from the secret's version history within the hour. The lasting changes were about not having one password at all. Each service now has two database users, and rotation alternates between them: the new password goes on the user nobody is using, services pick it up gradually, and the old user is only changed on the next cycle. Rotation is a scheduled job with its own alerts, not a side effect of an apply. Generated secrets carry no keepers, and our plan check marks the replacement of any random resource as disruptive, which needs a named approver. Services also reread the secret when authentication fails, once, before giving up.
A value that only changes by accident should not be able to change during a tidy up. Ours had been waiting for anyone to touch the tags.
– Sergey Shinder
Top comments (0)