In June a colleague spent a quiet Friday removing resources nothing referenced. Old roles, empty log groups, a couple of queues, and one KMS key whose alias looked like a leftover from a migration. The plan was clean, the review was quick, the apply printed a scheduled deletion date thirty days out, and everyone went home pleased. Twenty three days later a routine restore test failed with a message about a key that was pending deletion.
The key was referenced by nothing in our configuration, which was the reason it was chosen. It was referenced by something outside it. Our backup service, configured in a console two years earlier, encrypted every database snapshot copy it sent to the recovery account with that key. Terraform knew the key existed. It had no way to know who was using it, and grep across four repositories agreed.
KMS does not delete a key immediately. It schedules it, for between seven and thirty days, and until the date arrives you can cancel. Our module used the maximum, which somebody had set once without much thought and which is the only reason this is a story rather than a loss of every off site backup older than a month. We cancelled the deletion with seven days to spare.
A resource being unreferenced in code is a fact about your code. Whether it is unused is a fact about the world, and only the world can tell you. So removals of anything holding or protecting data now go through a separate job that asks: for a key, the audit log for any decrypt or generate data key call in the last ninety days, and which principals made them; for a role, when it was last assumed; for a queue, whether anything has sent to it. A single hit blocks the pull request and names the caller. Keys and snapshots carry prevent_destroy regardless. And any ScheduleKeyDeletion event pages the platform channel, because a thirty day window is only safety if somebody knows it has started.
Deleting what your configuration does not mention assumes your configuration is the whole list of users. For encryption keys in particular, it rarely is.
– Sergey Shinder
Top comments (0)