In August a customer asked for a copy of everything we held about them, and the export included phone call recordings from early 2024. Our privacy notice says recordings are deleted after ninety days. Our Terraform said the same thing, and it had said it correctly for two years.
The recordings bucket is created by our storage module, with a lifecycle rule that expires every object ninety days after it is written. We wrote it in 2022 and it worked. In early 2024 the security team rolled out a baseline module that turned on versioning for every bucket in the account, as protection against ransomware and careless deletes. It was a good change, applied to forty buckets, ours among them.
On a versioned bucket an expiration does not delete anything. S3 adds a delete marker on top of the object, the object disappears from normal listings and from the console, and the data stays underneath as a noncurrent version. A noncurrent version is kept until a rule for noncurrent versions says otherwise. Ours had no such rule, because when it was written the bucket had no versions to talk about.
So from the day versioning was turned on, every recording expired on schedule and none of them went anywhere. The console showed ninety days of files, exactly as promised. The bucket held forty one terabytes where we expected about six. The storage bill had been rising for nineteen months, and everyone had put that down to growing call volumes.
The rule now has a noncurrent version expiration of seven days, which keeps the week of protection versioning was turned on for, and a second rule cleans up the delete markers left behind. Lifecycle took about two days to work through the backlog, and our data protection officer handled the conversation with the customer. Every retention rule in the storage module now includes noncurrent versions, whether the bucket is versioned today or not, because that is decided in a different module by a different team. And a weekly job lists object versions in every bucket that declares a retention period and reports the oldest one it finds, taken from the API rather than from our code.
Two modules, each correct, written a year and a half apart. A retention rule promises that data is gone, and on a versioned bucket gone only means hidden until you also say for how long.
– Sergey Shinder
Top comments (0)