Last year we finished moving the last of our older applications off virtual machines in the cloud and onto managed services. As each application moved, its machines were shut down and, a few weeks later, deleted. By the summer about two hundred had gone, and we expected the compute bill to fall by around a third.
It fell by much less than that. When finance asked why, one of my engineers sorted the storage lines in the bill by age and found the answer within an hour. With the provider and the tools we use, deleting a virtual machine does not by default delete the disks attached to it. Our decommissioning script removed the machine and nothing else. Two hundred machines had left behind a little over four hundred disks, most of them on the faster storage tier, all attached to nothing and billed every month.
There was more beside them. Before any significant change we take a snapshot of each disk, so we can roll back. Nobody had ever written down when a snapshot could be removed, so none had been. We held about nineteen hundred, the oldest from 2021, several of them copies of copies. There were also network interfaces and public addresses reserved for machines that no longer existed. Together they cost a little over seven thousand pounds a month.
Nobody had been careless. Each item was created on purpose, by a script or a person doing their job. What we lacked was any idea of a machine as a set of parts. We had deleted the part with a name and kept the parts with identifiers.
Every resource we create now carries the name of the service it belongs to as a tag, and decommissioning removes everything with that tag, not just the machine. A weekly report lists every disk, snapshot and address that is attached to nothing, with its cost and owner, and anything unclaimed after thirty days is removed after a final snapshot is moved to cheap archive storage for ninety days. Snapshots taken before a change expire after a month unless someone labels them to keep.
The clean up took six weeks and the saving appeared in the next bill.
Switching something off in the cloud is a list of deletions, not one. Until the last item on the list is gone, the bill still believes the machine is there.
– Serguey Shinder
Top comments (0)