Most of our head office systems still run as virtual machines on a cluster in our own server room, with a single storage array behind them. The array has eighteen terabytes of usable space.
When we create a virtual machine, we give it disks of the size it might one day need, and the array only uses real space as data is written. This is called thin provisioning, and it is normal and sensible. A finance server given two terabytes that holds four hundred gigabytes uses four hundred gigabytes. Over six years we had created disks totalling thirty one terabytes on an array of eighteen. Each one had been reasonable on its own. Nobody had ever added them up.
Our monitoring watched each virtual machine's disks, and every one showed plenty of free space, because each was measured against the size it had been promised. The array's own alert was set at ninety five percent full and went to the mailbox of an engineer who had moved to the network team.
On the third working day of January, in the middle of month end, the finance system wrote its year end archive while the warehouse system rebuilt its indexes, and the array filled. The hypervisor did what it is designed to do when a thin disk cannot grow. It paused every virtual machine that tried to write. Twenty six stopped at once, including payroll, the finance system and depot label printing. We freed space by deleting old copies and had them running again in about three hours. Nothing was lost, but month end slipped by two days.
We now report two numbers for the array every week: what is physically used and what has been promised. The ratio between them is a decision, written down, and we have set ours at one and a half. Any new virtual machine that would take us past it has to come with a smaller disk or more storage. The array alert now warns at seventy five percent, goes to the infrastructure team's queue rather than to a person, and at our current growth gives us about four months' notice.
Thin provisioning lets you promise more than you own, and that is fine for as long as someone is counting the promises. If nobody is, the shortfall arrives all at once, and usually on your busiest day.
– Serguey Shinder
Top comments (0)