When we moved the finance reporting server to the cloud, we sized it from a year of monitoring. Its processor averaged about twelve percent and rarely passed forty, so the team chose one of the provider's burstable instance types, which is cheaper because it assumes you will be idle most of the time. For three weeks of every month that assumption was exactly right.
Month end began on the first working day of April. The consolidation jobs started at six, the finance team started running reports at eight, and by the middle of the second day a report that normally took four minutes was taking forty. Overnight jobs ran into the morning. The finance director asked me whether the cloud was simply slower than our old server room, and I could not answer her, because every graph I looked at showed the processor at twenty percent.
Twenty percent was the answer. A burstable instance earns credits while it is quiet and spends them when it is busy, and once they are gone it is held at its baseline, which for that size was a fifth of a processor. The graph was not showing a machine with room to spare. It was showing a machine pinned to its ceiling, and the ceiling looked like headroom. The provider published the credit balance the whole time, as a separate metric, and nobody had put it on a dashboard, because no server we had ever owned had such a thing.
That month's bill came in lower than forecast, which in hindsight was the clearest signal of all.
The reporting server now runs on a fixed performance instance for the last working day and first three working days of each month, and drops back afterwards on a schedule finance can see. Every remaining burstable instance alerts when its credit balance falls below forty percent. Our sizing template asks for the busiest consecutive day rather than the monthly average. Reviewing the rest of the estate found two more servers living on credits, and one database whose disk throughput had its own allowance that was used up every night by the backup.
A cloud resource can run out of things other than money. For every service we size, I now ask what it is quietly spending besides our budget, and whether anyone can see the balance.
– Serguey Shinder
Top comments (0)