The first day of our annual promotion went well for about six hours. Order volume was three times normal, the database was comfortably quick, and I remember saying at the midday check in that we had over provisioned. At twenty past three, write latency went from four milliseconds to over two hundred, and it stayed there until nearly midnight.
Nothing had changed at twenty past three. The database volume was a general purpose disk whose baseline throughput scales with its size, and above that baseline it can burst using a balance of credits that refills when the disk is quiet. On a normal day we used a fraction of the baseline, so the balance was always full and the burst performance was all we had ever seen. On promotion day we ran above baseline from nine in the morning, the balance drained steadily, and when it reached zero the disk dropped to the performance we had actually paid for.
The balance was a published metric. We were not graphing it. Our capacity model was built from observed latency and throughput on normal days, which faithfully measured a performance level that was only temporary, and our load test ran for forty minutes, which a full credit bucket covers with room to spare.
That day cost us roughly a fifth of the orders we had forecast for the afternoon and evening, mostly as timeouts at checkout. The following morning we moved the volume to provisioned throughput, which took a few minutes online and cost less per month than the redelivery credits we gave away that week.
Afterwards I went looking for other borrowed capacity and found three more: burstable instance classes on two of our internal services, which use a similar CPU credit scheme, and a managed cache whose network allowance has a burst above its baseline. Every one of them now has its balance on the service dashboard and an alert when it falls below half. Capacity planning uses the baseline figure only, and anything we need to hold for a day is tested at that load for longer than the burst could possibly last.
Some of the performance you measure is lent to you. It looks exactly like the performance you own, right up to the moment the loan is called in, which is usually the busiest day of your year.
– Sergey Shinder
Top comments (0)