The database failover took nine minutes, which was inside the number we had promised ourselves. The status page said resolved at 09:31. Customers were still seeing order confirmations arriving ninety minutes late at three in the afternoon, and we finished draining the backlog at 15:40. The failover lasted nine minutes and the incident lasted six hours, and until that day we had only ever measured the first number.
The shape of it is arithmetic. Normal inbound is about 2,100 messages a second. Our consumers handle roughly 2,400 a second, so we had called that fourteen percent headroom and moved on. During the failover the consumers could not write, so they stopped acknowledging, and the queue grew at full inbound rate for nine minutes: about 1.1 million messages. Draining 1.1 million at 300 a second of spare capacity takes over an hour in the best case. It was not the best case, because client retries had pushed inbound above normal, because duplicates from those retries added maybe twelve percent more work, and because the freshly promoted replica had a cold cache and ran at two-thirds throughput for the first twenty minutes.
The thing I had not internalised is that headroom is not a percentage of normal load. It is a recovery rate. Fourteen percent spare capacity means your backlog drains at one seventh of the rate it accumulated, so every minute of downtime costs you seven minutes of lateness, and nothing about that ratio appears on a dashboard until you need it.
We now size consumer capacity against drain time rather than utilisation, with a stated target: any ten minute stall is cleared within thirty minutes. That meant autoscaling consumers on queue depth and oldest-message age instead of CPU, which is the only signal that actually reflects the debt. Low-value traffic — analytics events, non-critical notifications — goes to a separate queue that we are willing to shed, and during recovery we shed it. Producers got jittered backoff with a retry budget, because the retries were making the hole deeper.
And the definition of an incident changed. It ends when the backlog is drained and the oldest message is fresh, not when the error rate comes down.
– Sergey Shinder
Top comments (0)