Before a campaign launch we load tested the checkout path at four times expected peak. It passed comfortably: p99 under 300ms, no errors, CPU around 40 percent. On launch day the service fell over at roughly 1.4 times normal traffic, and the graph of it looked nothing like anything the test had produced.
The test was wrong in three ways, all of which flattered us. It ramped up over ten minutes, so every cache was warm and every connection pool fully established before the measurement window. It used twelve test accounts, so the same rows were read over and over and the database served them entirely from buffer cache. And it arrived at a steady rate, evenly spaced, because that is what the tool does by default.
Real traffic did none of that. It arrived as a step function when the email went out, against cold caches, spread across two hundred thousand distinct users, with a burst structure that put several hundred requests into the same 50ms window. The connection pool, sized fine for smooth load, saturated instantly. Requests queued. The queue pushed latency past the client timeout, clients retried, and the retries became the majority of the load. We were not capacity-limited. We were failing at a fraction of the throughput we had proven we could handle.
Rewriting the test taught me more than running it. No ramp, or a ramp so short it is effectively a step. A dataset large enough that cache hit rates resemble production, which for us meant generating a hundred thousand accounts and accepting the test would be slower to set up. Poisson arrivals rather than fixed spacing, so bursts occur naturally. And the client must implement the same timeout and retry policy as the real client, because retry behaviour under stress is the thing that decides whether a degradation becomes an outage.
The rerun failed at 1.6 times peak, which was the first useful number we had ever gotten out of load testing. Fixing it was mostly pool sizing, a bulkhead on the slowest downstream call, and jittered backoff in the client.
A load test that doesn't reproduce the shape of your traffic measures your test harness.
– Sergey Shinder
Top comments (0)