The problem
You run a load test targeting 1,000 requests/second. The tool reports P50 = 12ms, P99 = 45ms. You ship it. In production, under the same nominal load, P99 is 400ms and customers are complaining about timeouts. Your load test wasn't wrong about the requests it measured — it was wrong about which requests it chose to measure at all.
Why it happens
Most load generators work like this: fire a request, wait for the response, record how long it took, then fire the next one. That sounds reasonable until you ask what happens when the system under test stalls — a GC pause, a lock, a slow downstream call.
Say your generator is supposed to fire one request every 1ms (1,000 QPS), but request #500 gets stuck for 200ms behind a stop-the-world GC pause. A naive closed-loop generator waits for #500 to finish before sending #501. It records one slow request (200ms) and then resumes firing at 1ms intervals as if nothing happened. In reality, requests #501 through #700 — roughly 200 of them — should have been sent during that stall and would have queued up behind it, arriving late too. But the tool never issued them during the stall, so it never has a "should have started at T, actually finished at T+200-ish" sample for any of them. It has one bad data point where there should have been hundreds.
This is coordinated omission (the term comes from Gil Tene's work on HdrHistogram): the measurement tool "coordinates" with the system's own hiccups by skipping exactly the samples that would have been worst, simply because it wasn't issuing requests during the stall. The pain doesn't disappear — it just never makes it into your report.
What to do about it
The fix is to decouple "when a request should have started" from "when it actually started," and score latency against the former:
scheduled_start = start_time + (request_number * interval)
actual_latency = response_time - scheduled_start // not - actual_send_time
Every load-testing tool that handles this correctly (wrk2, k6's arrival-rate executor, Gatling's open-loop injection) generates load on a fixed schedule regardless of whether earlier requests have finished, and measures latency against the scheduled time, not the send time. If a request was supposed to fire during a stall and got delayed, that delay counts against it. This is the difference between open-loop and closed-loop load generation, and it's rarely explained in tool documentation beyond a flag name.
Two smaller things worth checking while you're at it: make sure your histogram has enough tail resolution (linear-bucket histograms quietly truncate outliers; HdrHistogram-style log-linear buckets don't), and confirm your load generator isn't itself blocked on a single thread waiting for responses — that reintroduces the same coordination bug on the client side.
Key takeaways
- If your load generator waits for a response before sending the next request (closed-loop), it silently erases exactly the delays you most need to see.
- The gap you're missing scales with the length of the stall — a single 200ms hiccup can hide hundreds of "should-have-happened" bad samples.
- Score latency against scheduled send time, not actual send time, using an open-loop / constant-arrival-rate generator (wrk2, k6 arrival-rate, Gatling open injection).
- If your load test's P99 looks suspiciously cleaner than production's, check this before blaming hardware or "prod just being different."
Top comments (1)
"It was wrong about which requests it chose to measure at all" — that generalises well past load testing, and it names a failure I hit this month in a completely different domain.
I run a static scanner over my own code. It has an exclusion rule so a folder of deliberately-broken test fixtures isn't scored as real findings. That rule silently widened and started swallowing real source paths, and the run stayed perfectly green — because "zero findings" is trivially true when you've stopped scanning half the tree. Same structure as yours: the numbers weren't lies about the population measured, the population had quietly shrunk, and every metric downstream inherited that without a single anomaly to notice. Your closed-loop generator stops issuing during the stall; my scanner stopped visiting during the drift. Both report cleanly on what remained.
What fixed it was asserting the denominator instead of only the result: the run now has to confirm it actually visited specific sentinel files and stayed above a floor on total files scanned, and it fails if either slips. Findings alone can't distinguish "nothing wrong" from "nothing looked at" — you need a number that goes down when coverage does. Your scheduled-time scoring is the same instinct: measure against what should have happened, not against what the instrument happened to capture.