Your load test reports a p99 latency of 120 ms.
Everything looks healthy.
You increase traffic, run the test again, and the results still look reasonable.
But after deploying the same service to production, users start reporting requests that take several seconds.
What went wrong?
One possibility is that your load test never measured the worst part of the system's behavior.
This problem is known as coordinated omission.
It happens when a load generator unintentionally reduces its request rate while the system under test is slowing down. The slowest periods receive fewer measurements precisely because the generator is waiting for previous requests to finish.
The result is a misleading latency distribution.
This is particularly important when evaluating backend services where occasional pauses, dependency slowdowns, garbage collection, and traffic bursts can significantly affect tail latency.
In this article, we'll explore:
- What coordinated omission is and why it happens
- Closed-loop versus open-loop load testing
- Why average latency and percentiles can be misleading
- How measurement intervals affect latency distributions
- Building a simple open-loop load generator in Go
- Handling generator saturation and dropped requests
- Using HDR Histogram and latency correction
- Designing more reliable performance tests
The objective is simple: measure the latency users would actually experience, not just the latency your load generator happens to observe.
1. What Is Coordinated Omission?
Imagine a service normally processes requests in 20 milliseconds.
Your load generator sends one request, waits for the response, and then sends another.
Everything looks fine.
Now imagine the service experiences a temporary 2-second pause.
During those two seconds, the load generator is also waiting for the previous request to finish.
It doesn't send new requests.
When the service recovers, the load generator resumes sending requests.
Your test may record only one extremely slow request, even though a real production system could have received hundreds or thousands of requests during that same period.
This is coordinated omission.
The load generator and the system under test are effectively slowing down together.
Consider this example:
Normal operation:
Request 1 -> 20 ms
Request 2 -> 20 ms
Request 3 -> 20 ms
Request 4 -> 20 ms
Service slowdown:
Request 5 -> 2,000 ms
After recovery:
Request 6 -> 20 ms
Request 7 -> 20 ms
Request 8 -> 20 ms
A closed-loop load generator waits for Request 5 to finish before sending Request 6.
But in production, incoming traffic doesn't necessarily stop just because one request is taking longer.
During the 2-second pause, new requests may continue arriving and accumulate.
The load test therefore misses many of the requests that would have experienced the slowdown.
The problem isn't necessarily that the measured request duration is wrong. The problem is that the test failed to generate and measure requests that should have existed.
2. Closed-Loop vs. Open-Loop Load Testing
Understanding coordinated omission starts with understanding how load is generated.
There are two common approaches.
Closed-loop load generation
In a closed-loop model, the client waits for a response before sending its next request.
Client
|
v
Send Request
|
v
Wait for Response
|
v
Send Next Request
|
v
Wait for Response
For example:
for {
start := time.Now()
err := makeRequest(ctx)
duration := time.Since(start)
recordLatency(duration)
if err != nil {
recordError(err)
}
}
This is simple and useful for certain workloads.
However, the request rate depends on response time.
If each request takes 20 ms, one client can generate approximately 50 requests per second.
If the service slows down to 200 ms, that same client generates only approximately 5 requests per second.
The offered load decreases as the service becomes slower.
That's the core problem.
Closed-loop testing is not inherently incorrect. It models workloads where clients genuinely wait for one operation to finish before starting another.
But it can be misleading when used to represent independently arriving production traffic.
Open-loop load generation
In an open-loop model, requests are scheduled independently of previous responses.
For example:
Scheduled arrivals:
0 ms -> Request 1
10 ms -> Request 2
20 ms -> Request 3
30 ms -> Request 4
40 ms -> Request 5
50 ms -> Request 6
The arrival schedule continues even when earlier requests haven't completed.
Load Generator
|
+---- Request 1 -----------+
| |
+---- Request 2 ------+ |
| | |
+---- Request 3 --+ | |
| | | |
v v v v
Service Under Load
This better represents systems where incoming traffic is independent of response completion.
For example:
- Public HTTP APIs
- Event ingestion services
- Message consumers
- Webhook receivers
- Market data ingestion
- Background processing pipelines
The important difference is that the arrival rate remains independent of the system's response time.
This makes open-loop testing particularly useful for identifying saturation behavior and tail-latency problems.
3. A Simple Example of Measurement Bias
Let's compare two load generators.
Assume:
- Normal response time: 20 ms
- Temporary service pause: 2 seconds
- Target arrival rate: 100 requests/second
A closed-loop generator with one outstanding request behaves differently from an open-loop generator.
Closed-loop behavior
The generator sends a request and waits.
When the service pauses, the next request cannot be sent until the previous one completes.
The generator may observe:
Request 1: 20 ms
Request 2: 20 ms
Request 3: 20 ms
Request 4: 2,000 ms
Request 5: 20 ms
Request 6: 20 ms
Only a small number of requests represent the slowdown.
Open-loop behavior
At 100 requests per second, new requests are scheduled every 10 milliseconds.
During a 2-second pause, approximately 200 requests would arrive.
Those requests may experience substantial waiting time, depending on the system's queueing and processing behavior.
A simplified view:
Scheduled Request Completion
--------------------------------
0 ms 2,000 ms
10 ms 2,010 ms
20 ms 2,020 ms
30 ms 2,030 ms
...
1,990 ms 3,990 ms
The actual completion pattern depends on service capacity and scheduling, but the key point is that many requests are affected.
A closed-loop test may report one slow request.
An open-loop test can reveal a much larger group of requests experiencing elevated latency.
This is why the two approaches can produce dramatically different p99 measurements.
4. Why Percentiles Can Be Misleading
Performance reports often focus on:
- Average latency
- p50
- p95
- p99
- p99.9
These metrics are useful, but only if the underlying sample represents the workload being evaluated.
Consider this hypothetical result:
| Metric | Closed-loop test | Open-loop test |
|---|---|---|
| Average latency | 25 ms | 350 ms |
| p50 | 20 ms | 300 ms |
| p95 | 20 ms | 1,500 ms |
| p99 | 2,000 ms | 1,950 ms |
| Recorded requests | 1,000 | 10,000 |
The numbers are illustrative, not a universal outcome.
Notice something interesting: p99 might look relatively similar, even though the average and p95 are dramatically different.
Why?
Because the open-loop test captures more requests affected by the slowdown.
The important lesson is that a percentile is not meaningful independently of its sampling process.
A p99 calculated from a biased sample is still a biased p99.
The importance of the measurement window
Suppose your service experiences a 2-second pause once every minute.
If your load generator stops sending requests during that pause, your measurement window may contain very few samples from the affected period.
Even a large total request count doesn't guarantee representative sampling.
Always ask:
- How were requests scheduled?
- Did the generator wait for previous responses?
- Were requests skipped when the system slowed down?
- Was latency measured from scheduled arrival or actual transmission?
- Did the load generator itself become saturated?
- Were errors and dropped requests included in the report?
These questions are just as important as the latency numbers themselves.
5. Scheduled Time vs. Actual Start Time
One of the most important details in an open-loop load test is the starting point of latency measurement.
Consider a request scheduled for 100 ms.
But because the load generator is busy, it actually starts sending the request at 180 ms.
There are now two possible measurements.
Actual request duration:
Response time - Actual start time
This measures the time spent after the request begins transmission.
Scheduled end-to-end latency:
Response time - Scheduled arrival time
This includes the delay between the intended arrival and the actual request start.
For example:
Scheduled arrival: 100 ms
Actual start: 180 ms
Response: 250 ms
Actual request duration: 70 ms
Scheduled end-to-end latency: 150 ms
If we record only 70 ms, we hide 80 ms of delay caused by the load generator.
If the goal is to evaluate the system from the perspective of an independently arriving request, the scheduled arrival time is the more meaningful reference point.
However, there is an important distinction:
- If the generator is intentionally queueing requests before transmission, scheduled-to-response latency captures that queueing delay.
- If the generator cannot schedule or execute requests because it is saturated, the test may no longer represent the intended workload.
You should record generator delay separately instead of hiding it inside one latency number.
A useful set of measurements includes:
Scheduled arrival
Actual request start
Response completion
From these timestamps, we can calculate:
Generator scheduling delay = Actual start - Scheduled arrival
Request duration = Response completion - Actual start
Scheduled end-to-end latency = Response completion - Scheduled arrival
This makes it easier to distinguish system performance from load-generator limitations.
6. Building an Open-Loop Load Generator in Go
Let's implement a small open-loop load generator.
The objective is to schedule requests at a fixed interval without waiting for previous requests to complete.
We'll also enforce a maximum number of concurrent requests.
Most importantly, when the generator cannot launch a scheduled request, we'll count it as dropped instead of silently skipping it.
package main
import (
"context"
"errors"
"sync"
"sync/atomic"
"time"
)
var ErrConcurrencyLimit = errors.New("concurrency limit reached")
type RequestFunc func(context.Context) error
type Stats struct {
Scheduled atomic.Int64
Started atomic.Int64
Completed atomic.Int64
Dropped atomic.Int64
Failed atomic.Int64
}
type Result struct {
ScheduledAt time.Time
StartedAt time.Time
FinishedAt time.Time
Err error
}
type LoadGenerator struct {
interval time.Duration
maxInFlight int
request RequestFunc
stats Stats
results chan Result
}
func NewLoadGenerator(
interval time.Duration,
maxInFlight int,
request RequestFunc,
) (*LoadGenerator, error) {
if interval <= 0 {
return nil, errors.New("interval must be greater than zero")
}
if maxInFlight <= 0 {
return nil, errors.New("maxInFlight must be greater than zero")
}
if request == nil {
return nil, errors.New("request function cannot be nil")
}
return &LoadGenerator{
interval: interval,
maxInFlight: maxInFlight,
request: request,
results: make(chan Result, maxInFlight),
}, nil
}
func (g *LoadGenerator) Run(
ctx context.Context,
duration time.Duration,
) error {
if duration <= 0 {
return errors.New("duration must be greater than zero")
}
startedAt := time.Now()
deadline := startedAt.Add(duration)
semaphore := make(chan struct{}, g.maxInFlight)
var wg sync.WaitGroup
defer func() {
wg.Wait()
close(g.results)
}()
for scheduledAt := startedAt; scheduledAt.Before(deadline); scheduledAt = scheduledAt.Add(g.interval) {
wait := time.Until(scheduledAt)
if wait > 0 {
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
if !timer.Stop() {
<-timer.C
}
return ctx.Err()
case <-timer.C:
}
}
if err := ctx.Err(); err != nil {
return err
}
g.stats.Scheduled.Add(1)
select {
case semaphore <- struct{}{}:
wg.Add(1)
go func(scheduled time.Time) {
defer wg.Done()
defer func() { <-semaphore }()
actualStart := time.Now()
g.stats.Started.Add(1)
err := g.request(ctx)
finished := time.Now()
if err != nil {
g.stats.Failed.Add(1)
}
g.stats.Completed.Add(1)
g.results <- Result{
ScheduledAt: scheduled,
StartedAt: actualStart,
FinishedAt: finished,
Err: err,
}
}(scheduledAt)
default:
g.stats.Dropped.Add(1)
}
}
return nil
}
This example demonstrates several important ideas:
- Requests are scheduled independently of previous responses.
- The generator uses a fixed arrival interval.
- Maximum in-flight concurrency is bounded.
- Requests that cannot be started are counted as dropped.
- Scheduled, actual start, and completion timestamps are preserved.
- Request execution is measured separately from scheduling delay.
The implementation is intentionally a small demonstration rather than a complete benchmarking framework.
Understanding the request interval
Suppose we want to generate 100 requests per second.
The interval is:
interval := 10 * time.Millisecond
Because:
1 second / 100 requests = 10 milliseconds
We can initialize the generator:
generator, err := NewLoadGenerator(
10*time.Millisecond,
200,
func(ctx context.Context) error {
return makeRequest(ctx)
},
)
if err != nil {
panic(err)
}
ctx, cancel := context.WithTimeout(
context.Background(),
30*time.Second,
)
defer cancel()
if err := generator.Run(ctx, 30*time.Second); err != nil {
// Handle cancellation or test failure.
}
With this configuration:
- Target arrival rate: 100 requests/second
- Maximum in-flight requests: 200
- Test duration: 30 seconds
The maximum concurrency is a safety boundary for the generator.
It is not a guarantee that the target arrival rate will be achieved.
If requests take too long, the concurrency limit may be reached and scheduled requests will be dropped.
That's an important result, not something to hide.
A limitation of this example
The example uses one scheduling goroutine and launches a goroutine for each accepted request.
It is suitable for demonstrating open-loop scheduling, but a production-grade load generator needs additional protections:
- A dedicated result collector
- A non-blocking or carefully sized result pipeline
- Explicit generator CPU and memory monitoring
- Better cancellation and shutdown behavior
- Histograms for latency measurements
- Accurate reporting of missed schedules
- Protection against excessive goroutine creation
- Separate reporting of generator delay and service latency
In particular, if the result channel fills and no consumer is reading from it, request goroutines can block while reporting results.
A real implementation should run a dedicated collector concurrently with request generation.
The test harness itself must never become the hidden bottleneck.
7. Coordinated Omission Correction
Open-loop load generation is one way to reduce coordinated omission.
Another approach is to correct latency measurements for the expected arrival interval.
This technique is commonly associated with HDR Histogram.
Imagine:
Observed request latency: 1,000 ms
Expected arrival interval: 10 ms
If the generator was waiting for a previous response, many expected requests may not have been recorded.
A coordinated-omission-aware histogram can account for those missing observations by recording additional latency values at intervals related to the expected request cadence.
Conceptually:
Observed latency: 1,000 ms
Expected interval: 10 ms
Additional observations represent:
990 ms
980 ms
970 ms
...
10 ms
The exact recording behavior depends on the histogram API and correction method.
The important idea is that a long pause should not be represented by only one sample if the test model implies that many requests would have arrived during that period.
HDR Histogram in Go
HDR Histogram is useful for recording latency distributions with a wide dynamic range.
A Go implementation is available through:
A basic example:
package main
import (
"fmt"
"time"
hdrhistogram "github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
histogram := hdrhistogram.New(
1,
60_000_000,
3,
)
latencies := []time.Duration{
10 * time.Millisecond,
20 * time.Millisecond,
30 * time.Millisecond,
100 * time.Millisecond,
2 * time.Second,
}
for _, latency := range latencies {
histogram.RecordValue(
latency.Microseconds(),
)
}
fmt.Printf(
"p50: %d µs\n",
histogram.ValueAtQuantile(50),
)
fmt.Printf(
"p95: %d µs\n",
histogram.ValueAtQuantile(95),
)
fmt.Printf(
"p99: %d µs\n",
histogram.ValueAtQuantile(99),
)
}
This example records observed values.
It does not automatically correct coordinated omission.
Correction requires the expected arrival interval and an appropriate recording strategy.
For example, when using a histogram API that supports it, a corrected recording operation can account for the expected interval:
histogram.RecordCorrectedValue(
latency.Microseconds(),
expectedInterval.Microseconds(),
)
Always check the library version and API you're using before integrating this into a test harness.
Correction is not a replacement for a good load model
There is an important limitation.
Coordinated omission correction estimates observations that were not recorded based on an assumed arrival interval.
It cannot reconstruct every detail of a real production traffic pattern.
If production traffic is bursty, user-driven, or has complex dependencies, a constant-rate correction may not accurately represent it.
Use correction as a measurement technique, not as a substitute for realistic workload generation.
8. The Load Generator Can Become the Bottleneck
There's another problem worth discussing.
An open-loop generator is only useful if it can actually generate the intended workload.
Imagine the target is 50,000 requests per second.
But the load generator runs out of CPU, network bandwidth, file descriptors, or available connections.
Your test may report low throughput and high latency.
Is the service saturated?
Or is the generator saturated?
Without separate measurements, it's difficult to know.
A load generator should monitor:
| Metric | Why it matters |
|---|---|
| Scheduled requests | Intended workload |
| Started requests | Requests actually transmitted |
| Completed requests | Finished requests |
| Dropped requests | Requests not started |
| In-flight requests | Current generator concurrency |
| Generator scheduling delay | Delay before request execution |
| Generator CPU usage | Local resource pressure |
| Generator memory usage | Potential resource exhaustion |
| Network utilization | Possible bandwidth bottleneck |
For distributed load testing, multiple generator instances may be required.
But adding more generators introduces another consideration: their clocks and scheduling behavior should be sufficiently consistent for the measurements you're trying to make.
Don't assume that a test generating 50,000 requests per second is actually doing so just because that was the configured target.
Measure the achieved arrival rate.
9. Designing a Reliable Load Test
A good load test should answer a specific question.
Instead of simply asking:
"How many requests per second can this service handle?"
Ask:
"What happens to latency and error rates when the service receives 2,000 independently arriving requests per second?"
A practical testing strategy can include several phases.
Phase 1: Baseline
Run the service under low traffic.
Measure:
- Average latency
- p50, p95, p99
- Throughput
- Error rate
- Resource utilization
This establishes a baseline.
Phase 2: Gradual ramp-up
Increase the arrival rate gradually.
For example:
500 req/s
1,000 req/s
1,500 req/s
2,000 req/s
2,500 req/s
Observe where latency begins increasing disproportionately.
Phase 3: Sustained load
Hold the target arrival rate for a meaningful period.
This helps identify:
- Queue accumulation
- Memory growth
- Connection pool saturation
- Garbage collection effects
- Resource exhaustion
Phase 4: Burst traffic
Send short bursts above the normal arrival rate.
Observe how quickly the system recovers.
Phase 5: Dependency slowdown
Introduce latency into a downstream dependency.
Measure whether the application maintains predictable behavior.
Phase 6: Recovery
Return the system to normal traffic.
Check whether queue depth, latency, and resource usage return to baseline.
The recovery phase is important.
A service that survives a traffic spike but takes several minutes to recover may still have a serious resilience problem.
10. A Practical Checklist
Before trusting a performance test, ask:
- Does the load model represent the real production traffic pattern?
- Is the arrival rate independent of response completion where appropriate?
- Are requests scheduled at known intervals?
- Is latency measured from scheduled arrival time when required?
- Are generator scheduling delays recorded separately?
- Are dropped or missed requests visible?
- Is the achieved arrival rate close to the configured target?
- Is the load generator itself monitored?
- Are p95, p99, and p99.9 calculated from representative samples?
- Is coordinated omission correction appropriate for the workload?
- Have we tested both steady-state and burst traffic?
- Do we understand how the system behaves during recovery?
If you cannot answer these questions, your latency report may be less reliable than it looks.
Conclusion
Performance testing isn't just about sending requests and measuring how long they take.
It's about generating the right workload and ensuring that the measurement process doesn't hide the behavior we're trying to understand.
Coordinated omission is a good example of how a seemingly reasonable test can produce misleading results.
A closed-loop load generator may reduce its request rate when the service slows down, making the system appear healthier than it would under independently arriving production traffic.
Open-loop scheduling, accurate timestamps, generator monitoring, and appropriate histogram techniques help address this problem.
For Go engineers, the implementation details matter:
- Schedule arrivals independently of previous responses.
- Measure scheduled time and actual start time separately.
- Bound concurrency without silently hiding dropped work.
- Monitor the load generator as carefully as the service.
- Use latency histograms that reflect the workload being tested.
A performance test should reveal the system's limitations, not adapt to them.
If your load generator slows down whenever your service slows down, you may be measuring the generator's patience rather than your system's performance.
Top comments (0)