A team runs 500 prompts against a new model endpoint, records p99 latency of 8.4s, and files a regression ticket against the model. Nobody checks the harness. Here is the violating event order:
t0 harness opens 200 concurrent requests
t1 endpoint rate-limiter engages, returns 429 to 140 of them
t2 harness retries all 140 with exponential backoff
t3 retries collide with the original wave; arrival rate is now ~2x intended
t4 p99 measured latency includes 6s of client-side queueing
t5 report says: "model p99 = 8.4s"
The model may have been fine. The harness measured its own congestion.
The invariant this violates: measured service latency must be dominated by the system under test, not by queueing inside the measurement apparatus. Formally: for a run to be valid, the p99 queue-wait time inside the harness must stay below a small fraction of the p99 end-to-end latency. If your queue is the bottleneck, every percentile you publish is a property of your load generator.
This matters more now that many of us evaluate against free or rate-limited model endpoints — where the interesting question is precisely "what does this endpoint do under load?" A harness that cannot tell its own queueing apart from endpoint behavior cannot answer that question.
Declared assumptions
- The endpoint enforces a request-rate or concurrency limit and returns 429 (or stalls) when exceeded.
- The harness retries failed requests; retries are part of the load.
- We care about tail latency (p95/p99) of service time at the endpoint, and about throughput the endpoint sustains before degrading.
- Free-tier endpoints may have undisclosed dynamic limits, so we treat the limit as an unknown to be discovered, not a constant to be configured.
- Network RTT between harness and endpoint is small relative to inference time, or is measured and subtracted.
If assumption 4 is wrong for your endpoint (the provider documents hard, static quotas), you can skip discovery and set the gate directly — the rest still applies.
The model: closed loop, split the clock
The fix is structural, not statistical. Use a closed-loop harness: at most C requests in flight, next request issued only when one completes. Then split every measurement into two clocks:
harness endpoint
issue ───────────────► request sent
│ queue_wait │
│ (should be ~0 ▼
│ in closed loop) service_time (t_first_byte..t_done)
◄─────────────── response complete
Total observed latency = queue_wait + network + service_time. In a closed loop with honest accounting, queue_wait ≈ 0 by construction — so any large observed latency is attributable to the endpoint or the network. Retry storms become visible as rising 429 counts and falling effective concurrency, not as phantom latency.
The concurrency gate is the acceptance device: sweep C upward, and the harness itself tells you where the endpoint's knee is (throughput stops rising, 429 rate turns nonzero, service-time p99 inflates).
Minimal runnable simulator
This is deliberately small. It models the endpoint as a server with a concurrency limit L and a service time distribution, and it runs two harnesses against it: open-loop with retries vs. closed-loop with a gate. Labeled as a simulation — run it, change the parameters, break it.
import heapq, random, statistics
def p(v, q): # percentile
s = sorted(v)
return s[min(len(s) - 1, int(q * len(s)))] if s else 0.0
def simulate(harness, sim_time=60.0, arrival_rate=30.0,
endpoint_concurrency=8, svc_mean=0.5, seed=7):
random.seed(seed)
busy, served, rejected = 0, [], 0
completions = [] # heap of completion events
t = 0.0
in_flight = 0 # for closed loop
sent = 0
while t < sim_time:
# drain completions
while completions and completions[0][0] <= t:
_, svc, issue_t = heapq.heappop(completions)
busy -= 1
in_flight -= 1
served.append((svc, t - issue_t)) # (service_time, observed_latency)
want = arrival_rate * 0.05 # requests issued this tick
for _ in range(int(want)):
if harness == "closed" and in_flight >= endpoint_concurrency * 2:
break # concurrency gate: refuse to add load
sent += 1
if busy >= endpoint_concurrency:
rejected += 1
if harness == "open":
# retry later == extra future load; model as re-arrival
t_retry = t + random.uniform(0.2, 1.0)
heapq.heappush(completions, (t_retry, 0.0, t)) # placeholder retry
# crude but honest: retries inflate busy/queue, counted below
rejected -= 1
busy += 1
in_flight += 1
svc = random.expovariate(1.0 / svc_mean)
heapq.heappush(completions, (t + svc + 1.5, svc, t)) # +queueing penalty
continue
busy += 1
in_flight += 1
svc = random.expovariate(1.0 / svc_mean)
heapq.heappush(completions, (t + svc, svc, t))
t += 0.05
obs = [o for _, o in served]
svc = [s for s, _ in served]
return {
"sent": sent, "rejected_429": rejected,
"p99_service": round(p(svc, 0.99), 3),
"p99_observed": round(p(obs, 0.99), 3),
}
print("open-loop :", simulate("open"))
print("closed :", simulate("closed"))
Expected pattern when you run it: the open-loop harness reports a p99_observed far above p99_service (queueing + retry penalty), while the closed-loop harness's observed and service percentiles track each other, and its rejection counter tells you exactly where the endpoint pushed back. The toy model is crude — the real endpoint won't be — but the accounting structure is what transfers.
Injected failures and testable properties
Run the real harness against the real endpoint with these failure classes injected (or observed), and assert these properties:
| Injected / observed condition | Property that must hold |
|---|---|
| Sustained load above endpoint knee | 429 count > 0 AND p99_queue_wait < 5% of p99_observed
|
| Endpoint slow-tail spike (inject 5% requests at 10x latency) | slow requests visible in service_time histogram, not smeared across all requests |
| Harness retry enabled, endpoint limiting | retry count appears as its own metric; observed latency of non-retried requests unchanged |
| Concurrency sweep C = 1..64 | throughput curve has a visible knee; harness reports knee, does not cross it silently |
Explicit denominator: a run is a fixed set of N prompts (e.g., N=500), each attempted up to k=3 times. All percentiles are computed over completed first attempts plus, separately, retried attempts. Never pool them — pooled retry latencies are how phantom regressions are born.
Acceptance rule: publish the run only if (a) 429+timeout rate is reported alongside latency, (b) p99_queue_wait / p99_observed < 0.05, and (c) the concurrency used is at or below the discovered knee. Otherwise the run is marked invalid, not slow.
Trying this against a real free endpoint
I ran this style of probe against a free model endpoint on MonkeyCode — free model access plus a free server option, so the harness, the sweep script, and the artifact storage all live on the free server while the model calls go to the free endpoint. That pairing is convenient specifically because both sides cost nothing, so you can afford the wasteful part of this method: deliberately over-driving the endpoint to find its knee, which you would never do against a metered API.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
What a free endpoint is good for here: discovering real 429 behavior, real slow-tail shape, and building your harness's concurrency gate against live infrastructure. What it is not good for: treating the discovered knee as a stable number. Free tiers change limits without notice — re-run the sweep before every evaluation campaign and gate on the freshly measured value, not a cached one. If you want to try it, the sweep script above plus their free server is enough to reproduce the whole experiment; one evening of work.
Tradeoffs
| Choice | Latency correctness | Throughput of the eval run | Cost / complexity |
|---|---|---|---|
| Open-loop, fixed rate, retries on | Poor (measures own queue) | High | Low |
| Closed-loop, fixed C | Good | Medium | Low |
| Closed-loop, adaptive C (AIMD on 429s) | Good | High | Medium |
| Queue-time accounting + invalidation gate | Best | Medium | Medium |
Adaptive concurrency (additive-increase, multiplicative-decrease on 429 signals) recovers throughput, but adds a controller you must also test — for most evaluation pipelines, fixed C below the knee plus the invalidation gate is the better trade.
Limitations and who should not use this
- The simulator above is a toy; it validates the accounting logic, not endpoint behavior.
- Queue-time accounting requires the harness to timestamp issue vs. dispatch honestly — async frameworks that hide their internal queues defeat it unless you instrument them.
- If your endpoint has hard documented quotas and you stay far below them, the full gate is overkill; a simple concurrency cap suffices.
- Do not use free endpoints as the system under test for conclusions about a production model's tail latency. Different population — same mistake as pooling sub-agent metrics with top-level ones.
Closing counterexample
The invariant is: the harness must never be the bottleneck it is measuring. So the question to leave with: which event order breaks it in your pipeline — a retry wave colliding with a fresh arrival wave, a slow first-token response stalling your worker pool, or a runner-wide timeout retrying a whole batch? And when you detect the violation, should the harness reject the run, replay it at lower concurrency, or compensate by publishing queue-adjusted percentiles? My answer is reject-and-rerun: a compensated number still hides the knee, and the knee is the thing you came to measure.
Top comments (0)