A Noise Budget for Free Model Evals: Live vs. Replay

When a free coding model returns a slow or timed-out response, it is tempting to say the model failed. Often the failure is somewhere in the path between your runner and the model: DNS, TLS, an overloaded gateway, or a shared free tier that is busy. If your only metric is wall-clock success or failure, you cannot tell whether you are judging the model or the transport it happens to be sitting behind.

The examples in this article use MonkeyCode's free model access for the live channel and its free server option for the replay fixture. The method itself works with any HTTP model endpoint and any small server you control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a single latency number lies

An end-to-end model call is not one system. It is at least four:

• your local runner and HTTP client;
• DNS, TLS, and the network path;
• the provider gateway, authentication, and request queue;
• model inference and response tokenization.

When a free endpoint is slow or returns a 5xx, any one of those layers could be responsible. A model can look bad simply because the free tier is crowded at that moment, or because a retry is hitting a cold instance. Conversely, a model can look too good if you run it only at off-peak times.

A noise budget makes that explicit. Treat the observed time as:

total = transport_noise + queue_noise + inference_time

You cannot measure those parts perfectly from a client, but you can estimate the first two by sending the same request to a replay fixture that performs no model work. Any remaining gap between live and replay is the part worth investigating.

The live/replay method

Run every test request twice:

1. Live channel: send the prompt to the free model endpoint.
2. Replay channel: send a small fixture request to a control server that returns the same expected bytes after a fixed delay.

Interleave the channels instead of running all live requests first and all replay requests later. Free tiers can change from minute to minute, so sequential runs may compare two different network conditions.

For a coding model, use a deterministic task. A good choice is one where the correct output is tiny and stable, such as:

Return only the string ok.

That gives you a stable response shape for the replay fixture and lets you focus on timing rather than correctness. This is a diagnostic, not a complete quality benchmark.

Replay fixture

The fixture does not need a framework. This Python standard-library server accepts a POST, waits a fixed number of milliseconds, and returns the expected string:

from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import time

class ReplayHandler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        raw = self.rfile.read(length) if length else b'{}'
        payload = json.loads(raw)
        time.sleep(float(payload.get('delay_ms', 0)) / 1000)
        body = json.dumps({'text': payload.get('expected', '')}).encode()
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == '__main__':
    HTTPServer(('0.0.0.0', 8080), ReplayHandler).serve_forever()

Run it locally, or run the same fixture on the free server option if that fits your stack. The important part is that the replay response has the same approximate size and shape as the model response, so transfer time does not distort the comparison.

Sampling script

The script below records time-to-first-byte, total time, status, and response size for both channels. It catches transport errors instead of letting one failed request stop the run.

import math
import time
from dataclasses import dataclass

import httpx

@dataclass
class Sample:
    channel: str
    status: int
    ttfb_ms: float
    total_ms: float
    body_bytes: int
    error: str = ''

def sample(client, channel, url, headers, payload):
    start = time.perf_counter()
    try:
        with client.stream('POST', url, headers=headers, json=payload) as response:
            ttfb = (time.perf_counter() - start) * 1000
            body = response.read()
            total = (time.perf_counter() - start) * 1000
        return Sample(channel, response.status_code, ttfb, total, len(body))
    except Exception as exc:
        return Sample(channel, -1, 0.0, (time.perf_counter() - start) * 1000, 0, exc.__class__.__name__)

def percentile(values, p):
    if not values:
        return 0.0
    ordered = sorted(values)
    k = (len(ordered) - 1) * p
    f = int(math.floor(k))
    c = int(math.ceil(k))
    if f == c:
        return ordered[f]
    return ordered[f] + (ordered[c] - ordered[f]) * (k - f)

def summarize(samples):
    ok = [s for s in samples if s.error == '' and s.status < 500]
    total = [s.total_ms for s in ok]
    ttfb = [s.ttfb_ms for s in ok]
    errors = [s for s in samples if s.error != '' or s.status >= 500]
    if not total:
        return 'No successful samples'
    return {
        'n': len(ok),
        'errors': len(errors),
        'ttfb_p50': round(percentile(ttfb, 0.50), 1),
        'ttfb_p90': round(percentile(ttfb, 0.90), 1),
        'total_p50': round(percentile(total, 0.50), 1),
        'total_p90': round(percentile(total, 0.90), 1),
    }

def run_plan(live_url, replay_url, live_payload, replay_payload, headers, n=20):
    live, replay = [], []
    with httpx.Client(timeout=30.0) as client:
        for i in range(n):
            if i % 2 == 0:
                live.append(sample(client, 'live', live_url, headers, live_payload))
                replay.append(sample(client, 'replay', replay_url, headers, replay_payload))
            else:
                replay.append(sample(client, 'replay', replay_url, headers, replay_payload))
                live.append(sample(client, 'live', live_url, headers, live_payload))
    return live, replay

Adapt live_payload to your provider's request schema. The replay payload only needs the fields understood by the fixture:

live_payload = {'messages': [{'role': 'user', 'content': 'Return only the string ok.'}]}
replay_payload = {'expected': 'ok', 'delay_ms': 50}
headers = {'Content-Type': 'application/json'}

live, replay = run_plan(LIVE_URL, REPLAY_URL, live_payload, replay_payload, headers)
print(summarize(live))
print(summarize(replay))

Reading the results

Do not compare one average against another. Compare error count and p90 first, because a few slow requests can make a model look unreliable even when the median is fine.

Replay channel	Live channel	Interpretation
p90 high	p90 high	Transport or shared infrastructure dominates; do not blame the model yet.
p90 low	p90 high	Queueing or inference is adding time; check token count and retry behavior.
replay errors > 0	any live result	Your runner, fixture, or network path is failing; the model was not tested.
live errors > replay errors	replay clean	The live endpoint or request shape deserves attention before scoring model quality.

If both channels are highly variable, increase the sample size or move the run to a different network. One run of 20 paired requests is a diagnostic, not a formal benchmark.

Limitations

This technique isolates transport effects, but it does not score model correctness. You still need a rubric that checks whether the generated code compiles, passes tests, or follows instructions.

The replay response must match the live response size and content type. If the model returns 4,096 tokens and the fixture returns 3, the transfer time tells you little about inference overhead.

A free tier is not a fixed environment. Quota, routing, cold starts, and load can change between runs. Use the same comparison window whenever possible, and do not turn a single quiet-evening result into a permanent performance claim.

Finally, respect provider terms and rate limits. This method should help you read your own measurements, not help you hammer an endpoint or evade a quota.

Who should not use this

If your provider already exposes server-side timings, use those. If you need high-precision p99 latency for a production decision, a client-side 20-sample run is not enough. And if your task is highly non-deterministic, live/replay timing may just show normal model variance.

The method is most useful when you are evaluating a free coding model from outside the provider and need to know whether a bad result came from the model or the path to it. Adding a replay control is a cheap way to separate those two stories.