48-Hour Field Notes: The Benchmark Timed My Laptop, Not the API

Have you ever trusted a latency number because the script printed a tight average and never raised? I did that for almost two days, and the number was measuring my notebook, not the service. The chart looked calm, the p95 looked boring, and I was ready to call the endpoint healthy. Then I ran the same file on a quiet remote box, and the story fell apart in minutes.

What I thought I was measuring

I needed a small HTTP check for a JSON endpoint that background workers hit on every job. Local Wi-Fi felt fine, so I asked an assistant for a timing loop with time.perf_counter and a session. The first draft reused one requests.Session, which looked careful until it hid DNS and TLS after call one. Was I timing the API, or was I timing a warm socket on a laptop that already liked the host?

The assistant also printed a mean without discarding the first request as warmup. That opening row included name lookup, TLS setup, and a proxy my office network still injects. Mean latency dropped after request one, so the summary looked like an easy win. I almost pasted that number into a status note. Would you have noticed, if the only extra line was a friendly average?

Hours 0–12: the laptop version that lied politely

I started with a tiny script and a hard-coded URL, because I wanted something I could paste into a ticket. The loop ran fifty GETs, stored durations, and printed mean, median, and a homemade p95. It never failed, which should have been the first loud warning. Real services cough sometimes, and a harness that never sees a timeout is usually too kind.

Here is the first version, labeled as a local experiment, not a production probe.

# local_experiment.py — laptop-only notes, not an SLO probe
import statistics
import time

import requests

URL = "https://httpbin.org/json"
N = 50


def run() -> None:
    session = requests.Session()
    samples = []
    for i in range(N):
        t0 = time.perf_counter()
        response = session.get(URL, timeout=10)
        response.raise_for_status()
        dt_ms = (time.perf_counter() - t0) * 1000
        samples.append(dt_ms)
        print(f"{i:02d} {dt_ms:8.2f} ms status={response.status_code}")
    samples_sorted = sorted(samples)
    p95 = samples_sorted[int(0.95 * (len(samples_sorted) - 1))]
    print("mean", statistics.mean(samples))
    print("median", statistics.median(samples))
    print("p95", p95)


if __name__ == "__main__":
    run()

I ran it from my kitchen table, and every row sat in a suspiciously narrow band. That felt like evidence, and evidence is exactly how these notes usually start. It was evidence of a warm TCP session, a nearby resolver, and a network path I do not get in CI. Have you ever published a chart before you ran the file somewhere that is not your desk?

Hours 12–24: the things that actually broke

I forced a cold start by dropping the shared session, and the first request jumped hard. Did I expect that jump in theory? Yes, I did. Did the original summary show it to me? No, because the mean swallowed the spike whole. I also sent Connection: close and watched later rows stop looking like copies of each other. The API had not changed overnight. My client had changed, and the chart followed the client.

Then DNS started to matter in a way my laptop had been politely hiding. I printed socket.getaddrinfo before each call and saw cached answers that a fresh server did not keep. Have you checked whether your timing loop includes name resolution on every iteration? Mine did not, until I closed the session and lost the cozy resolver cache. After that, a slow remote lookup added milliseconds that never existed at home.

Clock resolution was the next embarrassment, and it arrived through a cleanup patch I accepted too quickly. I logged time.get_clock_info("perf_counter") and compared those deltas with time.time. On the laptop, perf_counter stayed honest enough for millisecond notes. On the remote host, a copy-paste branch used time.time, and a few samples collapsed into the same coarse bucket. The assistant mixed the two clocks, and I shipped the mix because the output still looked numeric.

# clock_check.py — print clock facts before you trust a histogram
import time

info = time.get_clock_info("perf_counter")
print("implementation", info.implementation)
print("resolution_seconds", info.resolution)
print("monotonic", info.monotonic)

t0 = time.time()
p0 = time.perf_counter()
time.sleep(0.05)
print("time.time_delta_ms", (time.time() - t0) * 1000)
print("perf_counter_delta_ms", (time.perf_counter() - p0) * 1000)

Failures I would write on a sticky note

• A reused session made p95 look calm after request zero, even when TLS was expensive.
• A single mean hid warmup, DNS, and one slow redirect behind a green average.
• time.time in a patched branch flattened samples that perf_counter would have kept apart.
• Laptop DNS cache disagreed with a cold remote resolver, and I blamed the API first.

Hours 24–48: a harness I would actually repeat

I rewrote the loop so each phase had a name I could argue with later. DNS, total time, status, and reuse became columns instead of a single heroic average. I stopped asking the model for a benchmark and asked it to review a checklist instead. That review is where MonkeyCode entered these notes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access to interrogate the harness, not to invent a percentile formula I already knew. The free server option mattered because I needed a second machine that did not share my laptop resolver or kitchen Wi-Fi. I pasted the same commands there and compared CSV rows side by side, without a dashboard in the middle. I am not going to name models, quotas, or hardware I cannot verify from where I sit. I compared two environments and one script, and the disagreement was finally readable.

The checklist I now run before I trust a number

1. Record whether the HTTP client reuses connections across iterations.
2. Log DNS separately from the request, even when the value looks tiny.
3. Discard or label warmup rows instead of folding them into the mean.
4. Prefer time.perf_counter, and print clock info once at process start.
5. Fail the run if any timeout, redirect loop, or unexpected status appears.
6. Write raw samples to CSV so a later median is not trapped in stdout.
7. Repeat the file on a host that is not the laptop that drafted it.

Artifact: a slightly less dishonest probe

# field_probe.py — reproducible timing notes, not an SLO tool
from __future__ import annotations

import csv
import socket
import statistics
import time
from dataclasses import dataclass
from urllib.parse import urlparse

import requests

URL = "https://httpbin.org/json"
N = 30
WARMUP = 3
OUT = "probe_samples.csv"


@dataclass
class Row:
    index: int
    kind: str
    dns_ms: float
    total_ms: float
    status: int
    reused: bool
    redirects: int


def dns_ms(host: str) -> float:
    t0 = time.perf_counter()
    socket.getaddrinfo(host, 443, type=socket.SOCK_STREAM)
    return (time.perf_counter() - t0) * 1000


def probe(reuse: bool) -> list[Row]:
    host = urlparse(URL).hostname or ""
    session = requests.Session() if reuse else None
    rows: list[Row] = []
    for i in range(WARMUP + N):
        kind = "warmup" if i < WARMUP else "sample"
        lookup = dns_ms(host)
        client = session if session is not None else requests
        t0 = time.perf_counter()
        response = client.get(URL, timeout=10)
        total = (time.perf_counter() - t0) * 1000
        row = Row(
            index=i,
            kind=kind,
            dns_ms=lookup,
            total_ms=total,
            status=response.status_code,
            reused=bool(reuse and i > 0),
            redirects=len(response.history),
        )
        rows.append(row)
        print(row)
        if response.status_code >= 400:
            raise RuntimeError(f"unexpected status {response.status_code}")
    return rows


def summarize(rows: list[Row]) -> None:
    samples = [r.total_ms for r in rows if r.kind == "sample"]
    samples_sorted = sorted(samples)
    p95 = samples_sorted[int(0.95 * (len(samples_sorted) - 1))]
    print("n", len(samples))
    print("mean_ms", round(statistics.mean(samples), 2))
    print("median_ms", round(statistics.median(samples), 2))
    print("p95_ms", round(p95, 2))


def write_csv(rows: list[Row], path: str) -> None:
    with open(path, "w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(
            ["index", "kind", "dns_ms", "total_ms", "status", "reused", "redirects"]
        )
        for row in rows:
            writer.writerow(
                [
                    row.index,
                    row.kind,
                    f"{row.dns_ms:.3f}",
                    f"{row.total_ms:.3f}",
                    row.status,
                    row.reused,
                    row.redirects,
                ]
            )


if __name__ == "__main__":
    print("clock", time.get_clock_info("perf_counter"))
    print("== reuse on ==")
    reused_rows = probe(reuse=True)
    summarize(reused_rows)
    write_csv(reused_rows, "probe_reuse_on.csv")
    print("== reuse off ==")
    cold_rows = probe(reuse=False)
    summarize(cold_rows)
    write_csv(cold_rows, "probe_reuse_off.csv")

Run both modes on the laptop, keep the CSV files, and then copy the same script to a second host. Do not edit constants between those two runs, because a quiet constant change is how I fooled myself on day one. If the medians only agree on your desk, you do not have an API number yet. You have a laptop number wearing an API costume.

python -m pip install requests
python field_probe.py
# later, on a different host, same file, same constants:
python field_probe.py
diff -u probe_reuse_on.csv probe_reuse_on_remote.csv || true
diff -u probe_reuse_off.csv probe_reuse_off_remote.csv || true

Decision table I wish I had on hour one

Symptom	Laptop story	Better question	Action
Tight cluster after request 0	"API is fast"	Did TLS and DNS happen only once?	Split warmup, log reuse
Remote p95 much higher	"Server is bad"	Is the resolver cold every time?	Print getaddrinfo cost
Mean looks great, median disagrees	"Outlier, ignore it"	Did a timeout get averaged away?	Fail on errors, keep CSV
Assistant rewrote the timer	"Refactor is fine"	Which clock is in the patch?	Print get_clock_info
Status always 200	"Healthy"	Are redirects silently followed?	Log response.history

What I would repeat, and what I would not

I would repeat the two-host comparison before I paste any percentile into a document again. I would also repeat the checklist with the model in the room, because it is decent at spotting missing warmup labels. I would not repeat asking a chat window to write a benchmark with no constraints on clocks or reuse. That prompt invites a mean of mixed phases, and mixed phases are how I burned the first day.

Should you treat this file as an SLO source for a real service? No, and I will say that twice if needed. This harness hits a public echo endpoint in the sample, uses a tiny N, and does not control noisy neighbors on either host. It will not prove capacity, and it will not settle a vendor argument. It will only show whether your client story survives a second machine.

People writing compliance reports, contractual load tests, or billing-grade latency should not borrow this script. Use a real load tool, a named region, and an error budget you can defend in review. The free model access did not replace that comparison work, and it should not. It only helped me list the phases I kept forgetting to name out loud.

The free server option did not make the API faster, which is the whole point of these notes. It made the lie harder to keep, because my kitchen Wi-Fi was no longer sitting inside the timing path. If you want a second pair of eyes and a machine that is not your laptop, try that same split in MonkeyCode and keep the CSV. I did not collect days of production traffic, and I did not rank assistants against each other for a scoreboard.

I compared a warm laptop with a colder box, and that was already enough to ruin a chart. The average I trusted on hour four did not survive hour forty, once reuse, DNS, and clocks had names. That is the finding I would repeat next week, before I trust another green mean.