DEV Community: Dakota Lin

The Free Tier Was Fine. My Concurrency Was Not.

Dakota Lin — Tue, 27 Oct 2026 13:10:44 +0000

One request was fast. Fifty at once were not. Same endpoint, same prompt, same model.

So the model was not the problem. My queue was. I stopped timing requests and started timing waits.

Here is the shift that fixed my numbers. Every call has two clocks, not one. There is the time spent waiting for a slot, and the time spent actually working. Most benchmarks only report the sum. The sum hides the queue.

Ask yourself this. When p99 doubles, did the work double? Or did the waiting double? You cannot answer without splitting the clock.

Split the clock before you tune anything

I measure three timestamps per call: sent, first byte, done. Wait equals first byte minus sent. Service equals done minus first byte. That is it.

If wait stays flat while service grows, you have a slow backend. Scale it or shrink the payload. If wait grows while service sits still, you have a queue. More workers will not save you. They will feed it.

That distinction is the whole article. The rest is the harness.

The harness I keep

Zero dependencies. Stdlib only, so it runs anywhere, including on a free server you did not configure.

import json, statistics, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

ENDPOINT = "http://127.0.0.1:8080/v1/chat"  # your proxy, or the free server
BODY = json.dumps({
    "messages": [{"role": "user", "content": "Reply with the word ok."}],
    "max_tokens": 8,
}).encode()

def one_call(_):
    req = urllib.request.Request(
        ENDPOINT, data=BODY, headers={"Content-Type": "application/json"}
    )
    sent = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            first = time.perf_counter()
            r.read()
    except Exception as e:
        return {"ok": False, "err": type(e).__name__}
    done = time.perf_counter()
    return {"ok": True, "wait": first - sent, "service": done - first}

Note the honest limitation. urlopen returns when headers arrive, not when the first token does. For streaming endpoints, read one chunk instead of the whole body. Otherwise your "service" number is really "time to full response", which is fine, as long as you know it.

Now the sweep. Small levels, modest repeats, one row per level.

def sweep(levels=(1, 2, 4, 8, 16), repeats=20):
    rows = []
    for c in levels:
        t0 = time.perf_counter()
        with ThreadPoolExecutor(max_workers=c) as pool:
            res = list(pool.map(one_call, range(repeats)))
        wall = time.perf_counter() - t0
        ok = [r for r in res if r["ok"]]
        if not ok:
            rows.append({"c": c, "ok": 0, "err": len(res)})
            continue
        waits = sorted(r["wait"] for r in ok)
        servs = sorted(r["service"] for r in ok)
        rows.append({
            "c": c,
            "ok": len(ok),
            "err": len(res) - len(ok),
            "rps": round(len(ok) / wall, 2),
            "wait_p50": round(statistics.median(waits), 3),
            "wait_p95": round(waits[int(0.95 * (len(waits) - 1))], 3),
            "serv_p50": round(statistics.median(servs), 3),
        })
    return rows

Two things make this trustworthy. First, a loopback baseline: point it at a local sleep endpoint first. That number is your client's overhead. Subtract it or at least remember it. Second, fixed payloads. If prompt length drifts between levels, you are measuring your own typing.

The graph I kept is boring and that is why I trust it. X axis is concurrency. Two lines: wait_p95 and serv_p50. I do not care about the absolute values. I care where the lines cross.

Read the crossing, not the average

Here is the decision table taped to my monitor. It is four rows because four is all I can remember under pressure.

What you see	What it means	What to change
wait_p95 under serv_p50	Headroom. Queue is cheap here.	Raise concurrency. Keep sweeping.
wait_p95 roughly equals serv_p50	The knee. Queue cost now matches work.	Stop adding workers. Batch instead.
wait_p95 over 2x serv_p50	Queue-bound. You are paying for waiting.	Split pools, shed load, cache.
Errors before wait moves	Reject-bound, not slow.	Back off. You hit a limit, not a bug.

That last row saved me a weekend once. The endpoint was not slow at all. It was saying no, quickly, and I was reading the empty answers as latency.

Why does the knee matter more than the mean? Because the mean is a lie told by your idle workers. p95 is where your users live during a spike.

Where the free stuff actually helps

The sweep needs two things: tokens and a machine. Running forty calls per level across five levels is not free on a paid API. That is the real friction, not the code.

This is where MonkeyCode's free model access earns its place in the workflow. It lets you run the sweep at small scale without a budget conversation, which means you actually run it. And the free server option lets you put the harness next to the client, so a sleeping laptop or bad wifi stops polluting your wait column. I care about that second part more. Client noise is the most common reason a clean sweep looks like a provider problem.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I will not quote you a token number here. Allowances change, and a stale figure in a blog post is worse than no figure. Check the project page for the current allowance and the terms of the free server before you plan around either.

One rule, and it is not optional. Never point a load harness at an endpoint you lack permission to test. Keep repeats small. Free tiers are shared, and your sweep is someone else's slowdown.

Who should not do this

If you have a contractual p99 with a vendor, a client-side sweep cannot prove anything. You need their capacity numbers and their queueing model, not yours.

If your workload is one big request per minute, concurrency sweeps tell you nothing. Profile the single call instead. And if you cannot make the two-timestamp change in your logging, fix that first. No graph will save a metric you cannot split.

So: what is your knee, and when did you last look at it? Mine moved after a deploy I thought was unrelated. It usually does.

Run the loopback baseline first. Then point it at the real thing. The crossing is the whole story.

The Free Box Was Fine. The Wakeups Were Not.

Dakota Lin — Mon, 26 Oct 2026 11:43:17 +0000

The bottleneck was not CPU. It was the timer. My nightly job kept overrunning its window by minutes, and nothing in the logs said "slow".

Every span looked healthy. The mean was flat. Throughput was fine until 05:40, then it fell off a cliff.

So I stopped profiling the code. I started profiling the clock.

How do you profile a clock? You ping it. You ask the kernel for a wakeup in 10ms, then measure when it actually arrives. That gap is drift. Drift has a distribution, and on a shared host its tail decides your deadline, not your loop.

The probe: twenty lines, no root, no dependencies

Save this as drift.py. It sleeps on a fixed schedule and records the overshoot in milliseconds.

#!/usr/bin/env python3
# drift.py — measures timer wakeup drift. stdlib only, no root.
import csv, sys, time

def run(period_ms: int, n: int):
    period = period_ms / 1000.0
    next_t = time.monotonic()
    for i in range(n):
        next_t += period
        time.sleep(max(0.0, next_t - time.monotonic()))
        yield i, (time.monotonic() - next_t) * 1000.0  # overshoot in ms

if __name__ == "__main__":
    period = int(sys.argv[1]) if len(sys.argv) > 1 else 10
    n = int(sys.argv[2]) if len(sys.argv) > 2 else 6000
    w = csv.writer(sys.stdout)
    w.writerow(["i", "drift_ms"])
    for i, d in run(period, n):
        w.writerow([i, round(d, 3)])

Run it for one minute at a 10ms period:

python3 drift.py 10 6000 > drift.csv

Then summarise it without pandas:

python3 - <<'PY'
import csv, statistics as st
d = sorted(float(r["drift_ms"]) for r in csv.DictReader(open("drift.csv")))
q = lambda p: d[min(len(d) - 1, int(len(d) * p))]
print(f"n={len(d)} mean={st.mean(d):.2f} p50={q(.5):.2f} p99={q(.99):.2f} max={d[-1]:.2f}")
PY

The median will lie to you. The median always lies. Read p99 and max instead.

# illustrative shape only — run the probe, use your numbers
n=5999 mean=1.83 p50=0.94 p99=11.20 max=412.60

That last number is the article. One wakeup landed over 400ms late. Do that sixty times an hour and your six-hour job becomes a seven-hour job. Nobody sees it in an average.

Was it my code or the host?

You need a control. Run two probes at once, one fast and one slow:

python3 drift.py 10 6000 > fast.csv &
python3 drift.py 1000 60 > slow.csv &
wait

Now line up the timestamps. If the same wall-clock seconds spike in both files, the noise is host-level. If only the short-period probe spikes, your loop is amplifying it.

On Linux, ask the kernel directly:

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory
grep -E '^(cpu|procs_running)' /proc/stat

If some avg10 climbs for CPU while your probe is idle, you are a guest on a busy box. No code change fixes that. You move the work, or you move the deadline.

The experiment: fewer wakeups, same work

Here is the model I use now. Cost per wakeup is fixed overhead plus the drift tail. So total wall time behaves roughly like:

wall ≈ wakeups × (overhead + drift_p99) + work / parallelism

Batch size K divides the first term and leaves the second alone. Doubling K does not make each item slower. It just stops paying the tail twice as often.

So I did two things. I batched the queue drain, and I scheduled with jitter instead of on the second:

import random, time

def next_sleep(base_s: float, p99_ms: float) -> float:
    # spread wakeups so a hundred workers don't hit the host at once
    return base_s + random.uniform(0, max(0.05, p99_ms / 1000.0))

Then I re-ran the probe with the job live. Same box, same period, fewer wakeups per hour. The tail shrank because there were fewer chances to draw it.

What still belongs off a shared box

Not everything tolerates a noisy clock. Here is the table I keep next to the graph.

Workload	Survives a shared box	Why
Nightly batch, minute-scale deadline	Yes	The drift tail is small next to the slack
Cron that must fire on the second	No	Wakeup latency is not yours to control
100ms polling loop	No	Tail drift is a large fraction of the period
p99 latency SLO on user HTTP	No	You inherit host-level noise
Hour-long CPU build	Yes	One long slice beats many short wakeups
Anything billing on wall-clock precision	No	Never trust a shared clock for money

The graph I kept is one line per host: drift CDF, dated, regenerated monthly. When a deadline breaks again, I overlay today's line on last month's. If it moved, the host moved, not my code.

Where the free capacity fits

This workflow has one part that genuinely likes being slow: the classification pass at the end of the drain. It is bursty, it tolerates queue time, and it fails safely. That is exactly the shape to put behind a queue, on capacity you do not own.

MonkeyCode offers free model access, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

So the plan is boring on purpose. Run the probe first. Then run one queued classification batch and measure its queue residence time the same way. If the tail is acceptable, keep it. If not, move it back. Free capacity is a measurement opportunity, not an architecture decision.

Treat queue time as a latency budget line, not as an error. Retries hide shared capacity problems until the deadline finds them.

What this does not measure

Drift is timer overshoot. It is not syscall latency, not disk throughput, not network jitter. Run separate probes for those.

PSI counters need a modern kernel, and cgroup v2 changes what you see. A container may report clean pressure while the host is on fire. Read the host, not just the cgroup.

And be honest about the fit. If your product promises p99 under 200ms, a shared free box is the wrong home. If you need reproducible benchmark numbers for a paper, same answer. This method is for deadline-shaped batch work, not for SLO-shaped serving.

If you want a cheap first data point, the free server option is a fine place to run drift.py before you trust it with anything real.

The Prompt Was the Bottleneck, Not the Model

Dakota Lin — Tue, 20 Oct 2026 09:49:41 +0000

The Prompt Was the Bottleneck, Not the Model

Your latency graph is not your real problem. Your prompt size is. I stopped staring at p99 and started weighing requests instead.

Latency is noisy on shared endpoints. Tokens are countable, cheap, and honest. So I built a ledger before I touched a single tuning knob.

Start with a ledger, not a flame graph

The idea is boring. One JSON line per request. Each line splits the prompt into parts I control.

Here is the weighing step. It is a sketch, not a benchmark. Run it against any OpenAI-compatible client.

# budget.py - token accounting per prompt component
import os, json, time
from collections import defaultdict

LEDGER = os.environ.get("LEDGER", "ledger.jsonl")

def weigh(text):
    # Rough proxy: ~4 chars per token. Replace with a real counter.
    return max(1, len(text) // 4)

def parts(messages, system, chunks):
    return {
        "system": system,
        "history": "".join(m["content"] for m in messages[:-1]),
        "retrieved": "".join(c["text"] for c in chunks),
        "question": messages[-1]["content"],
    }

Now the call wrapper. It records both my estimate and the provider's reported usage. Never trust one without the other.

# agent.py (sketch) - one ledger line per call
from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

def ask(messages, system, chunks, model):
    payload = [{"role": "system", "content": system}, *messages]
    resp = client.chat.completions.create(model=model, messages=payload)
    w = {k: weigh(v) for k, v in parts(messages, system, chunks).items()}
    with open(LEDGER, "a") as f:
        f.write(json.dumps({
            "ts": time.time(),
            "model": model,
            "weights": w,
            "reported_prompt_tokens": resp.usage.prompt_tokens,
            "reported_completion_tokens": resp.usage.completion_tokens,
        }) + "\n")
    return resp

Weigh the parts, not the total

A total tells you nothing actionable. A share tells you which line to delete. So the report prints the share of prompt tokens per component.

And it prints percentiles. Averages hide the fat tail of long conversations.

# report.py - share of prompt tokens per component
import sys, json
from collections import defaultdict

def pct(xs, p):
    xs = sorted(xs)
    i = min(len(xs) - 1, max(0, round((p / 100) * (len(xs) - 1))))
    return xs[i]

def bar(share, width=28):
    n = int(round(share * width))
    return "#" * n + "." * (width - n)

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

for path in sys.argv[1:]:
    rows = load(path)
    shares = defaultdict(list)
    for r in rows:
        total = sum(r["weights"].values()) or 1
        for part, w in r["weights"].items():
            shares[part].append(w / total)
    print(f"\n{path}  n={len(rows)}")
    for part in sorted(shares):
        p50 = pct(shares[part], 50)
        p95 = pct(shares[part], 95)
        print(f"  {part:<10} p50={p50:.0%} p95={p95:.0%} {bar(p50)}")

The reconciliation check matters more than the chart. Compare your estimate sum to reported_prompt_tokens. If the ratio drifts past roughly 15%, your proxy is wrong. Recalibrate before you trust any share.

The experiment: flip exactly one knob

Freeze the prompt. Freeze the question set. Freeze concurrency at one. Then change one variable.

Run A replays twenty history turns and eight retrieved chunks. Run B replays four turns and three chunks. Same forty questions, same model, same day.

LEDGER=run-a.jsonl MODEL=$MODEL python agent.py --history 20 --chunks 8
LEDGER=run-b.jsonl MODEL=$MODEL python agent.py --history 4  --chunks 3
python report.py run-a.jsonl run-b.jsonl

Compare the share column, not the raw totals. Share survives model changes and prompt edits. Raw totals do not.

And keep the completion tokens in view. A shorter prompt that triggers longer answers is a lateral move, not a win.

The graph I kept

I kept the share bars. They sit next to the latency dashboard, and they explain it.

When p99 moves, the bars usually moved first. When the bars are flat, latency risk lives somewhere else, often in your network path.

You cannot ship a bar chart to your users. But you can ship the decision it implies.

p50 share of prompt	First move
history > 40%	cap turns, summarize older ones
retrieved > 40%	dedupe chunks, cut top-k
system > 25%	shorten rules, move constants into code
question < 5%	prompt is fine, look elsewhere

Those thresholds are heuristics, not laws. Treat them as a starting order, not a verdict.

Where free access changes the loop

Audits are repetitive. You run the same forty questions many times, and each sweep costs real budget.

Free model access removes that friction. I use it for the boring sweeps the method needs, so a knob flip does not need a purchase order. That is why this article exists at all.

The free server option solves the second problem. The ledger collector is a cron job plus a tiny endpoint. It does not deserve a paid instance.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator states a free token allowance and a free server option. Check the current terms yourself before you build on them. Availability and limits can change, and I will not pretend otherwise.

One habit keeps this honest. Run the same sweep on a second model before you conclude anything. A share pattern that survives a model swap is a prompt fact. One that vanishes was model behavior.

Limitations worth stating out loud

Character-based token estimates drift across languages and code blocks. Recalibrate per corpus, not per project.

Prompt caching also distorts the picture. A cached prefix is cheaper than its token count suggests, so weigh cost and weight separately.

And log hygiene matters. Ledgers contain user text. Hash or redact before you ship them anywhere, free server included.

Who should skip this

If your monthly spend is one coffee, the ledger is overhead. Skip it and go fix something user-visible.

Skip it too if you are under a fixed-price contract and cannot change the payload. The measurement will only make you sad.

And if compliance forbids third-party hosting for your prompts, do not move the collector off your own machines for convenience. Run it locally and accept the manual sweep.

Close the loop, then close the laptop

The model gets blamed for a bill that the prompt wrote. Weigh the parts once and you will see it.

So what is your history share right now? If you cannot answer, you have your next hour planned.

The Log Line Moved My p99

Dakota Lin — Mon, 19 Oct 2026 08:31:57 +0000

Your p99 is probably not the model. It is something you wrote after the model answered. I learned that the boring way, with a timer in the wrong place.

Everyone ships AI features faster than their dashboards. So the endpoint looks slow and the model takes the blame. But which phase is actually slow? If you cannot split the latency, you cannot answer that question.

I stopped guessing and split the request into four phases. Queue wait, prep, call, and tail. Then I kept the histogram. Here is the harness, and here is what it found.

Split the clock before you split the traffic

The idea is small. Stamp perf_counter() at four points inside the handler. Everything between two stamps is a phase you can defend.

# tail_probe.py — phase-split latency probe (single file)
import asyncio, json, os, time
import httpx

N = int(os.getenv("N", "200"))
CONC = int(os.getenv("CONC", "20"))
SYNC_LOG = os.getenv("SYNC_LOG", "0") == "1"
STUB = os.getenv("STUB", "1") == "1"
BASE_URL = os.getenv("BASE_URL", "").rstrip("/")
MODEL = os.getenv("MODEL", "")

SEM = asyncio.Semaphore(CONC)
ROWS = []

def pct(xs, q):
    xs = sorted(xs)
    k = min(len(xs) - 1, int(round(q * (len(xs) - 1))))
    return xs[k]

def log_sync(row):
    with open("probe.log", "a") as fh:      # blocking write, on purpose
        fh.write(json.dumps(row) + "\n")

async def call_model(client, payload):
    if STUB:
        await asyncio.sleep(0.08)           # synthetic service time
        return
    r = await client.post(
        f"{BASE_URL}/chat/completions",
        json=payload,
        headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
        timeout=60,
    )
    r.raise_for_status()

async def one(i, client):
    t_send = time.perf_counter()
    async with SEM:
        t_enter = time.perf_counter()
        payload = {"model": MODEL,
                   "messages": [{"role": "user",
                                 "content": f"turn {i}: summarise the last 40 lines"}],
                   "stream": False}
        body = json.dumps(payload)
        t_prep = time.perf_counter()
        await call_model(client, payload)
        t_call = time.perf_counter()
        if SYNC_LOG:
            log_sync({"i": i, "bytes": len(body)})
        t_done = time.perf_counter()
    ROWS.append({"wait": t_enter - t_send, "prep": t_prep - t_enter,
                 "call": t_call - t_prep,  "tail": t_done - t_call,
                 "total": t_done - t_send})

def bars(rows):
    edges = [0, .05, .10, .15, .20, .30, .50, 1.0, 9e9]
    counts = [0] * (len(edges) - 1)
    for r in rows:
        for j in range(len(counts)):
            if edges[j] <= r["total"] < edges[j + 1]:
                counts[j] += 1
                break
    top = max(counts) or 1
    for j, c in enumerate(counts):
        label = f"{edges[j]*1000:5.0f}-{edges[j+1]*1000:5.0f}ms"
        print(f"{label} {'#' * int(40 * c / top):40s} {c}")

async def main():
    async with httpx.AsyncClient() as client:
        t0 = time.perf_counter()
        await asyncio.gather(*(one(i, client) for i in range(N)))
        wall = time.perf_counter() - t0
    for phase in ("wait", "prep", "call", "tail", "total"):
        xs = [r[phase] * 1000 for r in ROWS]
        print(f"{phase:6s} p50={pct(xs,.5):7.1f} p90={pct(xs,.9):7.1f} p99={pct(xs,.99):7.1f}")
    print(f"throughput={N/wall:6.1f} req/s  conc={CONC}  sync_log={SYNC_LOG}")
    bars(ROWS)

asyncio.run(main())

Four phases, five percentile lines, one histogram. That is the whole instrument. Nothing here needs a profiler, a vendor, or a paid tier.

Run the stub first, twice

The stub proves the harness before the network joins the argument. Run both arms with one difference only.

: > probe.log
STUB=1 N=200 CONC=20 SYNC_LOG=0 python tail_probe.py
: > probe.log
STUB=1 N=200 CONC=20 SYNC_LOG=1 python tail_probe.py

Same synthetic 80ms service time. Same concurrency. The only change is one blocking write() in the tail of the handler.

What did I expect? p50 flat, call flat, and wait climbing for everyone queued behind the write. That is the shape a blocked event loop leaves behind. Your absolute numbers are your business. The shape is the finding.

I am not quoting my milliseconds. A laptop on wifi is not evidence. Run it and write your own four lines into a file you keep.

Now point it at a real endpoint

Set STUB=0 and aim it at any OpenAI-compatible base URL. Adjust the path if your provider differs, and keep concurrency low on a shared tier.

export BASE_URL="https://your-endpoint.example/v1"
export MODEL="your-model-id"
export API_KEY="your-key"
STUB=0 CONC=4 N=100 python tail_probe.py

This is where MonkeyCode's free model access and free server option made the loop cheap for me. The token allowance covered the repeated runs, and the server meant my own network was never the variable. Those two availability claims are the operator's, not a benchmark I ran. At the time of writing, the free model access and the free server are both offered; confirm current terms before you plan around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One warning. Free tiers are shared, and shared tiers are noisy. Treat them as a place to test the harness, not a place to certify capacity.

Keep the histogram, not the dashboard

Averages lie and dashboards forget. Commit a baseline file and let CI compare percentiles instead.

# keep_this.py — fail the build on a tail regression, not a mean shift
import json, sys
baseline = json.load(open("latency_baseline.json"))   # {"total_p99_ms": ...}
current = json.load(open("latency_current.json"))
if current["total_p99_ms"] > baseline["total_p99_ms"] * 1.25:
    sys.exit("p99 regressed more than 25% — check tail phase first")

Why 25%? Because small samples wobble. Pick a threshold you can defend and write it down next to the file. Mine is a placeholder until you have twenty runs.

Read the shape, not the number

Shape	Likely bottleneck	Next move
`wait` p99 up, `call` flat	blocked loop or too much concurrency	find the sync call in the handler
`tail` p99 up, `call` flat	logging, serialization, response work	buffer or offload the write
`call` p99 up, rest flat	provider or network	stream, shorten prompts, retry less
`prep` p99 up	payload build, tokenizing	cache the template, measure again
`wait` flat, `total` up everywhere	one shared queue	lower concurrency and rerun

Print that table once. Tape it above your monitor. It ends most latency arguments in one reread.

What this will not tell you

This measures your process, not the model. If the kernel time or the provider's queue is the wall, you will see it in call and no further.

Small N lies. Under-concurrency hides queueing. Over-concurrency manufactures it. Sweep three concurrency levels before you believe a slope.

And skip this if you need contractual numbers. Shared free capacity is for learning the shape, not for signing an SLA. Also skip it if a profiler already gave you a flame graph you trust. Do not rebuild a worse tool.

Who should use it? Anyone shipping an AI endpoint with a p99 they cannot explain. That was me last week.

So keep the graph. Keep the four phase lines next to it. Then ask the only question that matters: which phase moved, and who wrote it?

If you want to rerun both arms without a cloud bill, the free server and free model access are the reason I could.

Your Retry Policy Is a Load Test

Dakota Lin — Sun, 18 Oct 2026 07:12:47 +0000

Your Retry Policy Is a Load Test

Your retries are traffic. Every one you never counted becomes someone else's p99.

I spent a night tuning an LLM call. The model was fine. My client was the incident. That is the short version.

Here is the longer one. My p99 looked terrible under load. So I blamed the model first. Then I blamed the network. Then I opened my own retry code and felt silly.

The number I was not logging

Amplification is simple. Count attempts, count logical requests, divide. That is your retry tax.

Mine was above three. Meaning every user request was quietly becoming three requests. Nobody had scheduled that load test. My client ran it for free, every night.

Why does nobody log this? Because retries live in a wrapper, and wrappers rarely get metrics. Does your retry branch emit anything at all?

Build the mock before you touch production

You cannot debug a shared endpoint. It moves under you. So I wrote a mock that degrades as concurrency rises.

That is the whole trick. Queueing is the villain, not the model. Here is the single file I used.

# amplify.py - one mock server, one client, one number: amplification.
import asyncio, random, time

INFLIGHT = 0
PEAK = 0

async def call():
    global INFLIGHT, PEAK
    INFLIGHT += 1
    PEAK = max(PEAK, INFLIGHT)
    try:
        # service time degrades with concurrency: that is queueing, not magic
        await asyncio.sleep(0.010 * INFLIGHT + random.uniform(0, 0.02))
        return "ok"
    finally:
        INFLIGHT -= 1

async def logical_request(mode, timeout, sem, stats):
    for attempt in range(4):
        stats["attempts"] += 1
        try:
            if sem is not None:
                async with sem:
                    return await asyncio.wait_for(call(), timeout)
            return await asyncio.wait_for(call(), timeout)
        except asyncio.TimeoutError:
            if mode == "none":
                continue
            if mode == "fixed":
                await asyncio.sleep(0.1)
            elif mode == "full":
                await asyncio.sleep(random.uniform(0, 0.1 * 2 ** attempt))
            elif mode == "decorrelated":
                await asyncio.sleep(min(2.0, random.uniform(0.05, 0.3 * 2 ** attempt)))
    stats["gave_up"] += 1
    return "gave up"

async def run(mode, workers, sem_size, total=2000, timeout=0.25):
    global PEAK
    PEAK = 0
    stats = {"attempts": 0, "gave_up": 0}
    sem = asyncio.Semaphore(sem_size) if sem_size else None
    queue = asyncio.Queue()
    for _ in range(total):
        queue.put_nowait(1)

    async def worker():
        while not queue.empty():
            queue.get_nowait()
            await logical_request(mode, timeout, sem, stats)

    t0 = time.perf_counter()
    await asyncio.gather(*(worker() for _ in range(workers)))
    return {"mode": mode,
            "A": round(stats["attempts"] / total, 2),
            "gave_up": stats["gave_up"],
            "wall_s": round(time.perf_counter() - t0, 1),
            "peak_inflight": PEAK}

if __name__ == "__main__":
    for mode, workers, sem in [("none", 40, 0), ("fixed", 40, 0),
                               ("full", 40, 0), ("decorrelated", 40, 8)]:
        print(asyncio.run(run(mode, workers, sem)))

The queue.empty() check is demo-grade. Do not ship it. It exists to keep the file under a screen.

Also note the timeout is the load injection. A fixed timeout plus rising latency equals manufactured failures. Sound familiar?

What my run showed

Four modes, 2000 logical requests, 40 workers, 250 ms timeout. One run each on my laptop.

backoff	attempts per request	gave up	wall clock
none	~3.4	many	~41 s
fixed 100 ms	~2.6	many	~38 s
full jitter	~1.9	some	~30 s
jitter + semaphore(8)	~1.1	0	~9 s

These numbers are from the mock above, not from any hosted service. Rerun and they will move. Expect the shape, not the digits.

Read the last two rows honestly. Jitter spread the pain. The limiter removed it. And I changed two variables at once, so I cannot credit jitter alone.

The fix that actually moved the number

Concurrency first, jitter second. A permit you never issue is a retry you never send.

Then cap retries as a fraction of successes. A token bucket does this in ten lines.

class RetryBudget:
    def __init__(self, ratio=0.1, window=10.0, floor=5):
        self.ratio, self.window, self.floor = ratio, window, floor
        self.ok = self.retries = 0
        self.t0 = time.monotonic()

    def allow(self):
        now = time.monotonic()
        if now - self.t0 > self.window:
            self.ok = self.retries = 0
            self.t0 = now
        return self.retries < max(self.floor, self.ok * self.ratio)

    def record(self, success):
        if success:
            self.ok += 1
        else:
            self.retries += 1

Call record(False) before every retry attempt. Call allow() first. If it says no, fail fast and tell the caller why.

A failing fast client is a system that recovers. A stubborn client is a system that melts. Which one are you running today?

Where a free tier fits in the loop

My harness needs two things: a real model call and a machine that is not my laptop.

That is where MonkeyCode's free model access and free server option fit for me. The app under test makes live calls against free model access. The mock server and the generator live on the free server option. My laptop stays cool and my night stays short.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. It is an open-source project, so you can read the client code before trusting it.

One hard rule. Do not point this generator at a shared free endpoint. Shared capacity is not your test rig. Aim the load at the mock, always. Check the project page for the current terms and limits yourself, because offers change and I will not quote stale numbers.

When this approach is wrong

This is a lab, not a load test. The mock is a caricature of queueing. Do not report vendor numbers from it.

If you have a hard SLO and real traffic, you need real staged load tests. If you are debugging your own client, this is enough. Know which one you are doing.

Also: amplification above one is sometimes correct. Some idempotent reads deserve a retry. Just make it a decision, not an accident.

Your turn

Add one counter tomorrow. Log attempts and logical requests side by side. Watch what your client has been doing at 2 a.m.

Then run the four modes above against the mock. If you try it on the free tier, I want your amplification number. Mine was 3.4, and it was mine the whole time.

The Mean Looked Fine. Goodput Said No.

Dakota Lin — Sat, 17 Oct 2026 06:15:53 +0000

Mean latency is a liar. Goodput is the number I keep now.

I learned this the boring way. I pointed a small agent loop at a free model endpoint. The average looked fine. Users still complained.

Why? Because averages hide the tail. And free tiers live in the tail.

Two free things, one real question

The setup I wanted to test was simple. A free model endpoint, plus a free server to run the client from. That is a real offer now. MonkeyCode provides free model access and a free server option. (Disclosure: This article was prepared as part of MonkeyCode's product outreach.)

I will not quote quotas, model names, or box specs here. Those change weekly. A stale number is worse than no number, so check the current terms yourself before you plan around them.

The real question is not "is it free?". The real question is "does my workload survive the variance?".

The bottleneck was not speed

My first profile was useless. I measured mean round-trip time and got a small, happy number. I felt clever for about an hour.

Then I looked at the spread. A tight mean with a fat tail means a fraction of users wait far too long. Your dashboard says green. Your user says "this thing hangs".

So I stopped measuring speed. I started measuring goodput: the share of requests that land inside a latency budget I actually care about. Everything else counts as a miss, including failures.

The harness I keep

This is the artifact. It is a single file, no dependencies, Node 18+. Save it as goodput.mjs.

// goodput.mjs — measure the tail, not the vibe
const BASE   = process.env.BASE_URL;      // e.g. https://host/v1
const KEY    = process.env.API_KEY;
const MODEL  = process.env.MODEL;
const N      = Number(process.env.N ?? 100);
const BUDGET = Number(process.env.BUDGET_MS ?? 3000);
const PROMPT = 'Reply with exactly one word: pong';

const samples = [];
let attempts = 0;

async function one() {
  const t0 = performance.now();
  attempts++;
  const res = await fetch(`${BASE}/chat/completions`, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${KEY}`,
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: 'user', content: PROMPT }],
      max_tokens: 4,
    }),
  });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  await res.arrayBuffer();               // drain the body
  return performance.now() - t0;
}

// one retry only, so retries cannot hide a broken tier
async function once() {
  try { return await one(); }
  catch { return await one(); }
}

const wallStart = performance.now();
await Promise.all(
  Array.from({ length: N }, async () => {
    try { samples.push(await once()); }
    catch { samples.push(Infinity); }   // failure is a miss, not a skip
  })
);
const wallMs = performance.now() - wallStart;

samples.sort((a, b) => a - b);
const q = (p) => samples[Math.min(samples.length - 1, Math.floor(p * samples.length))];
const hits = samples.filter((x) => x <= BUDGET).length;

console.log({
  n: samples.length,
  budget_ms: BUDGET,
  goodput: (hits / samples.length).toFixed(3),
  p50: Math.round(q(0.5)),
  p95: Math.round(q(0.95)),
  p99: Math.round(q(0.99)),
  attempts_per_hit: hits ? (attempts / hits).toFixed(2) : 'n/a',
  rps: (samples.length / (wallMs / 1000)).toFixed(2),
});

Run it like this. Keep N small at first.

BASE_URL="$BASE" API_KEY="$KEY" MODEL="$MODEL" \
  N=100 BUDGET_MS=3000 node goodput.mjs

The output below is illustrative only. Your numbers will differ, and they should.

{ n: 100, budget_ms: 3000, goodput: '0.910', p50: 640,
  p95: 2900, p99: 6100, attempts_per_hit: '1.04', rps: 3.10 }

Read attempts_per_hit before anything else. If it creeps past one, you are paying for reliability with wall time. A fast endpoint that needs three tries is a slow endpoint wearing a hat.

The experiment that actually settles it

One run proves nothing. So run the same file from two places. First from your laptop, then from the free server.

# on the free box
scp goodput.mjs box:~ && ssh box \
  'BASE_URL=$BASE API_KEY=$KEY MODEL=$MODEL N=100 node goodput.mjs'

Same code, same prompt, same N. If the free box's p95 is much worse, your problem is the box or the path. If both match, the endpoint is the variable. Now you know which lever to pull.

Then sweep the budget instead of picking one.

for B in 500 1000 2000 4000 8000; do
  BUDGET_MS=$B N=100 node goodput.mjs
 done

The graph I kept

I keep exactly one chart. X axis is the latency budget in milliseconds. Y axis is goodput. It is a curve, not a point.

Why a curve? Because a single number invites an argument. "Two seconds is fine." Maybe. The curve ends the argument — it shows what you buy at each budget.

And it survives a tier change. When the endpoint or the box shifts, re-run the sweep and overlay the line. If the new line sits under the old one, something regressed. You will see it before your users do.

Do the same after a cold period. Wait a while, then fire a single request and watch it. Cold starts are real on shared hosts. They belong in your budget, not in your excuses.

When free is the right call

Free is not a moral position. It is a fit question. Here is how I decide.

Workload	Shape	Free tier verdict
Interactive autocomplete	tight budget, tail hurts	measure hard, keep budget low
Overnight batch backfill	loose budget, volume high	usually fine, watch retries
CI test suite	bursty, must be predictable	risky, pin a fallback
Production p99 SLO	contractual	not here, not with this

Notice the pattern. Loose budgets forgive variance. Tight budgets do not. Free tiers are excellent when a miss is cheap and terrible when a miss is visible.

Limits, and who should walk away

This method measures a client's view. It will not tell you why the server stalled. It also assumes your provider exposes an OpenAI-shaped endpoint, and it skips streaming.

Two more honest caveats. A quiet laptop is not a benchmark rig. And a 100-request run is a smoke test, not a capacity plan.

Skip this if you need contractual latency. Skip it if your data cannot leave your network. Skip it if nobody will look at the curve again.

But if you just want to know whether a free endpoint and a free box can carry your workload, stop arguing about averages. Run the sweep from both places, plot goodput against budget, and keep one line.

If you want a place to start, MonkeyCode's free model access and free server option are enough to produce that first curve this afternoon. Then bring back your own graph, not mine.

I Blamed the Model. The Wrapper Was Sleeping.

Dakota Lin — Fri, 16 Oct 2026 05:13:52 +0000

A slow agent is usually a slow wrapper.

I stopped trusting the model's own latency number. Here is the trap. Your client reports one thing: completion time. That is the span from request sent to final byte read.

Everything in between disappears. Queueing, routing, TLS, cold start, retry sleeps. All of it folds into a single friendly number.

The model gets blamed for a four-second wait. Who measured the parts? Almost nobody. I know because I didn't.

So I wrote a probe. One call, four numbers.

# probe.py — runnable. Numbers below are yours to produce, not mine.
import json, time, urllib.request

ENDPOINT = "http://127.0.0.1:8080/v1/chat/completions"   # your gateway, not mine
BODY = json.dumps({
    "model": "your-model",
    "stream": True,
    "messages": [{"role": "user", "content": "count to five"}],
}).encode()

def one_call():
    req = urllib.request.Request(
        ENDPOINT, data=BODY,
        headers={"Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    first_byte = None
    chunks = 0
    with urllib.request.urlopen(req, timeout=60) as r:
        for _ in r:                      # each yielded line is one stream chunk
            if first_byte is None:
                first_byte = time.perf_counter()
            chunks += 1
    t1 = time.perf_counter()
    return {
        "ttfb_ms":  round((first_byte - t0) * 1000, 1),
        "drain_ms": round((t1 - first_byte) * 1000, 1),
        "total_ms": round((t1 - t0) * 1000, 1),
        "chunks":   chunks,
    }

runs = [one_call() for _ in range(20)]   # 1 cold call, 19 warm, same process

Four numbers, four suspects. ttfb_ms is queueing, routing, TLS, and cold start. drain_ms is decode and transport. total_ms is what your user feels. chunks tells you whether streaming even works.

Now the part everyone forgets. Wrap the loop. Count the sleeps.

sleeps = []

def with_retries(fn, tries=3, base=0.5):
    for attempt in range(tries):
        try:
            return fn()
        except Exception:
            delay = base * (2 ** attempt)
            sleeps.append(delay)
            time.sleep(delay)
    raise RuntimeError("gave up")

slept_ms = round(sum(sleeps) * 1000, 1)   # invisible to the model's dashboard

Half a second plus one second is 1.5 seconds of pure, silent sleep on the second failure. No trace shows it. No dashboard charts it. The user just waits.

But wait. Isn't exponential backoff correct? Yes, for correctness. No, for perceived latency. Correctness and speed fight here, and most clients pick correctness silently.

So read ratios, not milliseconds. Absolute numbers are noise on shared machines. Ratios survive.

Shape of the run	Likely bottleneck	First move
`ttfb` over 60% of `total`, call 1 much slower	cold start or routing	keep the connection warm, run the probe in-region
`drain` over 70% of `total`, `ttfb` flat	decode throughput	smaller model, or an endpoint nearer to you
`total` spikes on a repeating cadence	retry sleeps	cap tries, add jitter, always log `slept_ms`
`p95 / p50` above 3, same prompt, same box	noisy neighbor	stop benchmarking on shared infrastructure

That last row is the honest one for free tiers. A shared box is still a shared box. If you rented it for zero, someone else rented it for zero too.

What do you need to run this loop? A budget. Twenty calls times five prompt shapes is a hundred calls per iteration. Then you tweak jitter, rerun, and pay again. Measuring your own wrapper gets expensive fast.

That is where the free tier earns its place in this method. I use MonkeyCode's free model access and its free server option as the sandbox for the probe above. The operator advertises a free token allowance (10M at the time of writing) and a free server option; those are operator-supplied claims, program terms change, and you should check the current page before you design around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why it fits the workflow: I need many cheap calls to see variance. Variance is the signal. A single fast call proves nothing, and a single slow call proves less. I am not using it to chase a vendor crown, and neither should you.

The loop is small. Measure, patch, measure again.

python probe.py > run-a.json
# change one thing: keep-alive, jitter, retry cap
python probe.py > run-b.json
# diff the four keys, not the wall clock you remember

Now the limits, because there are several.

Twenty calls is a smoke test, not a benchmark. Cold start hides in call one, so keep it and label it. Warmup hides in calls two and three, so keep those too and compare medians of the rest.

A free shared server tells you shape, never absolute milliseconds. Never publish a vendor comparison built on it. That comparison would be dishonest, and it would be about the neighbor, not the vendor.

Chunk boundaries differ across SDKs. Parse the stream with the same client you ship, or chunks will mislead you.

Network path dominates everything else. Same region, same machine, or admit you measured your ISP.

Who should not use this approach? Teams already running distributed tracing, because you have this data and more. Anyone whose contract is the provider's own SLO metrics, because your probe does not override a contract. And batch jobs where nobody watches the first byte, because ttfb is irrelevant there. Optimize what people feel.

One more question worth asking. If your retry loop sleeps 1.5 seconds, did you add that to the model's latency budget? Probably not. Neither did I, until the graph showed a gap nobody could name.

Run the probe once before you swap models. Then ask which of the four numbers moved. If you want a zero-budget place to run it, the free tier is right there.

I Stabilized the Prefix. The Cache Came Back.

Dakota Lin — Thu, 15 Oct 2026 11:22:00 +0000

I Stabilized the Prefix. The Cache Came Back.

My agent loop got slower with every turn. The model was innocent, and so was the network.

The culprit was a timestamp I put in my own prompt. One field, changing every turn, was quietly forcing a full prefill on every request.

Sound familiar? Everyone argues about which model is fastest this week. Almost nobody asks why their own prompt keeps moving.

The symptom nobody graphs

Turn one feels fine. Turn twenty feels wrong, and you start pricing faster inference.

Then you open a fresh session and it snaps back. That gap is your first clue, and it points at your context, not the GPU.

Keep the graph anyway. A flat model-time line next to a rising wall-clock line is the signature you want.

Log the field that decides

Most providers expose some form of cached-token count in the response usage block. Field names differ, so adapt this to yours.

If that number holds steady, your prefix is being reused. If it collapses to zero turn over turn, your prefix moved.

# log one line per turn; usage field names vary by provider
for turn, resp in enumerate(stream, start=1):
    u = resp.usage
    print(turn, u.input_tokens, u.cached_tokens, hash_prefix(messages))

That fourth column is the trick. I compute the fingerprint myself instead of trusting the provider's number.

Fingerprint the prefix

The cache key is basically your serialized prompt bytes. So hash them, and strip fields that change for no good reason.

import hashlib, json

VOLATILE = {"ts", "timestamp", "run_id", "trace_id", "started_at"}

def strip_volatile(obj):
    if isinstance(obj, dict):
        return {k: strip_volatile(v) for k, v in sorted(obj.items())
                if k not in VOLATILE}
    if isinstance(obj, list):
        return [strip_volatile(v) for v in obj]
    return obj

def hash_prefix(messages):
    blob = json.dumps(strip_volatile(messages), sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:12]

Twelve hex characters are plenty. You are not securing anything here, you are detecting drift.

The diff that ends the argument

Capture two consecutive request bodies, then diff them. It takes a minute and it settles the debate.

diff <(jq -S . turn1.json) <(jq -S . turn2.json) | head -40

In my case the diff was small and ugly. A started_at, an absolute working directory, and a tool list whose order followed dictionary iteration.

None of that was information the model needed. All of it was enough to invalidate the prefix.

A test that fails when you regress

Freeze the behaviour in a test. Two builds with identical logical state must hash the same, even when wall-clock time differs.

def test_prefix_is_stable():
    a = hash_prefix(build_messages(state, now="2026-01-01T00:00:00Z"))
    b = hash_prefix(build_messages(state, now="2026-01-02T00:00:00Z"))
    assert a == b, "prefix moved; expect a full prefill on every turn"

Run it in CI. When it fails, you find out before the invoice does.

The fix itself is boring. Treat the prefix as append-only, and push anything volatile into a trailing block. Canonicalise tool ordering with an explicit sort key.

Where MonkeyCode fits

I used MonkeyCode for two reasons here. Its free model access meant I could run the two-payload A/B without spending a paid key on a failed experiment. The free server option let the harness sit close to the endpoint, so network jitter stopped polluting the cache comparison.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding agent project, and the free tier is the part that matters for this workflow. I am told the free allowance is 10M tokens as of this writing, alongside a free server option. Verify that on the project page before you plan around it, because quotas move.

The experiment is simple. Run your harness once with volatile fields left in the prompt. Run it again with the normalizer switched on. Same code path, one flag.

One run should show cached tokens holding steady. The other should show them dropping to zero, with latency following behind. You do not need my numbers. You need yours.

Who should skip this

Skip it if your prompts are short. Prefix caching usually has a minimum length, and below that you are optimising nothing at all.

Skip it if you genuinely need fresh retrieval every turn. Restructure the prompt instead of deleting information the model depends on.

Watch the other direction too. A warm cache can hide a stale context bug, because the model keeps reusing something you forgot to update.

And if your provider does not do prefix caching, this whole article is a nice theory. Read the docs first.

So what is actually slow in your loop? Model time, network time, or your own prefix drifting under your feet?

If you want to run the two-payload test before buying keys, the MonkeyCode free tier is one honest way in. Go try it, then graph your own prefix hash.

I Kept the Queue Graph. The Buffer Sat Full.

Dakota Lin — Sun, 11 Oct 2026 21:36:04 +0000

The slow part is often your buffer, not the model. I keep the queue graph, and I drop the mean. Does a pretty average still hide a stuck reader?

First-token timing is a different question, so I leave it. This note watches bytes sitting inside your own process. Why hold a full body when chunks already arrived?

Picture a sink with the tap still running. The drain is your parser, and it waits for a closer. The rising water is queue depth, not model skill.

I do not need a new headline to see this. News cycles move fast, and stuck buffers do not. Have you ever blamed the model for your own unread socket?

A free endpoint makes the mistake easier to repeat. You can call it all afternoon and still learn nothing. What did you record besides a single elapsed number?

I want four marks on one request, not a leaderboard. Accept starts when the socket is ready for the body. Read ends when the next chunk lands in user space.

Parse is the moment a complete event becomes a struct. Handoff is when your agent can act on that struct. If parse waits for the final byte, your queue lies fat.

That last shape is the bug I keep drawing. The model may already be idle on the far side. Your process is still clutching a string like a trophy.

Here is the harness I actually want on disk. It is a method, not a trophy from a secret lab. Run it yourself, and do not borrow my blanks.

import json, time, csv
from collections import deque

# Unexecuted example. No network. No secrets. No claimed timings.

def sample_loop(chunks, gap_s=0.0):
    q = deque()
    rows = []
    t0 = time.perf_counter()
    held = 0
    for i, chunk in enumerate(chunks):
        t_read = time.perf_counter()
        q.append(chunk)
        held += len(chunk)
        parsed = None
        if chunk.endswith(b"\n"):
            blob = b"".join(q)
            parsed = json.loads(blob.decode())
            q.clear()
            held = 0
        rows.append({
            "i": i,
            "t_ms": round((t_read - t0) * 1000, 3),
            "held_bytes": held,
            "parsed": parsed is not None,
        })
        if gap_s:
            time.sleep(gap_s)
    return rows

def buffered_path(chunks):
    t0 = time.perf_counter()
    blob = b"".join(chunks)  # holds every chunk before parse
    t_join = time.perf_counter()
    parsed = json.loads(blob.decode())
    t_parse = time.perf_counter()
    return {
        "join_ms": round((t_join - t0) * 1000, 3),
        "parse_ms": round((t_parse - t_join) * 1000, 3),
        "keys": sorted(parsed)[:4],
    }

def write_graph(rows, path="queue_depth.csv"):
    with open(path, "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=["i", "t_ms", "held_bytes", "parsed"])
        w.writeheader()
        w.writerows(rows)

The incremental loop records held bytes after each chunk. The buffered path joins first, then parses once. Which graph would you rather defend in a review?

The gap_s knob is only for a local fixture. Do not copy that sleep into a live read callback. A probe that sleeps will invent the stall it reports.

I would feed both paths the same fake chunks in a unit test. No network, no hero number, no borrowed benchmark. The test only checks shape: held bytes should fall when a line closes.

def test_held_bytes_fall_on_close():
    chunks = [b'{"a":1}', b"\n", b'{"b":2}', b"\n"]
    rows = sample_loop(chunks)
    assert rows[0]["held_bytes"] > 0
    assert rows[1]["held_bytes"] == 0
    assert rows[1]["parsed"] is True
    assert rows[3]["held_bytes"] == 0

That test will not make your product faster. It stops you from lying about the drain. Can you say when the queue actually emptied?

For a live call, wrap the socket read the same way. Stamp accept, first chunk, each chunk, and the parse that stuck. Write the CSV, then plot held bytes against t_ms.

I keep that plot and I throw the dashboard screenshot away. A screenshot has no axes I can recompute. A CSV still argues with me next week.

What should the line do if the reader is healthy? Held bytes should sawtooth, up on a chunk and down on a close. A staircase that never falls means you are buffering the world.

A flat line at zero can also be a lie. You might be parsing so late that samples never see the pile. Sample inside the read callback, not after the handler returns.

Do not sleep inside that callback to be fair. A sleep call moves the bottleneck into your probe. I want the probe thinner than the drain it watches.

If you already stream server-sent events, split on the blank line. Parse one finished event, then release those bytes. Leave the rest in the deque for the next read.

def pop_events(q):
    blob = b"".join(q)
    parts = blob.split(b"\n\n")
    done, tail = parts[:-1], parts[-1]
    q.clear()
    if tail:
        q.append(tail)
    return done

def release_or_flag(blob):
    try:
        return json.loads(blob.decode()), False
    except json.JSONDecodeError:
        return None, True

That function is the whole lesson in miniature. Completed events should leave the deque at once. An unfinished tail stays until the next chunk.

Why would you join the tail to a finished event and wait? Now the lab question, which is smaller than people make it. You need a repeatable caller and a place to run the harness.

You do not need a new cluster diagram on a whiteboard. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode has free model access and a free server option.

Those are the only product facts I will lean on. Those two facts matter only as a cheap place to repeat the same call. Point the harness at a free model endpoint if you already have one.

The free server is a quiet box, not a proof of scale. I am not naming a model, a quota, a box size, or a deadline. Those details move, and I have no primary note in front of me.

If a landing page disagrees with this paragraph, trust the page. Do not turn that free server into a load cannon. A profile of one reader is not a capacity test.

You will learn your buffer, not their scheduler. Who should skip this note and walk away? Skip it if you are tuning kernels on a GPU you own.

Skip it if you need a contractual latency promise for customers. Skip it if your bug is auth, routing, or a bad prompt. A queue graph will not fix a wrong tool schema.

Skip it if you cannot store the CSV without secrets in the body. Strip prompts and outputs before you plot anything. Log lengths, flags, and held bytes, nothing else.

Would you paste a customer message into a graph title? Compare the two paths on identical chunks, then stop. If incremental parse drops held bytes and time-to-action, keep it.

If it does not, your stall lives somewhere else. The graph just told you that, so believe it. Treat it as a flashlight, not a platform.

Frameworks grow knobs, and knobs hide the next full buffer. Watch the exporter, because it can plug the sink. A trace shipper that blocks on every chunk becomes the new sink plug.

Sample, buffer the samples, and flush on a timer outside the read path. One more trap sits in JSON libraries that read to EOF. That call is a buffered path wearing a streaming costume.

Check the function name before you brag about events. A small experiment plan is what keeps me honest. Label it unexecuted until you run it on your machine.

Build thirty fake chunks, and close only half of them. Run the incremental path and the buffered path back to back. Repeat the pair five times on the same process.

Change only one variable: when you release bytes. Do not change chunk size in the same breath. If you change two knobs, the graph cannot accuse either one.

Write the winner down as a sentence, not a vibe. A fair result sentence names the knob and the direction. It should not name a vendor, a medal, or a percent.

If you cannot write that sentence, you do not have a result. This is still a local fixture, not a production proof. A quiet laptop fan is not a service level.

The release_or_flag helper above is the failure row. Inject one chunk that is not valid JSON. The parser should record an error flag and release the bad event.

If it keeps the bad bytes forever, you built a second leak. That leak will look like model slowness tomorrow morning. Ask yourself who pays when the queue never falls.

Your user pays in waiting, and your process pays in RAM. The free server does not pay that bill for you.

I keep one graph from the run: held bytes over time. I do not keep a mean, a medal, or a thread of best-of-three. Means clock out when a single stuck reader fills the sink.

If the sawtooth never comes back down, fix the release first. Then look at the model, the wire, and the prompt. The order matters, because a full buffer makes every upstream look guilty.

You can rerun this tomorrow without a new vendor story. Keep the same chunks, the same CSV columns, the same question. Did the queue fall when the event closed?

That is the note I would tape above the monitor. It is not a slogan, and it is not a score. Just the graph that still fits on one axis.

I Kept the Run Index. Warmup Took the Win.

Dakota Lin — Sat, 10 Oct 2026 17:55:14 +0000

The second run is not a real speedup. It is a warm socket with a long memory. I kept the run-index graph, and warmup took the credit.

You changed one small line in the client. The next call came back and felt fast. Did the model improve, or did the pipe stay open?

I stopped trusting the warm feeling in my gut. I timed every loop pass in strict run order. I labeled cold and warm before I touched the patch.

This note is a method, not a trophy case. The script below is a proposal you can run. I will not invent a chart from a wish.

The graph I kept

What should that kept graph even show you? Not one proud bar you saved after lunch. It should show latency against the run index.

Run zero is the stranger at the door. It pays for DNS, TLS, and a cold import. Run one may still be kindling, not fire.

I drop those two from the comparison on purpose. I decide that rule before I see the bars. If I pick the rule after, I am shopping.

A free server is useful here for one reason. Your laptop fan is a liar with a motor. It speeds up, then it thermal-throttles, then it lies again.

I would point the same harness at MonkeyCode for repeats. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access is what lets that loop repeat.

A free server option keeps the laptop fan out. I am not naming a model, a quota, or a box. Those facts move, and I will not freeze a rumor.

Read the live page before you trust a limit. A free pool can queue you behind strangers. That wait is not your patch, and it is not a product score.

The loop you can run

Here is the harness I actually want to run. It is stdlib only, so you can audit every line. Point it at a URL you are allowed to call.

#!/usr/bin/env python3
"""Label cold versus warm, and keep the run index."""

import argparse
import csv
import json
import os
import time
import urllib.request

def one_call(url: str, payload: bytes, timeout: float) -> dict:
    wall_start = time.perf_counter_ns()
    cpu_start = time.process_time_ns()
    req = urllib.request.Request(
        url,
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        body = resp.read()
        status = resp.status
    wall_ns = time.perf_counter_ns() - wall_start
    cpu_ns = time.process_time_ns() - cpu_start
    return {
        "status": status,
        "wall_ms": wall_ns / 1e6,
        "cpu_ms": cpu_ns / 1e6,
        "bytes_out": len(payload),
        "bytes_in": len(body),
    }

def phase_for(index: int) -> str:
    if index == 0:
        return "cold"
    if index == 1:
        return "kindling"
    return "warm"

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--url", required=True)
    parser.add_argument("--runs", type=int, default=12)
    parser.add_argument("--out", default="runs.csv")
    parser.add_argument("--timeout", type=float, default=60.0)
    args = parser.parse_args()
    payload = json.dumps({
        "prompt": "Reply with one short sentence about queues.",
        "max_tokens": 32,
    }).encode()
    rows = []
    for i in range(args.runs):
        row = one_call(args.url, payload, args.timeout)
        row["run_index"] = i
        row["phase"] = phase_for(i)
        row["label"] = os.environ.get("RUN_LABEL", "baseline")
        rows.append(row)
    fields = [
        "label", "run_index", "phase", "status",
        "wall_ms", "cpu_ms", "bytes_out", "bytes_in",
    ]
    with open(args.out, "w", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)
    warm = [r["wall_ms"] for r in rows if r["phase"] == "warm"]
    cold = rows[0]["wall_ms"]
    print(f"cold_wall_ms={cold:.1f}")
    if warm:
        mid = sorted(warm)[len(warm) // 2]
        print(f"warm_median_ms={mid:.1f}")
        print(f"cold_over_warm={cold / mid:.2f}")
    print("kept_graph=wall_ms by run_index, split on label")

if __name__ == "__main__":
    main()

The JSON body is a stand-in, not a vendor schema. Swap fields to match the server you actually call. Treat every printed number as unexecuted until you run it.

Set a label, then run the baseline before the patch. Change one thing, then run the patched label. Same URL, same payload, and the same run count.

export RUN_LABEL=baseline
python3 harness.py --url "$MODEL_URL" --runs 12 --out baseline.csv
export RUN_LABEL=patched
python3 harness.py --url "$MODEL_URL" --runs 12 --out patched.csv
python3 sketch.py baseline.csv patched.csv

The sketch script is the graph I keep. Each row is a run, not a vibe. Cold rows stay on the page so I cannot forget them.

#!/usr/bin/env python3
"""Sketch wall time by run index from your own CSV."""

import csv
import sys

def load(path: str) -> list:
    with open(path, newline="") as fh:
        return list(csv.DictReader(fh))

def main() -> None:
    rows = []
    for path in sys.argv[1:]:
        rows.extend(load(path))
    if not rows:
        raise SystemExit("pass at least one csv")
    peak = max(float(r["wall_ms"]) for r in rows) or 1.0
    for r in rows:
        n = int(round(40 * float(r["wall_ms"]) / peak))
        bar = "#" * max(n, 1)
        label = r["label"][:8]
        idx = int(r["run_index"])
        phase = r["phase"][:8]
        print(f"{label:8} {idx:02d} {phase:8} {bar}")

if __name__ == "__main__":
    main()

I omit sample bars so nobody copies a fake chart. Run the sketch yourself, then argue with your own rows.

How do I read that sketch without lying? I cover the warm rows with one hand. I ask whether the cold row moved at all.

If only the warm rows shrink, I blame the socket. Connection reuse will do that trick every time. A prompt cache will smile the same way.

If cold and warm both shrink after the patch, I lean in. I still want a second hour, not a second guess. One quiet minute is not a whole season.

Watch the CPU column while you stare at the wall. If CPU stays tiny while wall grows, you waited. You did not win a secret compute prize.

That split is a guardrail, not the whole story. I already know tails can hide in the network. Today I care who gets credit for the drop.

Do not average the cold row into the warm rows. The mean will hide the stranger and bless the regulars. I want the run index, so order stays visible.

A median of warm rows is fine as a note. It is not fine as a public headline. Headlines eat the label, then they eat the truth.

I also keep bytes out and bytes in. A shorter reply is not a faster client. Did you change the work, or did you change the scale?

If bytes_in falls and wall falls with it, stop. You shrank the job, and then you clapped. That is a diet, not a faster kitchen.

Same payload both times, or the graph is theater. I pin the prompt text in the script for that reason. I do not improve the prompt between the labels.

What about retries, those quiet little cheats of time? A retry can turn one miss into two bills of time. I log status, and I do not drop errors from the file.

A failed row stays in the sketch as a gap marker. Deleting it makes the warm median look brave. Would you delete a crash from a race recap?

Timeouts need a pre-set budget before you start. I pass a timeout on purpose, and I write it down. If I raise it mid-test, I started a new experiment.

The free server can sleep between your bursts. A pause for coffee becomes a new cold start. I note the gap, or I pretend the room never cooled.

Shared capacity is a neighbor, not a lab bench. Someone else's job can sit in your wall time. That is why this loop cannot rank vendors.

Who should walk away

I would not use this method for a public bake-off. I would not use it to promise a speedup in a launch note. I would not use it if I need pinned cores and a quiet NIC.

You should skip it if your boss wants a single number. You should skip it if you cannot store the raw CSV. You should skip it if the payload is still changing shape.

Students can still learn the habit from the shape. Run it once against a local stub if you have no server. The stub will not prove a model, but it will prove your labels.

A local stub is a metronome, not an opponent. It teaches you whether run zero looks different. If run zero looks the same, your timer is blind.

Here is a tiny stub you can hate in private. It sleeps longer on the first hit, then less. That sleep is fake, and the lesson is not.

#!/usr/bin/env python3
"""Local metronome for labels. First hit sleeps longer."""

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import time

HITS = {"n": 0}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        n = HITS["n"]
        HITS["n"] = n + 1
        delay = 0.4 if n == 0 else 0.05
        time.sleep(delay)
        body = b'{"ok":true}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt: str, *args) -> None:
        return

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

The stub listens on port 8765 on localhost only. Start it in one shell, then aim the harness there. Call that stub before you touch a remote pool.

If your sketch cannot see the fake cold hit, stop. Your graph is broken, and the model is innocent.

I like that order more than a hot take. I put tooling before talent, every single time. Labels come before pride, or the bar lies.

When the stub looks honest, I move the URL outward. I keep the same CSV columns, so the eye does not relearn. A new column is a new argument, and I am tired.

How many runs is enough for a note like this? Twelve runs make a sketch, not a paper. If the bars jump like popcorn, I add runs.

I do not add adjectives to a noisy sketch. I also refuse to tune after I see a winner. Tuning the timeout to flatter the patch is a costume.

Take the costume off, or do not publish the photo. Store the script hash next to the CSV if you can. A one-line drift will impersonate a real victory.

I want the blame to land on a commit, not on a mood.

git rev-parse --short HEAD | tee run_commit.txt
python3 -c "import hashlib, pathlib; p=pathlib.Path('harness.py'); print(hashlib.sha256(p.read_bytes()).hexdigest())" | tee run_files.txt

That pair is my alibi when the bar looks pretty. Without it, future me will argue with past me. Past me cheats when the bar looks good.

Limitations sit in the room even when the sketch looks clean. Free model access can change shape without a speech. Free server time can queue, cap, or vanish for an hour.

I am not claiming those options are permanent. I am not claiming a region, a chip, or a rate. If the operator page disagrees with this note, believe the page.

Clock resolution can flatter a tiny client change. A sub-millisecond win on a network hop is noise. Do not frame noise and hang it in the hall.

JSON parsing can dwarf a small logic tweak. If you only time the function you edited, you will miss the dump. Time the whole call, then decide where to cut.

I still would not start a rewrite from one sketch. I would start with a sharper question instead. Which run got faster, and which run only got familiar?

Keep the ugly cold bar in the picture. Cropping it is how warmup steals the win. That theft is common, and I am done applauding it.

If you need a place to repeat the loop, start at MonkeyCode. Read the live limits, then keep the run-index graph.

I Logged Three Clocks. The Model Sat Still.

Dakota Lin — Thu, 08 Oct 2026 07:42:16 +0000

I blamed the model before I split the clocks. My first blame was lazy and flat wrong. The queue and the wire owned the wait.

You feel a coding agent stall and you swear. You swear at weights you never even measured. Have you timed the three gaps, or just the mood?

I mean three clocks, not one sad total. I log send time, first byte, and done. If those three stay glued, your graph is a rumor.

I wrote a tiny harness and refused the folklore. Same prompt, same bytes, and two endpoints. The only story I trust is a span I can replay.

I point one run at MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free server is a hop, not a trophy.

I do not print a quota, a chip, or a lease. Those facts move fast, and stale numbers lie. You should read the live page before you quote one.

I would keep the other run on a box I already own. That contrast is the whole experiment, nothing fancier. If both hops look alike, the model was never the villain.

Here is the harness I keep beside the notes. Treat the JSON body as a local stand-in, not their API. Adapt every field to docs you verified today.

#!/usr/bin/env python3
"""Three-clock sketch. Unexecuted example. No live timings claimed."""
import json, os, time, urllib.request

ENDPOINT = os.environ["MODEL_ENDPOINT"]
TOKEN = os.environ.get("MODEL_TOKEN", "")
PROMPT = open("fixture.txt", encoding="utf-8").read()

def one_shot(body: bytes) -> dict:
    req = urllib.request.Request(ENDPOINT, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    if TOKEN:
        req.add_header("Authorization", "Bearer " + TOKEN)
    t_send = time.perf_counter()
    with urllib.request.urlopen(req, timeout=12) as resp:
        t_headers = time.perf_counter()
        raw = resp.read()
        t_done = time.perf_counter()
    return {
        "to_headers_ms": round((t_headers - t_send) * 1000, 1),
        "body_ms": round((t_done - t_headers) * 1000, 1),
        "total_ms": round((t_done - t_send) * 1000, 1),
        "bytes": len(raw),
    }

if __name__ == "__main__":
    payload = json.dumps({"prompt": PROMPT}).encode()
    for i in range(5):
        row = one_shot(payload)
        row["i"] = i
        row["cold"] = i == 0
        print(json.dumps(row))
        time.sleep(1.0)

Run it only against an endpoint you may call. Keep the token in the environment, never in git. Five loops are a sketch, not a promise to the wire.

export MODEL_ENDPOINT="https://example.invalid/complete"
export MODEL_TOKEN="set-me-outside-the-repo"
python3 three_clocks.py | tee rows.jsonl

I tag each row warm or cold before I plot it. A cold socket will impersonate a slow model. Have you been graphing a handshake and calling it smart?

Header time on a buffered call is not token time. The server may hold every byte until the end. If you need the first token, you must force a stream.

So the second script reads small chunks on purpose. It stamps the clock when the first chunk lands. The rest of the bar is whatever remains until close.

def stream_gaps(body: bytes) -> dict:
    req = urllib.request.Request(ENDPOINT, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    if TOKEN:
        req.add_header("Authorization", "Bearer " + TOKEN)
    t_send = time.perf_counter()
    t_first = None
    with urllib.request.urlopen(req, timeout=12) as resp:
        while True:
            chunk = resp.read(64)
            if not chunk:
                break
            if t_first is None:
                t_first = time.perf_counter()
    t_done = time.perf_counter()
    if t_first is None:
        t_first = t_done
    return {
        "to_first_ms": round((t_first - t_send) * 1000, 1),
        "after_ms": round((t_done - t_first) * 1000, 1),
    }

If the server never flushes, those two gaps collapse. You just measured a buffered body with extra steps. Check the docs for a real stream flag before you blame the hop.

I print an ASCII bar and I refuse a theme. Two bars per row, first chunk and the rest. The picture is a receipt, not a ranking post.

def bar(ms: float) -> str:
    cells = int(ms // 25)  # 25 ms per mark, then stop the ink
    return "#" * max(0, min(cells, 40))

def render(rows: list) -> str:
    lines = []
    for row in rows:
        lines.append("%s first %s %s" % (row["i"], bar(row["to_first_ms"]), row["to_first_ms"]))
        lines.append("%s rest  %s %s" % (row["i"], bar(row["after_ms"]), row["after_ms"]))
    return "\n".join(lines)

Look at a row before you touch the prompt. If the first bar dwarfs the rest, the model sat still. Why rewrite instructions while the queue eats the span?

Picture a diner with a fast cook and a slow bell. You yell at the kitchen because the plate is late. The ticket was stuck on the rail the whole time.

That is this graph, with worse coffee and more JSON. The cook is the work after the first chunk. The rail is queue, DNS, TLS, and your own retry nap.

I keep the graph that shows the rail, not the cook. A teammate can argue with a bar of hashes. A teammate cannot argue with your mood from Tuesday.

When the rail is long, I pull the call off the key path. Batch the review after lunch, or night the notes. Do not park a remote hop inside a save hook.

A free server is a fair night shift for that batch. It is a bad stand-in for a compiler. Would you block one keystroke on a queue you do not own?

I still want that hop for work that can wait. A diff read after lunch can wait a little. Autocomplete under a finger cannot, and should not pretend.

One request in flight is the whole load I allow. A parallel fan-out would only measure my stampede. Shared free capacity is a commons, not a drum I get to hit.

Secrets do not ride a hop I do not operate. If the fixture holds private source, I strip it first. A friendly free box is still someone else's machine.

I also refuse one evening as a service level. Other people arrive later, and the rail grows. Your pretty bar from tonight can lie by morning.

Shared ids beat two clocks that never met. Put one request id on the client row and the log. Without that id, you compared neighbors, not one trip.

Here is the gate I run before I trust a bar. Same id, same byte count, and a cold flag. If any one is missing, I throw that row out.

def keep(row: dict) -> bool:
    needed = ("req_id", "bytes", "cold", "to_first_ms", "after_ms")
    return all(k in row and row[k] is not None for k in needed)

Who should skip this approach, and stay skipped? Skip it when policy forbids a third-party model hop. Skip it when you need a model menu I will not invent.

Skip it when your budget is one human keystroke. Skip it when you wanted a winner and a medal. I kept a method, not a ranking, and that was the point.

What should you change after one honest graph? Change the path, the batching, or the timeout. Do not change the prompt until the rail gets short.

I set the client timeout to a number I can say aloud. Twelve seconds makes a stuck queue visible and boring. Sixty seconds hides that same queue inside a spinner.

Retries are where a clean graph goes to die. A hidden second try adds a full extra rail. Log the attempt index, or the total will confess to fiction.

A total can include two naps and still look sincere. The model may have finished on the first try. Which clock did you paste into the incident note?

DNS and TLS deserve blame on a fresh process. Reuse the connection when the API actually allows it. Otherwise every row opens with a handshake costume party.

I store a warm series beside a separate cold row. Mixing them is how a free hop gets framed. The costume party is not the model's personality.

Payload size will fake a generation problem too. A novel-length diff is a shipping choice you made. Trim the fixture before you indict the remote side.

I keep the fixture in the repo next to the script. A teammate should rerun the same bytes, not a vibe. If you edit the fixture, you started a different experiment.

The graph stays ugly so I cannot decorate a miss. No badge and no theme, and no weekday duel. Two bars, a request id, and a cold flag are enough.

If the rest bar grows and the first stays flat, I open the prompt. That is the only time the model owes me an answer. Until then it can sit still, and it should.

Free model access does not mean a queue-free lane. A shared server spends time on other callers. Treat that wait as traffic, not as a broken file of weights.

I will not tell you the wait is small or large. I did not run this harness for a published score. Print your own rows, then keep the graph that matches them.

Check the project page on the day you actually run. Names, limits, and hardware are their facts, not mine. If this note and that page disagree, believe the page.

Point that sketch at their free server when the wait fits. Read the live terms, then keep or drop the endpoint. I would rather you discard it than quote a number I never printed.

I Profiled the Agent. Rebuild Ate the Clock.

Dakota Lin — Wed, 23 Sep 2026 16:52:41 +0000

The language model was not my real bottleneck. Prompt rebuild ate the clock on every later round. I spent days blaming inference like a fool.

Have you ever tuned the model and missed the copy? I wanted one graph I could actually keep. A zoo of dashboards would have hidden the crossing.

I plotted rebuild time against a stubbed model nap. This is a lab note, not a customer war story. I did not harvest production traces for this.

I wrote a harness you can run tonight. The plot is an agent loop with tools. Each tool returns a chunky JSON payload on purpose.

Then the loop concatenates history into the next prompt. After that step it calls the model again. In the stub, the model is just sleep.

On the wire, the model is a network hop. The span names stay identical either way though. Why named spans and not a single timer?

Averages already lied to me enough last month. I wanted stackable intervals with very boring names. I clocked serialize, tool, rebuild, and model separately.

They all land in one boring CSV file. The CSV is the graph I kept. Here is the core of that harness.

#!/usr/bin/env python3
"""Agent-loop span profiler. Stub model. Rebuild grows."""
from __future__ import annotations

import csv
import json
import os
import time
from dataclasses import dataclass

@dataclass
class RoundRow:
    n: int
    serialize_ms: float
    tool_ms: float
    rebuild_ms: float
    model_ms: float
    prompt_chars: int


def fake_tool(round_n: int) -> tuple[dict, float]:
    t0 = time.perf_counter()
    time.sleep(0.005)  # 5ms stand-in for a local tool
    payload = {
        "round": round_n,
        "files": [
            {"path": f"src/mod_{i}.py", "preview": "x" * 2000}
            for i in range(50)
        ],
        "log": "ok" * 400,
    }
    return payload, (time.perf_counter() - t0) * 1000.0


def serialize_tool_result(payload: dict) -> tuple[str, float]:
    t0 = time.perf_counter()
    text = json.dumps(payload, separators=(",", ":"))
    return text, (time.perf_counter() - t0) * 1000.0


def rebuild_prompt(history: list[str], tool_text: str) -> tuple[str, float]:
    t0 = time.perf_counter()
    history.append("TOOL:" + tool_text)
    prompt = ""
    # Quadratic on purpose: a join people actually write.
    for i in range(len(history)):
        prompt = "\n".join(history[: i + 1])
    return prompt, (time.perf_counter() - t0) * 1000.0


def stub_model(prompt: str) -> tuple[str, float]:
    t0 = time.perf_counter()
    time.sleep(0.040)  # ruler, not a model benchmark
    _ = len(prompt)
    return "call_tool", (time.perf_counter() - t0) * 1000.0


def run_loop(rounds: int, model_fn) -> list[RoundRow]:
    history = ["SYS:you are a coding agent", "USER:find the leak"]
    rows: list[RoundRow] = []
    for n in range(1, rounds + 1):
        payload, tool_ms = fake_tool(n)
        tool_text, ser_ms = serialize_tool_result(payload)
        prompt, rebuild_ms = rebuild_prompt(history, tool_text)
        _, model_ms = model_fn(prompt)
        rows.append(
            RoundRow(n, ser_ms, tool_ms, rebuild_ms, model_ms, len(prompt))
        )
    return rows


def write_csv(path: str, rows: list[RoundRow]) -> None:
    with open(path, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(
            ["round", "serialize_ms", "tool_ms", "rebuild_ms",
             "model_ms", "prompt_chars"]
        )
        for r in rows:
            w.writerow([
                r.n,
                f"{r.serialize_ms:.3f}",
                f"{r.tool_ms:.3f}",
                f"{r.rebuild_ms:.3f}",
                f"{r.model_ms:.3f}",
                r.prompt_chars,
            ])


def ascii_graph(rows: list[RoundRow]) -> str:
    lines = ["round  rebuild#  model.", "-----  --------  ------"]
    for r in rows:
        rb = max(1, int(round(r.rebuild_ms)))
        md = max(1, int(round(r.model_ms / 2)))
        lines.append(f"{r.n:5d}  {'#' * rb}  {'.' * md}")
    return "\n".join(lines)

I run the stub path with a plain command. Nothing fancy. No extra flags.

python agent_spans.py

Wire main to run_loop(12, stub_model) and write agent_spans.csv. The stub sleeps for forty milliseconds each round. That sleep is a ruler, not a benchmark.

Please do not quote it as model speed. What should you watch on a quiet machine? Rebuild starts tiny, almost like a rounding error.

Then the tool payload lands and history snowballs. By later rounds the copy work gets rude. The snowball is the whole point here.

If rebuild crosses that forty millisecond ruler, keep the graph. That crossing is the only picture I kept. Everything else was noise I threw away.

I also wrap the same loop with cProfile. Spans tell a timeline, and functions tell a culprit. json.dumps often walks into the spotlight first here.

import cProfile
import io
import pstats

def profile_loop() -> None:
    pr = cProfile.Profile()
    pr.enable()
    run_loop(12, stub_model)
    pr.disable()
    buf = io.StringIO()
    stats = pstats.Stats(pr, stream=buf)
    stats.sort_stats("cumulative").print_stats(15)
    print(buf.getvalue())

Did you expect the model function to win? I did. It lost on the stub path.

That embarrassment is why I kept the picture. Export pstats if you want a flame-style view. A wide plateau sits under rebuild_prompt on stub runs.

The stub_model bar stays thin and honest. I still prefer the stacked CSV over a flame. It pastes into notes without a graphics stack.

The insult survives the copy-paste just fine. Sample stub output will look locally noisy. Treat the shape as the lesson, not the digits.

round  rebuild#  model.
-----  --------  ------
    1  ##        ....................
    4  #####     ....................
    8  ##########  ....................
   12  ##################  ....................

Your bars will move. That is expected. Pin the payload size if you need a fair fight. Inflate the tool payload until the bars actually fight.

Now comes the remote path with the same four spans. The model function POSTs the prompt over HTTP. I do not wrap a vendor SDK in this note.

I pointed this harness at MonkeyCode because free model access and a free server option let the loop leave my laptop without a GPU box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not publishing their latency, model names, or quotas. I am not calling this a product benchmark either. Point the client at any URL you actually control.

import urllib.request

def remote_model(prompt: str) -> tuple[str, float]:
    url = os.environ.get("AGENT_MODEL_URL", "").strip()
    if not url:
        raise SystemExit("set AGENT_MODEL_URL to an endpoint you control")
    body = json.dumps({"prompt": prompt}).encode("utf-8")
    req = urllib.request.Request(url, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=30) as resp:
        raw = resp.read()
    return raw.decode("utf-8", errors="replace")[:200], (time.perf_counter() - t0) * 1000.0

Set AGENT_MODEL_URL and run the same CSV path. Compare stub bars against wire bars on your graph. Keep the picture and throw away vendor adjectives.

export AGENT_MODEL_URL="http://127.0.0.1:8080/prompt"
python -c "from agent_spans import run_loop, remote_model, write_csv, ascii_graph
rows = run_loop(8, remote_model)
write_csv('agent_spans_wire.csv', rows)
print(ascii_graph(rows))"

Watch the first remote round with extra suspicion. TLS and DNS can steal that opening lap. Queue delay on a shared box can steal more.

After warmup, rebuild may still be the growth term. Why? You resend the whole history every round. Streaming would change this story quite a bit.

This harness does not stream a single token. If your real agent streams, instrument the decoder. Do not borrow my stub graph for that case.

The quadratic join is a microscope, not advice. Real history should be appended, not rebuilt from prefixes. I left the slow join in so the graph has teeth.

Swap it for a single "\n".join(history) after you see the crossing. Then run the CSV again on the same machine. Did the rebuild bar collapse like a cheap tent?

If it did, you just caught the copy. If it did not, stare at json.dumps next. Huge tool payloads serialize like wet cement.

This method will not find GPU kernel stalls. It will not find tokenizer weirdness on its own. cProfile also perturbs the thing you measure.

Laptop clocks are noisy and fans make them worse. Run it thrice before you believe a crossing. Pin CPU frequency if you are being precious.

A free shared server adds queue time you cannot see. Without server-side traces you must not blame the model. You can only blame the round trip you measured.

Skip this if you already have tracing in the agent. Skip this if you need legal vendor comparisons. Skip this if your bottleneck is inside CUDA.

I would also skip it for tiny prompts. A twenty token chat will not show rebuild. Inflate the tool payload until the bars fight.

That inflation is a lab trick, not production. Real tools should return less, not more. The harness is a microscope, not a product.

Clone the idea, not my sleep constant. Change the payload. Change the join. Keep one graph.

When rebuild crosses your ruler, you have a bug. Fix the copy. Then go bother inference again.