DEV Community: Dakota Lin

I Stabilized the Prefix. The Cache Came Back.

Dakota Lin — Thu, 15 Oct 2026 11:22:00 +0000

I Stabilized the Prefix. The Cache Came Back.

My agent loop got slower with every turn. The model was innocent, and so was the network.

The culprit was a timestamp I put in my own prompt. One field, changing every turn, was quietly forcing a full prefill on every request.

Sound familiar? Everyone argues about which model is fastest this week. Almost nobody asks why their own prompt keeps moving.

The symptom nobody graphs

Turn one feels fine. Turn twenty feels wrong, and you start pricing faster inference.

Then you open a fresh session and it snaps back. That gap is your first clue, and it points at your context, not the GPU.

Keep the graph anyway. A flat model-time line next to a rising wall-clock line is the signature you want.

Log the field that decides

Most providers expose some form of cached-token count in the response usage block. Field names differ, so adapt this to yours.

If that number holds steady, your prefix is being reused. If it collapses to zero turn over turn, your prefix moved.

# log one line per turn; usage field names vary by provider
for turn, resp in enumerate(stream, start=1):
    u = resp.usage
    print(turn, u.input_tokens, u.cached_tokens, hash_prefix(messages))

That fourth column is the trick. I compute the fingerprint myself instead of trusting the provider's number.

Fingerprint the prefix

The cache key is basically your serialized prompt bytes. So hash them, and strip fields that change for no good reason.

import hashlib, json

VOLATILE = {"ts", "timestamp", "run_id", "trace_id", "started_at"}

def strip_volatile(obj):
    if isinstance(obj, dict):
        return {k: strip_volatile(v) for k, v in sorted(obj.items())
                if k not in VOLATILE}
    if isinstance(obj, list):
        return [strip_volatile(v) for v in obj]
    return obj

def hash_prefix(messages):
    blob = json.dumps(strip_volatile(messages), sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:12]

Twelve hex characters are plenty. You are not securing anything here, you are detecting drift.

The diff that ends the argument

Capture two consecutive request bodies, then diff them. It takes a minute and it settles the debate.

diff <(jq -S . turn1.json) <(jq -S . turn2.json) | head -40

In my case the diff was small and ugly. A started_at, an absolute working directory, and a tool list whose order followed dictionary iteration.

None of that was information the model needed. All of it was enough to invalidate the prefix.

A test that fails when you regress

Freeze the behaviour in a test. Two builds with identical logical state must hash the same, even when wall-clock time differs.

def test_prefix_is_stable():
    a = hash_prefix(build_messages(state, now="2026-01-01T00:00:00Z"))
    b = hash_prefix(build_messages(state, now="2026-01-02T00:00:00Z"))
    assert a == b, "prefix moved; expect a full prefill on every turn"

Run it in CI. When it fails, you find out before the invoice does.

The fix itself is boring. Treat the prefix as append-only, and push anything volatile into a trailing block. Canonicalise tool ordering with an explicit sort key.

Where MonkeyCode fits

I used MonkeyCode for two reasons here. Its free model access meant I could run the two-payload A/B without spending a paid key on a failed experiment. The free server option let the harness sit close to the endpoint, so network jitter stopped polluting the cache comparison.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding agent project, and the free tier is the part that matters for this workflow. I am told the free allowance is 10M tokens as of this writing, alongside a free server option. Verify that on the project page before you plan around it, because quotas move.

The experiment is simple. Run your harness once with volatile fields left in the prompt. Run it again with the normalizer switched on. Same code path, one flag.

One run should show cached tokens holding steady. The other should show them dropping to zero, with latency following behind. You do not need my numbers. You need yours.

Who should skip this

Skip it if your prompts are short. Prefix caching usually has a minimum length, and below that you are optimising nothing at all.

Skip it if you genuinely need fresh retrieval every turn. Restructure the prompt instead of deleting information the model depends on.

Watch the other direction too. A warm cache can hide a stale context bug, because the model keeps reusing something you forgot to update.

And if your provider does not do prefix caching, this whole article is a nice theory. Read the docs first.

So what is actually slow in your loop? Model time, network time, or your own prefix drifting under your feet?

If you want to run the two-payload test before buying keys, the MonkeyCode free tier is one honest way in. Go try it, then graph your own prefix hash.

I Kept the Queue Graph. The Buffer Sat Full.

Dakota Lin — Sun, 11 Oct 2026 21:36:04 +0000

The slow part is often your buffer, not the model. I keep the queue graph, and I drop the mean. Does a pretty average still hide a stuck reader?

First-token timing is a different question, so I leave it. This note watches bytes sitting inside your own process. Why hold a full body when chunks already arrived?

Picture a sink with the tap still running. The drain is your parser, and it waits for a closer. The rising water is queue depth, not model skill.

I do not need a new headline to see this. News cycles move fast, and stuck buffers do not. Have you ever blamed the model for your own unread socket?

A free endpoint makes the mistake easier to repeat. You can call it all afternoon and still learn nothing. What did you record besides a single elapsed number?

I want four marks on one request, not a leaderboard. Accept starts when the socket is ready for the body. Read ends when the next chunk lands in user space.

Parse is the moment a complete event becomes a struct. Handoff is when your agent can act on that struct. If parse waits for the final byte, your queue lies fat.

That last shape is the bug I keep drawing. The model may already be idle on the far side. Your process is still clutching a string like a trophy.

Here is the harness I actually want on disk. It is a method, not a trophy from a secret lab. Run it yourself, and do not borrow my blanks.

import json, time, csv
from collections import deque

# Unexecuted example. No network. No secrets. No claimed timings.

def sample_loop(chunks, gap_s=0.0):
    q = deque()
    rows = []
    t0 = time.perf_counter()
    held = 0
    for i, chunk in enumerate(chunks):
        t_read = time.perf_counter()
        q.append(chunk)
        held += len(chunk)
        parsed = None
        if chunk.endswith(b"\n"):
            blob = b"".join(q)
            parsed = json.loads(blob.decode())
            q.clear()
            held = 0
        rows.append({
            "i": i,
            "t_ms": round((t_read - t0) * 1000, 3),
            "held_bytes": held,
            "parsed": parsed is not None,
        })
        if gap_s:
            time.sleep(gap_s)
    return rows

def buffered_path(chunks):
    t0 = time.perf_counter()
    blob = b"".join(chunks)  # holds every chunk before parse
    t_join = time.perf_counter()
    parsed = json.loads(blob.decode())
    t_parse = time.perf_counter()
    return {
        "join_ms": round((t_join - t0) * 1000, 3),
        "parse_ms": round((t_parse - t_join) * 1000, 3),
        "keys": sorted(parsed)[:4],
    }

def write_graph(rows, path="queue_depth.csv"):
    with open(path, "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=["i", "t_ms", "held_bytes", "parsed"])
        w.writeheader()
        w.writerows(rows)

The incremental loop records held bytes after each chunk. The buffered path joins first, then parses once. Which graph would you rather defend in a review?

The gap_s knob is only for a local fixture. Do not copy that sleep into a live read callback. A probe that sleeps will invent the stall it reports.

I would feed both paths the same fake chunks in a unit test. No network, no hero number, no borrowed benchmark. The test only checks shape: held bytes should fall when a line closes.

def test_held_bytes_fall_on_close():
    chunks = [b'{"a":1}', b"\n", b'{"b":2}', b"\n"]
    rows = sample_loop(chunks)
    assert rows[0]["held_bytes"] > 0
    assert rows[1]["held_bytes"] == 0
    assert rows[1]["parsed"] is True
    assert rows[3]["held_bytes"] == 0

That test will not make your product faster. It stops you from lying about the drain. Can you say when the queue actually emptied?

For a live call, wrap the socket read the same way. Stamp accept, first chunk, each chunk, and the parse that stuck. Write the CSV, then plot held bytes against t_ms.

I keep that plot and I throw the dashboard screenshot away. A screenshot has no axes I can recompute. A CSV still argues with me next week.

What should the line do if the reader is healthy? Held bytes should sawtooth, up on a chunk and down on a close. A staircase that never falls means you are buffering the world.

A flat line at zero can also be a lie. You might be parsing so late that samples never see the pile. Sample inside the read callback, not after the handler returns.

Do not sleep inside that callback to be fair. A sleep call moves the bottleneck into your probe. I want the probe thinner than the drain it watches.

If you already stream server-sent events, split on the blank line. Parse one finished event, then release those bytes. Leave the rest in the deque for the next read.

def pop_events(q):
    blob = b"".join(q)
    parts = blob.split(b"\n\n")
    done, tail = parts[:-1], parts[-1]
    q.clear()
    if tail:
        q.append(tail)
    return done

def release_or_flag(blob):
    try:
        return json.loads(blob.decode()), False
    except json.JSONDecodeError:
        return None, True

That function is the whole lesson in miniature. Completed events should leave the deque at once. An unfinished tail stays until the next chunk.

Why would you join the tail to a finished event and wait? Now the lab question, which is smaller than people make it. You need a repeatable caller and a place to run the harness.

You do not need a new cluster diagram on a whiteboard. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode has free model access and a free server option.

Those are the only product facts I will lean on. Those two facts matter only as a cheap place to repeat the same call. Point the harness at a free model endpoint if you already have one.

The free server is a quiet box, not a proof of scale. I am not naming a model, a quota, a box size, or a deadline. Those details move, and I have no primary note in front of me.

If a landing page disagrees with this paragraph, trust the page. Do not turn that free server into a load cannon. A profile of one reader is not a capacity test.

You will learn your buffer, not their scheduler. Who should skip this note and walk away? Skip it if you are tuning kernels on a GPU you own.

Skip it if you need a contractual latency promise for customers. Skip it if your bug is auth, routing, or a bad prompt. A queue graph will not fix a wrong tool schema.

Skip it if you cannot store the CSV without secrets in the body. Strip prompts and outputs before you plot anything. Log lengths, flags, and held bytes, nothing else.

Would you paste a customer message into a graph title? Compare the two paths on identical chunks, then stop. If incremental parse drops held bytes and time-to-action, keep it.

If it does not, your stall lives somewhere else. The graph just told you that, so believe it. Treat it as a flashlight, not a platform.

Frameworks grow knobs, and knobs hide the next full buffer. Watch the exporter, because it can plug the sink. A trace shipper that blocks on every chunk becomes the new sink plug.

Sample, buffer the samples, and flush on a timer outside the read path. One more trap sits in JSON libraries that read to EOF. That call is a buffered path wearing a streaming costume.

Check the function name before you brag about events. A small experiment plan is what keeps me honest. Label it unexecuted until you run it on your machine.

Build thirty fake chunks, and close only half of them. Run the incremental path and the buffered path back to back. Repeat the pair five times on the same process.

Change only one variable: when you release bytes. Do not change chunk size in the same breath. If you change two knobs, the graph cannot accuse either one.

Write the winner down as a sentence, not a vibe. A fair result sentence names the knob and the direction. It should not name a vendor, a medal, or a percent.

If you cannot write that sentence, you do not have a result. This is still a local fixture, not a production proof. A quiet laptop fan is not a service level.

The release_or_flag helper above is the failure row. Inject one chunk that is not valid JSON. The parser should record an error flag and release the bad event.

If it keeps the bad bytes forever, you built a second leak. That leak will look like model slowness tomorrow morning. Ask yourself who pays when the queue never falls.

Your user pays in waiting, and your process pays in RAM. The free server does not pay that bill for you.

I keep one graph from the run: held bytes over time. I do not keep a mean, a medal, or a thread of best-of-three. Means clock out when a single stuck reader fills the sink.

If the sawtooth never comes back down, fix the release first. Then look at the model, the wire, and the prompt. The order matters, because a full buffer makes every upstream look guilty.

You can rerun this tomorrow without a new vendor story. Keep the same chunks, the same CSV columns, the same question. Did the queue fall when the event closed?

That is the note I would tape above the monitor. It is not a slogan, and it is not a score. Just the graph that still fits on one axis.

I Kept the Run Index. Warmup Took the Win.

Dakota Lin — Sat, 10 Oct 2026 17:55:14 +0000

The second run is not a real speedup. It is a warm socket with a long memory. I kept the run-index graph, and warmup took the credit.

You changed one small line in the client. The next call came back and felt fast. Did the model improve, or did the pipe stay open?

I stopped trusting the warm feeling in my gut. I timed every loop pass in strict run order. I labeled cold and warm before I touched the patch.

This note is a method, not a trophy case. The script below is a proposal you can run. I will not invent a chart from a wish.

The graph I kept

What should that kept graph even show you? Not one proud bar you saved after lunch. It should show latency against the run index.

Run zero is the stranger at the door. It pays for DNS, TLS, and a cold import. Run one may still be kindling, not fire.

I drop those two from the comparison on purpose. I decide that rule before I see the bars. If I pick the rule after, I am shopping.

A free server is useful here for one reason. Your laptop fan is a liar with a motor. It speeds up, then it thermal-throttles, then it lies again.

I would point the same harness at MonkeyCode for repeats. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access is what lets that loop repeat.

A free server option keeps the laptop fan out. I am not naming a model, a quota, or a box. Those facts move, and I will not freeze a rumor.

Read the live page before you trust a limit. A free pool can queue you behind strangers. That wait is not your patch, and it is not a product score.

The loop you can run

Here is the harness I actually want to run. It is stdlib only, so you can audit every line. Point it at a URL you are allowed to call.

#!/usr/bin/env python3
"""Label cold versus warm, and keep the run index."""

import argparse
import csv
import json
import os
import time
import urllib.request

def one_call(url: str, payload: bytes, timeout: float) -> dict:
    wall_start = time.perf_counter_ns()
    cpu_start = time.process_time_ns()
    req = urllib.request.Request(
        url,
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        body = resp.read()
        status = resp.status
    wall_ns = time.perf_counter_ns() - wall_start
    cpu_ns = time.process_time_ns() - cpu_start
    return {
        "status": status,
        "wall_ms": wall_ns / 1e6,
        "cpu_ms": cpu_ns / 1e6,
        "bytes_out": len(payload),
        "bytes_in": len(body),
    }

def phase_for(index: int) -> str:
    if index == 0:
        return "cold"
    if index == 1:
        return "kindling"
    return "warm"

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--url", required=True)
    parser.add_argument("--runs", type=int, default=12)
    parser.add_argument("--out", default="runs.csv")
    parser.add_argument("--timeout", type=float, default=60.0)
    args = parser.parse_args()
    payload = json.dumps({
        "prompt": "Reply with one short sentence about queues.",
        "max_tokens": 32,
    }).encode()
    rows = []
    for i in range(args.runs):
        row = one_call(args.url, payload, args.timeout)
        row["run_index"] = i
        row["phase"] = phase_for(i)
        row["label"] = os.environ.get("RUN_LABEL", "baseline")
        rows.append(row)
    fields = [
        "label", "run_index", "phase", "status",
        "wall_ms", "cpu_ms", "bytes_out", "bytes_in",
    ]
    with open(args.out, "w", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)
    warm = [r["wall_ms"] for r in rows if r["phase"] == "warm"]
    cold = rows[0]["wall_ms"]
    print(f"cold_wall_ms={cold:.1f}")
    if warm:
        mid = sorted(warm)[len(warm) // 2]
        print(f"warm_median_ms={mid:.1f}")
        print(f"cold_over_warm={cold / mid:.2f}")
    print("kept_graph=wall_ms by run_index, split on label")

if __name__ == "__main__":
    main()

The JSON body is a stand-in, not a vendor schema. Swap fields to match the server you actually call. Treat every printed number as unexecuted until you run it.

Set a label, then run the baseline before the patch. Change one thing, then run the patched label. Same URL, same payload, and the same run count.

export RUN_LABEL=baseline
python3 harness.py --url "$MODEL_URL" --runs 12 --out baseline.csv
export RUN_LABEL=patched
python3 harness.py --url "$MODEL_URL" --runs 12 --out patched.csv
python3 sketch.py baseline.csv patched.csv

The sketch script is the graph I keep. Each row is a run, not a vibe. Cold rows stay on the page so I cannot forget them.

#!/usr/bin/env python3
"""Sketch wall time by run index from your own CSV."""

import csv
import sys

def load(path: str) -> list:
    with open(path, newline="") as fh:
        return list(csv.DictReader(fh))

def main() -> None:
    rows = []
    for path in sys.argv[1:]:
        rows.extend(load(path))
    if not rows:
        raise SystemExit("pass at least one csv")
    peak = max(float(r["wall_ms"]) for r in rows) or 1.0
    for r in rows:
        n = int(round(40 * float(r["wall_ms"]) / peak))
        bar = "#" * max(n, 1)
        label = r["label"][:8]
        idx = int(r["run_index"])
        phase = r["phase"][:8]
        print(f"{label:8} {idx:02d} {phase:8} {bar}")

if __name__ == "__main__":
    main()

I omit sample bars so nobody copies a fake chart. Run the sketch yourself, then argue with your own rows.

How do I read that sketch without lying? I cover the warm rows with one hand. I ask whether the cold row moved at all.

If only the warm rows shrink, I blame the socket. Connection reuse will do that trick every time. A prompt cache will smile the same way.

If cold and warm both shrink after the patch, I lean in. I still want a second hour, not a second guess. One quiet minute is not a whole season.

Watch the CPU column while you stare at the wall. If CPU stays tiny while wall grows, you waited. You did not win a secret compute prize.

That split is a guardrail, not the whole story. I already know tails can hide in the network. Today I care who gets credit for the drop.

Do not average the cold row into the warm rows. The mean will hide the stranger and bless the regulars. I want the run index, so order stays visible.

A median of warm rows is fine as a note. It is not fine as a public headline. Headlines eat the label, then they eat the truth.

I also keep bytes out and bytes in. A shorter reply is not a faster client. Did you change the work, or did you change the scale?

If bytes_in falls and wall falls with it, stop. You shrank the job, and then you clapped. That is a diet, not a faster kitchen.

Same payload both times, or the graph is theater. I pin the prompt text in the script for that reason. I do not improve the prompt between the labels.

What about retries, those quiet little cheats of time? A retry can turn one miss into two bills of time. I log status, and I do not drop errors from the file.

A failed row stays in the sketch as a gap marker. Deleting it makes the warm median look brave. Would you delete a crash from a race recap?

Timeouts need a pre-set budget before you start. I pass a timeout on purpose, and I write it down. If I raise it mid-test, I started a new experiment.

The free server can sleep between your bursts. A pause for coffee becomes a new cold start. I note the gap, or I pretend the room never cooled.

Shared capacity is a neighbor, not a lab bench. Someone else's job can sit in your wall time. That is why this loop cannot rank vendors.

Who should walk away

I would not use this method for a public bake-off. I would not use it to promise a speedup in a launch note. I would not use it if I need pinned cores and a quiet NIC.

You should skip it if your boss wants a single number. You should skip it if you cannot store the raw CSV. You should skip it if the payload is still changing shape.

Students can still learn the habit from the shape. Run it once against a local stub if you have no server. The stub will not prove a model, but it will prove your labels.

A local stub is a metronome, not an opponent. It teaches you whether run zero looks different. If run zero looks the same, your timer is blind.

Here is a tiny stub you can hate in private. It sleeps longer on the first hit, then less. That sleep is fake, and the lesson is not.

#!/usr/bin/env python3
"""Local metronome for labels. First hit sleeps longer."""

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import time

HITS = {"n": 0}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        n = HITS["n"]
        HITS["n"] = n + 1
        delay = 0.4 if n == 0 else 0.05
        time.sleep(delay)
        body = b'{"ok":true}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt: str, *args) -> None:
        return

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

The stub listens on port 8765 on localhost only. Start it in one shell, then aim the harness there. Call that stub before you touch a remote pool.

If your sketch cannot see the fake cold hit, stop. Your graph is broken, and the model is innocent.

I like that order more than a hot take. I put tooling before talent, every single time. Labels come before pride, or the bar lies.

When the stub looks honest, I move the URL outward. I keep the same CSV columns, so the eye does not relearn. A new column is a new argument, and I am tired.

How many runs is enough for a note like this? Twelve runs make a sketch, not a paper. If the bars jump like popcorn, I add runs.

I do not add adjectives to a noisy sketch. I also refuse to tune after I see a winner. Tuning the timeout to flatter the patch is a costume.

Take the costume off, or do not publish the photo. Store the script hash next to the CSV if you can. A one-line drift will impersonate a real victory.

I want the blame to land on a commit, not on a mood.

git rev-parse --short HEAD | tee run_commit.txt
python3 -c "import hashlib, pathlib; p=pathlib.Path('harness.py'); print(hashlib.sha256(p.read_bytes()).hexdigest())" | tee run_files.txt

That pair is my alibi when the bar looks pretty. Without it, future me will argue with past me. Past me cheats when the bar looks good.

Limitations sit in the room even when the sketch looks clean. Free model access can change shape without a speech. Free server time can queue, cap, or vanish for an hour.

I am not claiming those options are permanent. I am not claiming a region, a chip, or a rate. If the operator page disagrees with this note, believe the page.

Clock resolution can flatter a tiny client change. A sub-millisecond win on a network hop is noise. Do not frame noise and hang it in the hall.

JSON parsing can dwarf a small logic tweak. If you only time the function you edited, you will miss the dump. Time the whole call, then decide where to cut.

I still would not start a rewrite from one sketch. I would start with a sharper question instead. Which run got faster, and which run only got familiar?

Keep the ugly cold bar in the picture. Cropping it is how warmup steals the win. That theft is common, and I am done applauding it.

If you need a place to repeat the loop, start at MonkeyCode. Read the live limits, then keep the run-index graph.

I Logged Three Clocks. The Model Sat Still.

Dakota Lin — Thu, 08 Oct 2026 07:42:16 +0000

I blamed the model before I split the clocks. My first blame was lazy and flat wrong. The queue and the wire owned the wait.

You feel a coding agent stall and you swear. You swear at weights you never even measured. Have you timed the three gaps, or just the mood?

I mean three clocks, not one sad total. I log send time, first byte, and done. If those three stay glued, your graph is a rumor.

I wrote a tiny harness and refused the folklore. Same prompt, same bytes, and two endpoints. The only story I trust is a span I can replay.

I point one run at MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free server is a hop, not a trophy.

I do not print a quota, a chip, or a lease. Those facts move fast, and stale numbers lie. You should read the live page before you quote one.

I would keep the other run on a box I already own. That contrast is the whole experiment, nothing fancier. If both hops look alike, the model was never the villain.

Here is the harness I keep beside the notes. Treat the JSON body as a local stand-in, not their API. Adapt every field to docs you verified today.

#!/usr/bin/env python3
"""Three-clock sketch. Unexecuted example. No live timings claimed."""
import json, os, time, urllib.request

ENDPOINT = os.environ["MODEL_ENDPOINT"]
TOKEN = os.environ.get("MODEL_TOKEN", "")
PROMPT = open("fixture.txt", encoding="utf-8").read()

def one_shot(body: bytes) -> dict:
    req = urllib.request.Request(ENDPOINT, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    if TOKEN:
        req.add_header("Authorization", "Bearer " + TOKEN)
    t_send = time.perf_counter()
    with urllib.request.urlopen(req, timeout=12) as resp:
        t_headers = time.perf_counter()
        raw = resp.read()
        t_done = time.perf_counter()
    return {
        "to_headers_ms": round((t_headers - t_send) * 1000, 1),
        "body_ms": round((t_done - t_headers) * 1000, 1),
        "total_ms": round((t_done - t_send) * 1000, 1),
        "bytes": len(raw),
    }

if __name__ == "__main__":
    payload = json.dumps({"prompt": PROMPT}).encode()
    for i in range(5):
        row = one_shot(payload)
        row["i"] = i
        row["cold"] = i == 0
        print(json.dumps(row))
        time.sleep(1.0)

Run it only against an endpoint you may call. Keep the token in the environment, never in git. Five loops are a sketch, not a promise to the wire.

export MODEL_ENDPOINT="https://example.invalid/complete"
export MODEL_TOKEN="set-me-outside-the-repo"
python3 three_clocks.py | tee rows.jsonl

I tag each row warm or cold before I plot it. A cold socket will impersonate a slow model. Have you been graphing a handshake and calling it smart?

Header time on a buffered call is not token time. The server may hold every byte until the end. If you need the first token, you must force a stream.

So the second script reads small chunks on purpose. It stamps the clock when the first chunk lands. The rest of the bar is whatever remains until close.

def stream_gaps(body: bytes) -> dict:
    req = urllib.request.Request(ENDPOINT, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    if TOKEN:
        req.add_header("Authorization", "Bearer " + TOKEN)
    t_send = time.perf_counter()
    t_first = None
    with urllib.request.urlopen(req, timeout=12) as resp:
        while True:
            chunk = resp.read(64)
            if not chunk:
                break
            if t_first is None:
                t_first = time.perf_counter()
    t_done = time.perf_counter()
    if t_first is None:
        t_first = t_done
    return {
        "to_first_ms": round((t_first - t_send) * 1000, 1),
        "after_ms": round((t_done - t_first) * 1000, 1),
    }

If the server never flushes, those two gaps collapse. You just measured a buffered body with extra steps. Check the docs for a real stream flag before you blame the hop.

I print an ASCII bar and I refuse a theme. Two bars per row, first chunk and the rest. The picture is a receipt, not a ranking post.

def bar(ms: float) -> str:
    cells = int(ms // 25)  # 25 ms per mark, then stop the ink
    return "#" * max(0, min(cells, 40))

def render(rows: list) -> str:
    lines = []
    for row in rows:
        lines.append("%s first %s %s" % (row["i"], bar(row["to_first_ms"]), row["to_first_ms"]))
        lines.append("%s rest  %s %s" % (row["i"], bar(row["after_ms"]), row["after_ms"]))
    return "\n".join(lines)

Look at a row before you touch the prompt. If the first bar dwarfs the rest, the model sat still. Why rewrite instructions while the queue eats the span?

Picture a diner with a fast cook and a slow bell. You yell at the kitchen because the plate is late. The ticket was stuck on the rail the whole time.

That is this graph, with worse coffee and more JSON. The cook is the work after the first chunk. The rail is queue, DNS, TLS, and your own retry nap.

I keep the graph that shows the rail, not the cook. A teammate can argue with a bar of hashes. A teammate cannot argue with your mood from Tuesday.

When the rail is long, I pull the call off the key path. Batch the review after lunch, or night the notes. Do not park a remote hop inside a save hook.

A free server is a fair night shift for that batch. It is a bad stand-in for a compiler. Would you block one keystroke on a queue you do not own?

I still want that hop for work that can wait. A diff read after lunch can wait a little. Autocomplete under a finger cannot, and should not pretend.

One request in flight is the whole load I allow. A parallel fan-out would only measure my stampede. Shared free capacity is a commons, not a drum I get to hit.

Secrets do not ride a hop I do not operate. If the fixture holds private source, I strip it first. A friendly free box is still someone else's machine.

I also refuse one evening as a service level. Other people arrive later, and the rail grows. Your pretty bar from tonight can lie by morning.

Shared ids beat two clocks that never met. Put one request id on the client row and the log. Without that id, you compared neighbors, not one trip.

Here is the gate I run before I trust a bar. Same id, same byte count, and a cold flag. If any one is missing, I throw that row out.

def keep(row: dict) -> bool:
    needed = ("req_id", "bytes", "cold", "to_first_ms", "after_ms")
    return all(k in row and row[k] is not None for k in needed)

Who should skip this approach, and stay skipped? Skip it when policy forbids a third-party model hop. Skip it when you need a model menu I will not invent.

Skip it when your budget is one human keystroke. Skip it when you wanted a winner and a medal. I kept a method, not a ranking, and that was the point.

What should you change after one honest graph? Change the path, the batching, or the timeout. Do not change the prompt until the rail gets short.

I set the client timeout to a number I can say aloud. Twelve seconds makes a stuck queue visible and boring. Sixty seconds hides that same queue inside a spinner.

Retries are where a clean graph goes to die. A hidden second try adds a full extra rail. Log the attempt index, or the total will confess to fiction.

A total can include two naps and still look sincere. The model may have finished on the first try. Which clock did you paste into the incident note?

DNS and TLS deserve blame on a fresh process. Reuse the connection when the API actually allows it. Otherwise every row opens with a handshake costume party.

I store a warm series beside a separate cold row. Mixing them is how a free hop gets framed. The costume party is not the model's personality.

Payload size will fake a generation problem too. A novel-length diff is a shipping choice you made. Trim the fixture before you indict the remote side.

I keep the fixture in the repo next to the script. A teammate should rerun the same bytes, not a vibe. If you edit the fixture, you started a different experiment.

The graph stays ugly so I cannot decorate a miss. No badge and no theme, and no weekday duel. Two bars, a request id, and a cold flag are enough.

If the rest bar grows and the first stays flat, I open the prompt. That is the only time the model owes me an answer. Until then it can sit still, and it should.

Free model access does not mean a queue-free lane. A shared server spends time on other callers. Treat that wait as traffic, not as a broken file of weights.

I will not tell you the wait is small or large. I did not run this harness for a published score. Print your own rows, then keep the graph that matches them.

Check the project page on the day you actually run. Names, limits, and hardware are their facts, not mine. If this note and that page disagree, believe the page.

Point that sketch at their free server when the wait fits. Read the live terms, then keep or drop the endpoint. I would rather you discard it than quote a number I never printed.

I Profiled the Agent. Rebuild Ate the Clock.

Dakota Lin — Wed, 23 Sep 2026 16:52:41 +0000

The language model was not my real bottleneck. Prompt rebuild ate the clock on every later round. I spent days blaming inference like a fool.

Have you ever tuned the model and missed the copy? I wanted one graph I could actually keep. A zoo of dashboards would have hidden the crossing.

I plotted rebuild time against a stubbed model nap. This is a lab note, not a customer war story. I did not harvest production traces for this.

I wrote a harness you can run tonight. The plot is an agent loop with tools. Each tool returns a chunky JSON payload on purpose.

Then the loop concatenates history into the next prompt. After that step it calls the model again. In the stub, the model is just sleep.

On the wire, the model is a network hop. The span names stay identical either way though. Why named spans and not a single timer?

Averages already lied to me enough last month. I wanted stackable intervals with very boring names. I clocked serialize, tool, rebuild, and model separately.

They all land in one boring CSV file. The CSV is the graph I kept. Here is the core of that harness.

#!/usr/bin/env python3
"""Agent-loop span profiler. Stub model. Rebuild grows."""
from __future__ import annotations

import csv
import json
import os
import time
from dataclasses import dataclass

@dataclass
class RoundRow:
    n: int
    serialize_ms: float
    tool_ms: float
    rebuild_ms: float
    model_ms: float
    prompt_chars: int


def fake_tool(round_n: int) -> tuple[dict, float]:
    t0 = time.perf_counter()
    time.sleep(0.005)  # 5ms stand-in for a local tool
    payload = {
        "round": round_n,
        "files": [
            {"path": f"src/mod_{i}.py", "preview": "x" * 2000}
            for i in range(50)
        ],
        "log": "ok" * 400,
    }
    return payload, (time.perf_counter() - t0) * 1000.0


def serialize_tool_result(payload: dict) -> tuple[str, float]:
    t0 = time.perf_counter()
    text = json.dumps(payload, separators=(",", ":"))
    return text, (time.perf_counter() - t0) * 1000.0


def rebuild_prompt(history: list[str], tool_text: str) -> tuple[str, float]:
    t0 = time.perf_counter()
    history.append("TOOL:" + tool_text)
    prompt = ""
    # Quadratic on purpose: a join people actually write.
    for i in range(len(history)):
        prompt = "\n".join(history[: i + 1])
    return prompt, (time.perf_counter() - t0) * 1000.0


def stub_model(prompt: str) -> tuple[str, float]:
    t0 = time.perf_counter()
    time.sleep(0.040)  # ruler, not a model benchmark
    _ = len(prompt)
    return "call_tool", (time.perf_counter() - t0) * 1000.0


def run_loop(rounds: int, model_fn) -> list[RoundRow]:
    history = ["SYS:you are a coding agent", "USER:find the leak"]
    rows: list[RoundRow] = []
    for n in range(1, rounds + 1):
        payload, tool_ms = fake_tool(n)
        tool_text, ser_ms = serialize_tool_result(payload)
        prompt, rebuild_ms = rebuild_prompt(history, tool_text)
        _, model_ms = model_fn(prompt)
        rows.append(
            RoundRow(n, ser_ms, tool_ms, rebuild_ms, model_ms, len(prompt))
        )
    return rows


def write_csv(path: str, rows: list[RoundRow]) -> None:
    with open(path, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(
            ["round", "serialize_ms", "tool_ms", "rebuild_ms",
             "model_ms", "prompt_chars"]
        )
        for r in rows:
            w.writerow([
                r.n,
                f"{r.serialize_ms:.3f}",
                f"{r.tool_ms:.3f}",
                f"{r.rebuild_ms:.3f}",
                f"{r.model_ms:.3f}",
                r.prompt_chars,
            ])


def ascii_graph(rows: list[RoundRow]) -> str:
    lines = ["round  rebuild#  model.", "-----  --------  ------"]
    for r in rows:
        rb = max(1, int(round(r.rebuild_ms)))
        md = max(1, int(round(r.model_ms / 2)))
        lines.append(f"{r.n:5d}  {'#' * rb}  {'.' * md}")
    return "\n".join(lines)

I run the stub path with a plain command. Nothing fancy. No extra flags.

python agent_spans.py

Wire main to run_loop(12, stub_model) and write agent_spans.csv. The stub sleeps for forty milliseconds each round. That sleep is a ruler, not a benchmark.

Please do not quote it as model speed. What should you watch on a quiet machine? Rebuild starts tiny, almost like a rounding error.

Then the tool payload lands and history snowballs. By later rounds the copy work gets rude. The snowball is the whole point here.

If rebuild crosses that forty millisecond ruler, keep the graph. That crossing is the only picture I kept. Everything else was noise I threw away.

I also wrap the same loop with cProfile. Spans tell a timeline, and functions tell a culprit. json.dumps often walks into the spotlight first here.

import cProfile
import io
import pstats

def profile_loop() -> None:
    pr = cProfile.Profile()
    pr.enable()
    run_loop(12, stub_model)
    pr.disable()
    buf = io.StringIO()
    stats = pstats.Stats(pr, stream=buf)
    stats.sort_stats("cumulative").print_stats(15)
    print(buf.getvalue())

Did you expect the model function to win? I did. It lost on the stub path.

That embarrassment is why I kept the picture. Export pstats if you want a flame-style view. A wide plateau sits under rebuild_prompt on stub runs.

The stub_model bar stays thin and honest. I still prefer the stacked CSV over a flame. It pastes into notes without a graphics stack.

The insult survives the copy-paste just fine. Sample stub output will look locally noisy. Treat the shape as the lesson, not the digits.

round  rebuild#  model.
-----  --------  ------
    1  ##        ....................
    4  #####     ....................
    8  ##########  ....................
   12  ##################  ....................

Your bars will move. That is expected. Pin the payload size if you need a fair fight. Inflate the tool payload until the bars actually fight.

Now comes the remote path with the same four spans. The model function POSTs the prompt over HTTP. I do not wrap a vendor SDK in this note.

I pointed this harness at MonkeyCode because free model access and a free server option let the loop leave my laptop without a GPU box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not publishing their latency, model names, or quotas. I am not calling this a product benchmark either. Point the client at any URL you actually control.

import urllib.request

def remote_model(prompt: str) -> tuple[str, float]:
    url = os.environ.get("AGENT_MODEL_URL", "").strip()
    if not url:
        raise SystemExit("set AGENT_MODEL_URL to an endpoint you control")
    body = json.dumps({"prompt": prompt}).encode("utf-8")
    req = urllib.request.Request(url, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=30) as resp:
        raw = resp.read()
    return raw.decode("utf-8", errors="replace")[:200], (time.perf_counter() - t0) * 1000.0

Set AGENT_MODEL_URL and run the same CSV path. Compare stub bars against wire bars on your graph. Keep the picture and throw away vendor adjectives.

export AGENT_MODEL_URL="http://127.0.0.1:8080/prompt"
python -c "from agent_spans import run_loop, remote_model, write_csv, ascii_graph
rows = run_loop(8, remote_model)
write_csv('agent_spans_wire.csv', rows)
print(ascii_graph(rows))"

Watch the first remote round with extra suspicion. TLS and DNS can steal that opening lap. Queue delay on a shared box can steal more.

After warmup, rebuild may still be the growth term. Why? You resend the whole history every round. Streaming would change this story quite a bit.

This harness does not stream a single token. If your real agent streams, instrument the decoder. Do not borrow my stub graph for that case.

The quadratic join is a microscope, not advice. Real history should be appended, not rebuilt from prefixes. I left the slow join in so the graph has teeth.

Swap it for a single "\n".join(history) after you see the crossing. Then run the CSV again on the same machine. Did the rebuild bar collapse like a cheap tent?

If it did, you just caught the copy. If it did not, stare at json.dumps next. Huge tool payloads serialize like wet cement.

This method will not find GPU kernel stalls. It will not find tokenizer weirdness on its own. cProfile also perturbs the thing you measure.

Laptop clocks are noisy and fans make them worse. Run it thrice before you believe a crossing. Pin CPU frequency if you are being precious.

A free shared server adds queue time you cannot see. Without server-side traces you must not blame the model. You can only blame the round trip you measured.

Skip this if you already have tracing in the agent. Skip this if you need legal vendor comparisons. Skip this if your bottleneck is inside CUDA.

I would also skip it for tiny prompts. A twenty token chat will not show rebuild. Inflate the tool payload until the bars fight.

That inflation is a lab trick, not production. Real tools should return less, not more. The harness is a microscope, not a product.

Clone the idea, not my sleep constant. Change the payload. Change the join. Keep one graph.

When rebuild crosses your ruler, you have a bug. Fix the copy. Then go bother inference again.

I Kept the Waterfall. The Client Paid.

Dakota Lin — Tue, 22 Sep 2026 15:11:17 +0000

The remote model rarely owned my wall clock. Local client work stole those seconds from me instead. I kept one waterfall graph after seeing that.

Blame is cheap when a spinner hangs on you. Everyone points at weights and batch size first. I did the same, then I finally measured.

Have you ever watched a slow model that barely ran? I have, on one quiet evening at the desk. The GPU was not the villain that night.

A coding loop is a relay race, not a cannon shot. Prompt build hands off to JSON encoding next. JSON then hands off to the wire itself.

The client is a kitchen line during dinner. The model is only the oven at the back. You can still wait all night on chopping.

The wire hands off to parse, then a tool. Each baton pass looks innocent in isolation today. The stack of those passes is the real tax.

I needed a picture a skeptic would actually trust. Means lie when the tail gets fat fast. One waterfall tells the true order of pain.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I pointed this harness at MonkeyCode's free model access and free server. I am not selling a quota story here.

The product was just the remote side of this. The lesson still lives in the client. Remove the name and the method still holds.

What I actually recorded

I did not start with perf on a kernel. I wrapped the Python client in named spans. Each span got a name, a start, and an end.

Wall clock beats CPU time for this bug class. You can burn no cores and still wait. Waiting is still user pain in this loop.

So I treated the assistant like a tiny distributed system. It was one process with many sequential hops. There was no magic hiding in the stack.

The harness below is a lab notebook, not a production claim. Run it on your own machine later. Keep the graph that you actually measured yourself.

# span_waterfall.py
from __future__ import annotations

import json
import os
import time
import urllib.request
from dataclasses import dataclass, field
from typing import List


@dataclass
class Span:
    name: str
    start: float
    end: float = 0.0
    detail: str = ""

    @property
    def ms(self) -> float:
        return (self.end - self.start) * 1000.0


@dataclass
class Trace:
    spans: List[Span] = field(default_factory=list)

    def start(self, name: str, detail: str = "") -> Span:
        span = Span(name=name, start=time.perf_counter(), detail=detail)
        self.spans.append(span)
        return span

    def stop(self, span: Span) -> None:
        span.end = time.perf_counter()

    def t0(self) -> float:
        return self.spans[0].start if self.spans else 0.0

That is the whole religion for this note. Named intervals sit on a monotonic clock only. Nothing else gets to argue with the bars.

I then added a painter for the review. ASCII is enough when the fight is order. SVG can wait for a prettier dashboard later.

def render_ascii(trace: Trace, width: int = 56) -> str:
    if not trace.spans:
        return "(empty trace)"
    t0 = trace.t0()
    t1 = max(s.end for s in trace.spans)
    total = max(t1 - t0, 1e-9)
    lines = [f"total {total * 1000:.1f} ms  |  {len(trace.spans)} spans"]
    lines.append("span".ljust(16) + " waterfall".ljust(width + 2) + " ms")
    for s in trace.spans:
        a = int(((s.start - t0) / total) * width)
        b = max(int((s.ms / 1000.0 / total) * width), 1)
        bar = (" " * a + "#" * b)[:width].ljust(width)
        lines.append(f"{s.name[:16].ljust(16)}|{bar}| {s.ms:7.1f}")
    return "\n".join(lines)

Look at that bar across the page. The gap before the hash is queue time. The hash is work, or wait dressed as work.

Which of those hashes is the model, really? Only the HTTP span talks to remote weights. The rest is your process talking to itself.

The experiment I actually ran

I faked a two-tool coding turn on purpose. First turn asks for a file from disk. Second turn tries to answer with context.

That shape matches how agents chew a repo. It also multiplies the quiet client tax. A single model call will hide that tax.

Plug your base URL in through the environment. Do not hardcode a fantasy path tonight. I did not invent a vendor API here.

def post_json(url: str, payload: dict, timeout: float = 60.0) -> dict:
    raw = json.dumps(payload).encode("utf-8")
    req = urllib.request.Request(
        url,
        data=raw,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read().decode("utf-8"))


def run_turn(trace: Trace, url: str, prompt: str, tools: int) -> None:
    s = trace.start("build_prompt", detail=str(len(prompt)))
    body = {"prompt": prompt, "max_tokens": 64}
    trace.stop(s)

    s = trace.start("json_dumps")
    _ = json.dumps(body)
    trace.stop(s)

    s = trace.start("http_post")
    try:
        _resp = post_json(url, body)
    except Exception as exc:
        _resp = {"error": type(exc).__name__}
    trace.stop(s)

    s = trace.start("json_parse")
    _ = json.dumps(_resp)
    trace.stop(s)

    for i in range(tools):
        s = trace.start(f"tool_{i}")
        time.sleep(0.05)  # labeled stand-in for a small local tool
        trace.stop(s)

The sleep is labeled as a stand-in tool. Swap it for a real tool if you want blood. Do not ship the sleep into production code.

Then I drive two turns and print the graph. That printed graph is the artifact I keep. I throw away every chart that cannot show order.

def main() -> None:
    url = os.environ.get("REMOTE_URL", "http://127.0.0.1:9/does-not-exist")
    trace = Trace()
    run_turn(trace, url, "find the handler", tools=1)
    run_turn(trace, url, "now patch it", tools=1)
    s = trace.start("render")
    picture = render_ascii(trace)
    trace.stop(s)
    print(picture)


if __name__ == "__main__":
    main()

Run it like a grown-up with captured stdout. Do not screenshot a spinner and call it data. A spinner has no owner and no duration.

python3 span_waterfall.py
REMOTE_URL="$REMOTE_URL" python3 span_waterfall.py
python3 span_waterfall.py > /tmp/waterfall.txt
wc -l /tmp/waterfall.txt

Example output below is illustrative from a dry run. Your numbers will move on your network. Steal the layout, not these fake milliseconds.

total 312.4 ms  |  11 spans
span            waterfall                                                ms
build_prompt    |##                                                      |     4.1
json_dumps      |  #                                                     |     0.8
http_post       |   ################                                     |   140.2
json_parse      |                   #                                    |     1.1
tool_0          |                    ###                                 |    50.4
build_prompt    |                       ##                               |     3.9
json_dumps      |                         #                              |     0.7
http_post       |                          ################              |   138.6
json_parse      |                                          #             |     1.0
tool_0          |                                           ###          |    50.3
render          |                                              #         |     0.6

See the two HTTP blocks in that print? That is the model budget, maybe, if the post worked. See the tools sitting between those HTTP blocks?

I even dumped JSON twice by accident. The second dump hid inside post_json itself. The waterfall cannot split a function you did not wrap.

I almost optimized the model first last week. Would you have done the same thing too? The graph killed that idea on contact.

The bottleneck that survived the graph

JSON was not the villain on short prompts. It will be later, with a fat repo dump. Watch the dump span when context swells hard.

The serial tool was the quiet tax. Fifty milliseconds feels like nothing at review time. Two tools and two turns eat a lunch.

Network failure is also a span, by design. A refused connection still draws a bar. That bar is honesty, not shame, for once.

When the http_post bar dominates, I tune the remote path. When the tool bars dominate, I stop blaming tokens. When build_prompt grows, I treat prompt assembly as the leak.

I kept that ASCII waterfall in the ticket. I threw out the pie chart the same hour. Pies hide order, and order is the plot.

A slow second call after a slow tool is a story. A mean of eighty milliseconds is only a shrug. I needed the story, not the shrug.

What I changed after the picture

I overlapped nothing at the start of this. I only named the crime on the page. Naming is cheaper than threads and queues.

Then I cached the prompt header in memory. Then I refused to dump the same file twice. Then I batched the cheap local tools together.

Did the model get faster after those cuts? No, and I honestly did not care then. The user-facing pause shrank because the client stopped napping between posts.

That is the whole punchline of the note. Remote free inference can be fast enough here. Your relay race may still be the problem.

A remote URL was a convenient far end. I did not need a local GPU theater. The spans would look the same against localhost.

If you try this, keep the graph that argues with you. Delete the graph that only flatters the client.

Who should not take this path

Do not use this harness as a model benchmark. It measures your client, on purpose, every time. Vendor leaderboards will laugh at these client bars.

Do not ship the time.sleep tool to production. It is a labeled stand-in for a local tool. Replace it or go home with empty hands.

If you cannot send prompts off-box, skip remote. Run those same spans against localhost instead. The waterfall still teaches without leaving the machine.

If you need hard SLAs, this note is not a contract. It is a debug habit for messy loops. Habits do not sign your uptime pages.

If you are tuning kernels, you are in another movie. Bring perf, not these ASCII bars, with you. We are not the same audience tonight.

The graph I still keep

I keep the waterfall with two HTTP hashes. I keep the gaps that sit between them. I keep the rule that follows that picture.

Client spans come first, model spans come second. When someone says the model got slower, I ask. Where is the span that actually grew today?

Show me the bar or drop the debate. That sounds harsh in a calm standup. It saved me from another fake optimization week.

Short sentences keep me honest on timers too. A long paragraph hides a missing timer easily. A missing timer hides a missing owner too.

So here is the practice I still run. Wrap the loop and print the bars. Keep the ugly one that argues back.

The pretty mean can leave the ticket. The waterfall has to stay in the review. I will not debate without that picture.

I Chased Tokens Per Second. First Token Won.

Dakota Lin — Mon, 21 Sep 2026 13:21:46 +0000

Tokens per second looks like a clean scientific score. It often hides the stall your users actually feel. I stopped averaging the whole stream this week.

Why did the demo feel snappy while the chart looked sad? Why did the next prompt reverse that story? The wait sat between accept and first token.

I wanted a graph I could actually keep. I did not want another standup wallpaper chart. I wanted a plot that survived a skeptical teammate.

So I treated streaming like a packet capture. I timed the wait and every later chunk. The method stays boring on purpose, and that helps.

A long completion can post a fat throughput number. The user still stared at a blank bubble. Did your pretty score measure work, or waiting?

I ran this as a local lab notebook. The numbers below are local script fixtures only. Please treat them as examples, not production truth.

The bottleneck I actually wanted

I used to log total elapsed time. Then I divided elapsed time by chunk count. That ratio feels like honest throughput.

That ratio is still a blended cocktail, though. Queue delay, warmup, and sampler loops hide inside it. One slow first token poisons the sip.

Later chunks can sprint and still look heroic. The average never files a complaint. Want a better question for the next incident?

Ask when the first byte actually moved. Then ask whether later bytes arrived steadily. Those two clocks split stall from engine.

I kept a heatmap of inter-chunk gaps. The average line was a polite little liar. A single long hole felt like a freeze.

Users do not experience means on a chat bubble. They experience the hole in the drip. That hole is the whole argument here.

A tiny probe you can rerun

This probe is labeled as a local experiment. It records start, first token, and finish. It also records gaps between streamed chunks.

I used a plain HTTP streaming client. No vendor SDK sat in that path. Point it at any OpenAI-style chat endpoint.

# stream_probe.py
# Labeled experiment: local clocks, not a vendor benchmark.

import json
import os
import sys
import time
import urllib.request


def percentile(values, p):
    if not values:
        return None
    ordered = sorted(values)
    idx = min(len(ordered) - 1, int(round((p / 100.0) * (len(ordered) - 1))))
    return ordered[idx]


def iter_sse_lines(resp):
    buf = b""
    while True:
        piece = resp.read(256)
        if not piece:
            if buf:
                yield buf
            break
        buf += piece
        while b"\n" in buf:
            line, buf = buf.split(b"\n", 1)
            yield line


def probe(url, payload, timeout=120):
    data = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    token = os.environ.get("API_TOKEN")
    if token:
        headers["Authorization"] = "Bearer " + token
    req = urllib.request.Request(url, data=data, headers=headers, method="POST")
    t0 = time.perf_counter()
    first = None
    chunks = 0
    gaps = []
    last = t0
    text = []

    with urllib.request.urlopen(req, timeout=timeout) as resp:
        for raw in iter_sse_lines(resp):
            now = time.perf_counter()
            line = raw.decode("utf-8", errors="replace").strip()
            if not line.startswith("data:"):
                continue
            body = line[5:].strip()
            if body == "[DONE]":
                break
            try:
                obj = json.loads(body)
            except json.JSONDecodeError:
                continue
            delta = (
                obj.get("choices", [{}])[0]
                .get("delta", {})
                .get("content")
            )
            if not delta:
                continue
            if first is None:
                first = now
            gaps.append(now - last)
            last = now
            chunks += 1
            text.append(delta)

    end = time.perf_counter()
    ttft = None if first is None else (first - t0)
    gen = 0.0 if first is None else (end - first)
    tps = 0.0 if gen <= 0 else (chunks / gen)
    later = gaps[1:]
    return {
        "ttft_s": ttft,
        "total_s": end - t0,
        "chunks": chunks,
        "tps_after_first": tps,
        "gap_p50_s": percentile(later, 50),
        "gap_p99_s": percentile(later, 99),
        "chars": sum(len(x) for x in text),
        "gaps_s": later,
    }


if __name__ == "__main__":
    url = sys.argv[1]
    prompt = sys.argv[2] if len(sys.argv) > 2 else "Explain tail latency in one paragraph."
    payload = {
        "model": os.environ.get("MODEL", "local"),
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "max_tokens": 256,
    }
    result = probe(url, payload)
    printable = {k: v for k, v in result.items() if k != "gaps_s"}
    print(json.dumps(printable, indent=2))

Save it as stream_probe.py and run it from a shell. You should use the same prompt every time. Change one variable per run, or you will learn noise.

python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions
python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions "Count from 1 to 40."

Warm the process with a throwaway prompt first. Then capture five runs of the real prompt. Discarding the cold start keeps the graph honest.

Sample fixture output from that local run follows. Do not read it as a product claim. Do not paste it into a vendor table.

{
  "ttft_s": 1.82,
  "total_s": 4.11,
  "chunks": 94,
  "tps_after_first": 41.0,
  "gap_p50_s": 0.018,
  "gap_p99_s": 0.240,
  "chars": 612
}

See the split in those three clocks? Throughput after first token looks perfectly fine. The user still paid almost two silent seconds.

Would you ship that graph to a product manager? I would not ship that blended score anyway. I would ship those two split clocks instead.

The experiment I kept

I repeated the same prompt five times. I discarded the first run as a warmup. I plotted TTFT right beside gap P99.

Throughput wandered around and taught me almost nothing. TTFT explained the thinking pause in the bubble. Gap P99 explained that mid-sentence freeze well.

I also logged chunk sizes on the side. Some servers drip one token at a time. Some servers dump a whole phrase at once.

If your client paints on every chunk, burst still stutters. The wire itself can still be perfectly fine. Then the renderer gets bored and then flooded.

That is why I refuse a single score now. A single score is just a blender. You cannot unmix waiting from working later on.

I wrote the five runs into a CSV. Then I sketched TTFT against gap P99 next. That pair is the graph I still have open.

# keep_graph.py
# Proposal: turn probe JSON lines into a tiny CSV.

import json
import sys

print("run,ttft_s,gap_p99_s,tps_after_first,chunks")
for i, line in enumerate(sys.stdin, 1):
    line = line.strip()
    if not line:
        continue
    row = json.loads(line)
    print(
        f"{i},{row['ttft_s']:.3f},{row['gap_p99_s']:.3f},"
        f"{row['tps_after_first']:.1f},{row['chunks']}"
    )

Pipe the probe into it after each run. Please do not average those ugly rows away. The ugly run is the one users will feel.

python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions >> runs.jsonl
python3 keep_graph.py < runs.jsonl

Look at run three before you trust run two. If TTFT jumps while later gaps stay flat, the engine was waiting, not writing. If later gaps jump while TTFT stays calm, the stall moved into the stream.

That split is the whole profile. Everything else is interior decoration. Keep the pair that argues with the demo.

Where a nearby free endpoint helps

I needed a cheap loop for the probe. Paid APIs punish retries and those broken parsers. You want to break the client on purpose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I pointed the probe at MonkeyCode for that loop. Free model access and a free server option kept the retries local. I am not naming models, quotas, or hardware here.

I am naming a setup I could restart without paperwork. The clocks remain the subject, not the vendor. If you already have a compatible URL, the same script works.

If you need a nearby loop like that, start with that free setup. Then keep the TTFT plot instead of a throughput trophy.

What the heatmap taught me

I dumped gaps into a histogram in the terminal. I skipped Grafana and the twelve-panel shrine. A mean line would have clapped for the wrong room.

def ascii_hist(gaps, width=40):
    if not gaps:
        return
    lo, hi = min(gaps), max(gaps)
    span = max(hi - lo, 1e-9)
    buckets = [0] * 10
    for g in gaps:
        i = min(9, int((g - lo) / span * 10))
        buckets[i] += 1
    peak = max(buckets) or 1
    for i, n in enumerate(buckets):
        bar = "#" * int(n / peak * width)
        edge = lo + (span / 10.0) * i
        print(f"{edge * 1000:7.1f} ms | {bar}")

Call it on the later gaps after each run. Skip the first gap if you already print TTFT. The first gap is the wait you already named.

The mass sat near twenty milliseconds in my fixture. Then a lonely tail jumped past two hundred. That lonely tail is the freeze users see.

The mean never voted the tail into office. I used to worship those total time bars. This week I kept the gap histogram instead.

Is the model slow, or is the stream polite then silent? That question is the whole debugging job here. Those simple throughput averages cannot answer that question.

What I changed after the plot

I stopped printing tokens per second in the demo script. I printed TTFT and gap P99 beside the answer. The room talked about those two clocks immediately.

I also delayed first paint until the first real chunk. A spinning glyph for 1.8 seconds felt honest. A frozen empty bubble just felt broken.

On the server side I looked at write flushing next. A buffered SSE stream merges polite tokens into late bursts. Your client then draws a stall that the model did not cause.

None of that work required a new model. It required a graph that argued with me. The probe is a flashlight, not a leaderboard.

I also stopped comparing cold runs with warm runs. Mixing them rebuilds the blender I just smashed. Cold start belongs on its own axis, beside queue depth.

If you only change the prompt, say so on the plot. If you only change the client flush, say that too. Silent legend changes are how throughput becomes folklore.

Limitations, because the script is rude

This probe counts chunks, not true tokenizer tokens. A chunk may hold a word or a clause. Do not publish it as paper-grade throughput work.

It also ignores TLS, DNS, and proxy buffers. A nearby server still hides those extra costs. A public edge will not hide them at all.

HTTP line buffering can merge events on you. Your P99 gap may be the codec itself. Flush settings and runtime buffering both still matter.

Who should skip this whole trick today? Anyone shipping an SLO from one laptop run should stop. Anyone comparing vendors with this tiny fixture should stop too.

Skip it for token-accurate billing math as well. Also skip it if your product is pure batch. If you only need a file, first-token wait is vanity.

Measure job time and failure rate instead here. Do not use it to crown a free server as faster. Localhost quietly removes the ocean sitting in between.

That ocean is where your real users live. A local heatmap is a rehearsal, not opening night. Bring the same clocks to a client near those users.

The graph I still have open

I did not keep the old throughput chart. I kept TTFT versus gap P99 for one prompt. When TTFT jumps, I look at cold start and queue.

When gap P99 jumps, I look at sampler and write path. I re-run the probe after every quick fix. If the first token does not move, you optimized the wrong room.

If throughput soars and the bubble still blinks, you already know why. Please keep the graph that argues with you. Then throw away the one that claps.

I Kept the CDF. The Mean Clocked Out.

Dakota Lin — Sun, 20 Sep 2026 11:28:18 +0000

The mean never caught the stall I measured. I kept the CDF instead of it.

One average can hide a whole outage today. You already know that ugly fact. Do you still paste the mean into Slack?

I did that. Then a retry wrote the real story.

I was timing a small coding-assistant HTTP client loop. Not a cluster and not a formal lab. Just a script that sent prompts and waited. The sticky note said two hundred milliseconds even. The room still felt much slower than that. Why did those two clocks disagree so hard?

The mean is a tourist with a short itinerary. It visits the easy hours right after lunch. It leaves before the tail even sits down.

So I threw out the single cheerful number. I kept every sample in a simple list. Then I drew a CDF in the terminal.

The trap I walked into first

I logged only total wall time per call. I averaged the blob without shame. I shipped a screenshot with a tiny smile.

That screenshot lied with a perfectly straight face. P50 sat near the mean like a cousin. P99 lived on another planet entirely. Did I even record time to first byte?

No. I recorded a hug from start to end. Handshake, queue, tokens, and JSON parse. One blob. Completely useless for a later argument.

I wanted shape from this round of timing. Not another glossy dashboard panel. Shape you can read from a chair.

The fixture, not a fairy tale

I did not point this harness at production traffic. I built a local liar on purpose. The server sleeps a little on most calls. Sometimes it sleeps a lot instead.

That is the whole joke of the fixture. Your model endpoint does this under load too. Queue. Cold start. Retry. Name the goblin after you see the cliff.

Label this clearly for anyone reading along. These numbers come from localhost only. They teach a shape. They are not a vendor benchmark.

# cdf_fixture.py
# Local bimodal latency fixture. Not a product claim.
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import random
import time

class Handler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        return

    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(n)
        # Most calls are cheap. A few are not.
        if random.random() < 0.08:
            time.sleep(random.uniform(0.35, 0.90))
        else:
            time.sleep(random.uniform(0.04, 0.09))
        body = b'{"ok":true,"n":16}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run it and leave it in a corner quietly.

python cdf_fixture.py

The client that keeps every sample

Averages die inside a fold. A list does not die. I wrote a client that stores every latency sample. Then it prints a tiny CDF.

No pandas. No hosted dashboard in this note. Just ranks you can argue with.

# cdf_client.py
import json
import time
import urllib.request

URL = "http://127.0.0.1:8765/work"
N = 200

def one():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=5) as resp:
        _ = resp.read()
    return (time.perf_counter() - t0) * 1000.0

def cdf(samples):
    xs = sorted(samples)
    n = len(xs)
    marks = [50, 90, 95, 99, 100]
    out = []
    for p in marks:
        i = min(n - 1, max(0, int(round(p / 100.0 * n)) - 1))
        if p == 100:
            i = n - 1
        out.append((p, xs[i]))
    return out, xs

if __name__ == "__main__":
    for _ in range(20):
        one()
    samples = [one() for _ in range(N)]
    mean = sum(samples) / len(samples)
    table, xs = cdf(samples)
    print(f"n={len(samples)} mean={mean:.1f}ms min={xs[0]:.1f} max={xs[-1]:.1f}")
    for p, v in table:
        bar = "#" * max(1, int(v / 10))
        print(f"p{p:>3} {v:7.1f}ms {bar}")

Run that against the fixture on the same machine.

python cdf_client.py

You will see a mean that looks almost polite. Then p99 walks in very late. Same machine. Same code. Different story.

Does that surprise you after all these years? It should not surprise you now. It still does, every single time.

Why eight percent owns the graph

The fixture sends eight percent of calls down the long sleep. That is not a rounding error in review. That is a person staring at a spinner.

Think of a bus schedule that is usually fine. One bus in twelve never arrives on time. Do you quote the average wait to your team?

I printed nearest-rank percentiles because they stay honest enough. Sort the list. Pick a rank. Stop smoothing the pain away.

p = 99
n = len(xs)
i = min(n - 1, max(0, round(p / 100 * n) - 1))

People fight about percentile estimators late at night. Nearest-rank is good for a terminal. If you need Hyndman-Fan later, fine. First keep the samples in order.

The mean is a weighted hug of the easy buses. Eight percent still owns the street. Your users ride the street, not the hug.

The graph I actually kept

I do not keep pretty Grafana permalinks around. I keep the terminal block. Something like this showed up in my scrollback later.

Your numbers will move a bit. The cliff should not move.

n=200 mean=112.4ms min=41.2 max=887.0
p 50    68.1ms ######
p 90    84.4ms ########
p 95   402.7ms ########################################
p 99   741.0ms ##########################################################################
p100   887.0ms ########################################################################################

Those bars are from one local run of this fixture. Treat them as a postcard, not a contract. Look at p50 sitting near the mean. Look at p95 falling off a cliff. Would you still ship the mean after that?

I kept that block in the note. I deleted the average from the same note. The cliff is the product you actually shipped. Users do not experience means at all. They experience the bus that never came.

Clocks, warmup, and other small lies

I used time.perf_counter on purpose here. time.time can jump without asking you. NTP is not your friend during a microbench. Have you ever watched a negative duration appear?

Warm the client before you trust the bins. The first calls pay import cost. They pay DNS cache fills. They pay code objects getting hot.

for _ in range(20):
    one()
samples = [one() for _ in range(N)]

Throw those twenty away without mourning them. They are not your distribution today. They are the kettle coming up to temperature.

Garbage collection can steal a bin too. If p99 spikes once, look at GC. A CDF with one poisoned sample still talks. It may tell the wrong story though.

I also printed a crude hash bar per percentile. That is not science class. It is a wall I can see from a chair.

First byte, then the rest of the body

I stopped asking if it was fast. I asked who was waiting on me. I added a second clock for first byte. Not the full body. First byte.

def one_ttfb():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=5) as resp:
        ttfb = (time.perf_counter() - t0) * 1000.0
        _ = resp.read()
        total = (time.perf_counter() - t0) * 1000.0
    return ttfb, total

On this fixture they move together closely. That is the point of the liar. On a real stream they split apart. First byte can look just fine. The rest can crawl home slowly. Then you know which hallway to walk at night.

I also toggled connection reuse on the client. urllib can hide a cold socket. Want a nastier tail in the CDF? Force a new connection every time.

import http.client

def one_cold():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    t0 = time.perf_counter()
    conn = http.client.HTTPConnection("127.0.0.1", 8765, timeout=5)
    conn.request(
        "POST",
        "/work",
        body=payload,
        headers={"Content-Type": "application/json"},
    )
    resp = conn.getresponse()
    _ = resp.read()
    conn.close()
    return (time.perf_counter() - t0) * 1000.0

Cold sockets add jitter even on localhost today. On a remote name they add DNS lookups. They add TLS. They add a handshake you never budgeted for. The mean still shrugs at you. The CDF does not shrug.

Change one knob. Keep the new postcard. That is the whole experiment.

Where a free model and free server fit

I needed a quiet box and a second pair of eyes. Not a chorus. A scratch loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access on the CDF printer. I used the free server option to run the fixture. That is the whole cameo in this note. The harness still works if you delete those sentences.

The model did not invent the bins for me. I still ranked every sample by hand. You should rank them too. Do not let a chat window average your night.

What this will not save you from

This is not load testing in any serious sense. Two hundred sequential calls are only a sketch. They are not a soak or a region test. They still are not your written production SLO.

Do not use a localhost CDF to pick a vendor. Do not paste these bars into a pricing fight. Do not trust a mean from a warm laptop either.

Skip this if traces already give you real histograms. Skip this if you only need token clocks. Skip this when you cannot steer the server sleep. A CDF of pure noise is still noise.

Also skip it if you will not keep raw samples. Binned vanity is just a mean in a costume. People are arguing about tests for models again. Fine. Start with the client you actually ship. If your timer is a single average, the debate is already lost.

The rule I taped to the monitor

If you can only keep one picture, keep the tail. Ask the mean to sit down for a minute. Ask p99 who hurt it last.

Then change one thing only after that. Reuse the socket. Cut the retry storm. Drop a giant tool schema. Run the client again after the change. Keep the new CDF beside the old one.

Did the cliff move after the change? Good. You learned a real thing. Did only the mean move a little? You learned nothing useful tonight.

I still catch myself averaging on busy days. Then I look at that old terminal block again. The mean had clocked out early. The CDF stayed for the whole shift.

I Split the Wait. The Wire Ate P99.

Dakota Lin — Sat, 19 Sep 2026 09:32:00 +0000

The generate call was not the slow part. The wait around it was.

I stopped saying the model felt slow. That sentence is a shrug. It hides DNS, TLS, queueing, and decode together.

You ever stare at a spinner and blame tokens? I did, for too long. Then I split the wait into clocks I could name. P99 jumped out of the handshake. The body looked almost polite.

This is not a take on AI slogans. I do not need another slogan. I need a phase I can point at. If you cannot name the phase, you are guessing.

I work in a tight loop. Prompt, remote generate, apply a patch, run one test. The happy path looks fine in demos. The ugly path lives in p99. That is the graph I kept.

A mean latency is a lullaby. It soothes a standup. It also hides the stall that kills flow. I wanted a CDF, not a vibe. Fifty samples first. Then two hundred. I watched p50 and p99 diverge. That gap is the whole story.

Think of a subway commute. Average wait looks civil on paper. One jammed train ruins the morning. Remote inference has that jammed train. It is often TLS or a cold socket. Sometimes it is a queue you never see.

Local metal will not show this tax. A laptop model skips the wire. That is why a remote path belongs in this lab. You need the network in the picture. Hide the wire and you will blame tokens forever.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option. A free remote path is enough to put the wire on the chart. Point this harness at that option, or at any URL you already trust.

I am not selling a bake-off. I am selling a habit. Stamp the phases. Keep the CDF. Change one variable. Blame tokens last.

I picked a small coding task on purpose. Explain a failing test. Ask for a patch. Apply it. Run pytest on one file. I did not chase quality scores. Quality is another lab. This lab is time. If the patch is nonsense, the test phase still counts. Failed tests are still clocks.

Each run wrote one JSON line. Phase stamps. Status code. Byte count. Test exit. No model names in the log. The log is about clocks, not brands. I ran a dry-run first. Then a live endpoint. The mock taught me my client was sloppy. Connection reuse was off. That bug would have blamed the model. Sound familiar?

I use two tools, not one dashboard. curl splits the handshake. Python keeps the CDF. urllib will not give you honest DNS on every box. I refuse to invent that column. curl -w still can. So I let curl name TLS. I let Python name the tail across many loops.

Save the Python as split_wait.py. Stdlib only. No extra install.

#!/usr/bin/env python3
"""Phase-split timer for a generate call plus a local test.

Dry-run prints a shape. Live mode needs --url.
This is a lab harness, not a vendor benchmark.
"""
from __future__ import annotations

import argparse
import json
import random
import statistics
import sys
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass


@dataclass
class Sample:
    connect_ms: float
    ttfb_ms: float
    body_ms: float
    apply_ms: float
    pytest_ms: float
    bytes_in: int
    status: int
    dry_run: bool


def pct(xs, q):
    if not xs:
        return 0.0
    ys = sorted(xs)
    i = min(len(ys) - 1, max(0, int(round((q / 100) * (len(ys) - 1)))))
    return ys[i]


def dry_sample():
    # Labeled synthetic shape for the article. Not a measurement.
    def jitter(mid, spread):
        return max(0.5, random.gauss(mid, spread))

    ttfb = jitter(130, 35)
    if random.random() < 0.05:
        ttfb += 180  # fake tail, dry-run only
    return Sample(
        connect_ms=jitter(18, 8),
        ttfb_ms=ttfb,
        body_ms=jitter(95, 20),
        apply_ms=jitter(7, 2),
        pytest_ms=jitter(90, 15),
        bytes_in=2048,
        status=200,
        dry_run=True,
    )


def live_sample(url, payload, timeout):
    t0 = time.perf_counter()
    req = urllib.request.Request(url, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            ttfb = time.perf_counter()
            data = resp.read()
            t1 = time.perf_counter()
            status = getattr(resp, "status", 200)
    except urllib.error.URLError as exc:
        raise SystemExit(f"request failed: {exc}") from exc

    t_apply0 = time.perf_counter()
    time.sleep(0.005)  # stand-in for patch apply
    apply_ms = (time.perf_counter() - t_apply0) * 1000

    t_py0 = time.perf_counter()
    time.sleep(0.02)  # replace with subprocess pytest
    pytest_ms = (time.perf_counter() - t_py0) * 1000

    return Sample(
        connect_ms=(ttfb - t0) * 1000,
        ttfb_ms=(ttfb - t0) * 1000,
        body_ms=(t1 - ttfb) * 1000,
        apply_ms=apply_ms,
        pytest_ms=pytest_ms,
        bytes_in=len(data),
        status=status,
        dry_run=False,
    )


def summarize(path):
    rows = []
    with open(path, encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    if not rows:
        print("no rows")
        return
    phases = ["connect_ms", "ttfb_ms", "body_ms", "apply_ms", "pytest_ms"]
    print(f"n={len(rows)}  dry_run={rows[0].get('dry_run')}")
    print(f"{'phase':<12} {'p50_ms':>8} {'p99_ms':>8}")
    for p in phases:
        xs = [r[p] for r in rows]
        print(f"{p:<12} {pct(xs, 50):8.1f} {pct(xs, 99):8.1f}")


def main():
    ap = argparse.ArgumentParser(description="Split wait time. Blame tokens last.")
    ap.add_argument("--url")
    ap.add_argument("--n", type=int, default=50)
    ap.add_argument("--out", default="wait.jsonl")
    ap.add_argument("--dry-run", action="store_true")
    ap.add_argument("--summarize")
    ap.add_argument("--timeout", type=float, default=30.0)
    args = ap.parse_args()
    if args.summarize:
        summarize(args.summarize)
        return
    payload = json.dumps({"prompt": "lab: do not send secrets"}).encode()
    with open(args.out, "w", encoding="utf-8") as out:
        for i in range(args.n):
            if args.dry_run or not args.url:
                sample = dry_sample()
            else:
                sample = live_sample(args.url, payload, args.timeout)
            out.write(json.dumps(asdict(sample)) + "\n")
            print(f"wrote {i + 1}/{args.n}", file=sys.stderr)
    summarize(args.out)


if __name__ == "__main__":
    main()

Run the dry-run until the file format bores you. Then point it at a real URL.

python split_wait.py --dry-run --n 50 --out wait.jsonl
python split_wait.py --summarize wait.jsonl

Live mode is one flag more. Keep secrets out of the payload.

python split_wait.py --url "$ENDPOINT" --n 50 --out wait.jsonl

The summarize command prints a CDF slice. P50 and p99 side by side. I keep that pair. The mean can wait in the hallway.

Handshake detail still belongs to curl. Python told me the tail existed. Curl told me where the handshake sat. I keep this next to the harness.

curl -s -o /dev/null \
  -w "dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n" \
  "$ENDPOINT"

Run that twenty times on a cold client. Then run a reused session. Compare the tls field. You will feel the subway metaphor in your hands.

Here is a dry-run shape from the harness. Treat it as a shape, not a benchmark. I generated it with --dry-run. It is not a product claim. It is not my laptop's truth either.

n=50  dry_run=True
phase          p50_ms   p99_ms
connect_ms       17.4     41.2
ttfb_ms         128.0    312.7
body_ms          94.1    148.6
apply_ms          6.8     12.1
pytest_ms        89.4    129.0

Look at TTFB p99 in that shape. That is the jammed train. Tokens did not do that. Body did not do that. Pytest did not do that. I kept that table anyway, as a reminder of the form. Live numbers have to come from your endpoint. Mine will not travel.

What did I change after I trusted the split? First I enabled keep-alive. One session. Many calls. Connect tails shrank. P50 barely moved. P99 fell hard. That is the lesson in one move. Averages hide reuse. Tails reveal it.

Then I stopped opening a new client per prompt. Sounds obvious in a code review. Watch your agent loop anyway. Many wrappers build a fresh client every turn. They look clean. They tax the handshake. The spinner still smiles.

I added a cheap probe too. A tiny POST with no real prompt. If the probe is already fat, skip the model blame. Fix the path. I also logged bytes. A fat prompt can look like a slow decode. It is often upload. Split TTFB from body or you will misread it.

I do not keep a pretty dashboard for this. I keep a two-line CDF. P50 as a calm line. P99 as a rude one. When they kiss, the path is honest. When they split, I hunt the phase with the bigger gap. For a cold remote path, the gap often sits before the first byte. Decode can win on another day. Then the CDF would say so. I would believe the CDF. I would not believe the spinner.

Here is the decision I actually use after the chart settles. If p99 lives in connect or TLS, reuse the client. If p99 lives in TTFB, suspect queueing or a cold remote. If p99 lives in body, then you may talk about tokens. If p99 lives in pytest, stop tuning prompts. Fix the test. That last one stings. I have tuned prompts while the suite did extra I/O. The chart called me out.

Free remote inference is a spotlight, not a trophy. A free server puts the wire in your lap. You see handshake tax. You see cold starts. You see what a laptop model will never show. Spend the free samples on the tail. Do not spend them on another debate about whether AI coding counts as engineering. The clocks do not care.

This split does not grade models. It grades your wait. Do not use it to rank vendors. Do not use it as an SLA. Do not paste secrets into the payload. A free remote path is still a remote path. Redact prompts. Use a dummy repo. If your company forbids egress, stop. This is not for you.

Do not treat dry-run numbers as evidence. They teach shape. Live numbers are yours. If you need guaranteed latency, this is the wrong lab. If you need eval scores, this is the wrong lab. If you just want to ship a patch tonight, skip the CDF. Write the patch.

The method also assumes you can name phases. If your SDK hides HTTP, you will blame the SDK. That is still useful. Wrap lower. If you cannot log bytes, you will confuse upload with decode. Log bytes.

I will not quote a speedup factor. I already learned that lesson elsewhere. I will not name models I did not pin. I will not invent quotas. I will not pretend a free server is permanent. Availability changes. The split still works on the next URL.

I will not tell you AI coding is engineering. I will not tell you it is not. I will tell you where my seconds went. You can pick the slogan later. Bring a CDF to that argument or stay home.

The spinner was a liar. The CDF was not. Split the wait. Keep p99. Change one variable. Blame tokens last.

I Timed the Packer. The Model Waited.

Dakota Lin — Fri, 18 Sep 2026 07:52:48 +0000

The real bottleneck was prompt packing, not generation. I blamed the remote model for a full week. Does that confession sound a little too familiar?

I kept one graph after all the noise. It plotted packed bytes against later cited bytes. The model waited on my packer almost every run.

This week's feeds keep arguing about vibe coding. I still want a profile before any loud verdict. Can we call it engineering without honest timestamps?

I pointed the client at MonkeyCode's free server and free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The open-source project states a ten-million-token free allowance.

I am not ranking vendors in this short note. I am timing my own packing client instead. Brand names will rot. The file walker will not.

Here is the failure I kept seeing locally. The assistant asked for one tiny numeric patch. My client still shipped half the tree anyway.

Why do we keep doing that to ourselves? We fear a missing import more than latency. Then we rename the wait as model slowness.

Think of a weekend bag versus a steamer trunk. The model asked for a toothbrush. I mailed the garage.

I built a packing-efficiency harness in plain Python. Treat every line as a proposal you can run. Point it at a fixture, never at production secrets.

The harness records five facts on each request. Files scanned. Files packed. Bytes packed. Bytes later cited. Client wait until first byte.

Cited bytes are a messy proxy. I know that already. They still beat a vibes-only dashboard, right?

# pack_profile.py — proposal harness, not a production tracer
from __future__ import annotations

import argparse, json, re, time
from pathlib import Path

IGNORE = {".git", "node_modules", "dist", "__pycache__", ".venv"}
TEXT_OK = {".py", ".md", ".toml", ".txt", ".json"}

def walk(root: Path) -> list[Path]:
    out: list[Path] = []
    for p in root.rglob("*"):
        if not p.is_file():
            continue
        if any(part in IGNORE for part in p.parts):
            continue
        if p.suffix.lower() in TEXT_OK:
            out.append(p)
    return sorted(out)

def pack(paths: list[Path], query: str, allow: set[str] | None) -> str:
    chunks: list[str] = [f"Query:\n{query}\n"]
    for p in paths:
        rel = str(p.as_posix())
        if allow and not any(a in rel for a in allow):
            continue
        try:
            body = p.read_text(encoding="utf-8", errors="replace")
        except OSError:
            continue
        chunks.append(f"\n# FILE {rel}\n{body}")
    return "".join(chunks)

def cited_bytes(prompt: str, reply: str) -> int:
    hits = 0
    for m in re.finditer(r"# FILE (\S+)", prompt):
        name = Path(m.group(1)).name
        if name and name in reply:
            start = m.start()
            nxt = prompt.find("# FILE ", start + 1)
            block = prompt[start: nxt if nxt != -1 else len(prompt)]
            hits += len(block.encode("utf-8"))
    return hits

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--root", required=True)
    ap.add_argument("--query", required=True)
    ap.add_argument("--allow", default="")
    ap.add_argument("--reply-file", default="")  # paste a saved reply for the proxy
    args = ap.parse_args()
    root = Path(args.root)
    t0 = time.perf_counter()
    scanned = walk(root)
    allow = {x for x in args.allow.split(",") if x}
    prompt = pack(scanned, args.query, allow or None)
    packed_paths = [p for p in scanned if not allow or any(a in str(p) for a in allow)]
    pack_ms = (time.perf_counter() - t0) * 1000
    reply = Path(args.reply_file).read_text() if args.reply_file else ""
    row = {
        "scanned": len(scanned),
        "packed": len(packed_paths),
        "packed_bytes": len(prompt.encode("utf-8")),
        "cited_bytes": cited_bytes(prompt, reply) if reply else 0,
        "pack_ms": round(pack_ms, 1),
    }
    print(json.dumps(row, indent=2))
    Path("pack_last.json").write_text(json.dumps(row))

if __name__ == "__main__":
    main()

Run the walker against a throwaway fixture first. Do not aim it at a customer tree. Secrets do not belong in a packing experiment.

python pack_profile.py \
  --root ./fixture_repo \
  --query "fix the off-by-one in total.py" \
  --reply-file ./sample_reply.txt

The fixture stays tiny on purpose. Two modules and one test file. You should watch the packer swallow comments and junk docs.

# fixture_repo/total.py
def total(xs):
    n = 0
    for x in xs:
        n = n + x
    return n  # off-by-one lives in the test, not here

# fixture_repo/total_test.py
from total import total

def test_total():
    assert total([1, 2, 3]) == 6

# fixture_repo/NOTES.md
This file is bait. A hungry packer will ship it.

Illustrative output has this shape only. Do not quote it as a study.

# fat pack, no allow list
{"scanned": 8, "packed": 8, "packed_bytes": 17240, "cited_bytes": 640, "pack_ms": 14.2}

# tight pack, allow=total.py,total_test.py
{"scanned": 8, "packed": 2, "packed_bytes": 1880, "cited_bytes": 580, "pack_ms": 3.1}

See the first row clearly? Eight files packed, almost nothing cited. The second row follows one boring allow list. Same question. Less luggage on the wire.

I kept that pair as the graph. Two dots. One lesson. Stop shipping noise the reply never names.

How did I draw it without a dashboard product? I printed a tiny CSV. Then I plotted it with a short script.

printf '%s\n' 'packed_bytes,cited_bytes,pack_ms' \
  '17240,640,14.2' \
  '1880,580,3.1' > pack.csv
python plot_pack.py

# plot_pack.py — labeled example, not a claimed production chart
import csv
from pathlib import Path

rows = list(csv.DictReader(Path("pack.csv").open()))
print("packed -> cited (pack_ms)")
for r in rows:
    pb = int(float(r["packed_bytes"]))
    cb = int(float(r["cited_bytes"]))
    ms = r["pack_ms"]
    bar = "#" * max(1, pb // 800)
    cit = "." * max(1, cb // 800)
    print(f"{bar}\n{cit}  {pb}->{cb} bytes  {ms} ms")

The plot is not pretty. I still kept it. Pretty charts hide the overpack every time.

Network time still matters on a free remote hop. I did not pretend it vanished overnight. Packing waste moved first on my client, though.

Want a sanity check before you trust one fat wait number? Split DNS, TLS, and first byte. One blob named "generation" will lie to you.

# ttfb_probe.py — proposal only, swap in your real endpoint
import socket, ssl, time

host = "example.invalid"  # replace; do not paste secrets
port = 443

def timed(fn):
    t = time.perf_counter()
    fn()
    return (time.perf_counter() - t) * 1000

dns_ms = timed(lambda: socket.getaddrinfo(host, port, type=socket.SOCK_STREAM))
print({"dns_ms": round(dns_ms, 1)})

I also logged which paths entered the prompt. The list shamed me immediately. README, changelog, a screenshot path. None of them appeared in the reply.

So I added a cheap allow list after the first graph. Tests plus the module named in the query. Nothing else unless the reply asked.

That rule will miss a real import someday. I accept that miss on purpose. I would rather miss once than wait on every run.

Is this still vibe coding if I time it? Maybe it is. I just refuse to vibe the clock itself.

A coding agent on a free server is a flashlight. The packer decides where you point it. A bright light on the wrong wall still wastes the night.

If you work in a secret-heavy repo, skip remote packing. Keep the tree on your side of the door. A free server is not an access-control story.

If you need a signed SLA, this note is not yours. A free server can be busy without warning. I am not promising queue times or hardware.

If your monorepo has no ignore file, stop here. You will profile a disaster you already know. Fix the ignore file before you blame tokens.

Client clocks drift under load. Repeat the run three times at least. Throw away the first warm-up. Caches lie in both directions.

Do not ship this harness into CI as a gate. It measures packing waste on the client. It does not measure correctness of the patch.

The testing debate this week missed this dull layer. People ask if models outgrow the tests. I ask if our prompts outgrow the bug.

A test that never runs because the packer is fat still fails. Slow feedback is a test failure. You just named it generation, didn't you?

I now refuse to tune temperature before I tune includes. Temperature is a spicy leftover knob. Includes are the actual meal on the table.

First-token time is not the same as useful time. A fast hello with a useless tree is still waste. Did your last trace even show the packed byte count?

I keep the packing graph next to the patch. If cited bytes stay tiny, I cut the prompt. If cited bytes jump, I inspect the allow list.

That is the whole method, said plainly. Profile the suitcase. Then talk about the model. Anything else is theater with a spinner.

If you already have a free remote coding endpoint, run the harness on a fixture tonight. Keep the ugly graph. Drop the trunk.

I Forced Fsync. The Model Looked Fine.

Dakota Lin — Thu, 17 Sep 2026 05:54:59 +0000

The remote model was not my bottleneck today. The work after the last token was. I kept that waterfall because prettier charts lied.

Why did first token feel like the whole wait? Product dashboards adore that one shiny latency number. Editors feel a later and much heavier stall.

I ran a small local experiment on purpose. I wanted one graph I could actually defend.

Generation received one span on that graph. Apply received five distinct spans beside it.

The setup stayed boring on purpose here. I used a fixture payload and a timer.

I used a printer for the waterfall. There was no production traffic in this run.

I am not claiming a speedup number. I still needed a remote generator for contrast.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access here. I also used its free server option. Those were the remote side of contrast runs.

They were not a benchmark laboratory for ranking. I will not freeze model names in this note.

I will not publish quotas that drift weekly. Marketing pages change faster than a fixture patch.

The question stayed simpler than any launch post. Where did wall clock actually go then?

Could I see it without buying a tracer? I wrote a Python timer around apply.

It calls perf_counter for each named span. It records those spans in arrival order.

Then it prints a compact ASCII waterfall.

#!/usr/bin/env python3
"""apply_waterfall.py — local spans around patch apply."""
from __future__ import annotations

import argparse
import json
import os
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable


@dataclass
class Span:
    name: str
    start: float
    end: float = 0.0

    @property
    def ms(self) -> float:
        return (self.end - self.start) * 1000.0


@dataclass
class Clock:
    spans: list[Span] = field(default_factory=list)

    def measure(self, name: str, fn: Callable[[], Any]) -> Any:
        span = Span(name, time.perf_counter())
        try:
            return fn()
        finally:
            span.end = time.perf_counter()
            self.spans.append(span)

    def print_waterfall(self) -> None:
        if not self.spans:
            print("no spans")
            return
        t0 = self.spans[0].start
        t1 = max(span.end for span in self.spans)
        width = 48
        total = t1 - t0 or 1e-9
        print(f"{'span':<18} {'ms':>8}  waterfall")
        for span in self.spans:
            left = int(((span.start - t0) / total) * width)
            bar = max(1, int((span.ms / 1000.0 / total) * width))
            print(f"{span.name:<18} {span.ms:8.1f}  {' ' * left}{'#' * bar}")
        print(f"{'total':<18} {total * 1000:8.1f}")


def apply_files(root: Path, files: list[dict], do_fsync: bool) -> None:
    for item in files:
        path = root / item["path"]
        path.parent.mkdir(parents=True, exist_ok=True)
        with path.open("w", encoding="utf-8") as handle:
            handle.write(item["content"])
            handle.flush()
            if do_fsync:
                os.fsync(handle.fileno())


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--fixture", type=Path, required=True)
    parser.add_argument("--root", type=Path, default=Path("./_apply_tmp"))
    parser.add_argument("--fsync", action="store_true")
    args = parser.parse_args()

    clock = Clock()
    args.root.mkdir(parents=True, exist_ok=True)
    raw = clock.measure("load_fixture", lambda: args.fixture.read_text(encoding="utf-8"))
    files = clock.measure("json_decode", lambda: json.loads(raw))
    label = "write_and_fsync" if args.fsync else "write_nofsync"
    clock.measure(label, lambda: apply_files(args.root, files, args.fsync))
    clock.measure("post_write_idle", lambda: time.sleep(0.02))
    clock.print_waterfall()
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

I fed the timer a tiny JSON fixture. Two files landed in a throwaway root.

Nothing in that fixture payload was clever. That boredom was the entire measurement point.

[
  {"path": "hello.py", "content": "def hello():\n    return \"ok\"\n"},
  {"path": "notes.txt", "content": "notes\none more line\n"}
]

Sample output from my machine follows below. Treat these numbers as one local capture. Do not paste them into a vendor slide.

$ python apply_waterfall.py --fixture fixture.json
span                     ms  waterfall
load_fixture            0.4  #
json_decode             0.8  #
write_nofsync           3.9  ##
post_write_idle        20.1       ###########
total                  25.2

I ran it again with fsync forced on. The write span stopped pretending to be cheap.

$ python apply_waterfall.py --fixture fixture.json --fsync
span                     ms  waterfall
load_fixture            0.4  #
json_decode             0.7  #
write_and_fsync        41.6  #####################
post_write_idle        20.0            ##########
total                  62.7

Look at that write span on the second run. Generation would have looked almost polite beside it. Does that match the story in your gut?

Mine did not match that story at all. I had blamed the free server first.

I had blamed the model right after. The disk was laughing under both takes.

Real editors make the same stall worse. A write wakes the recursive file watcher.

The watcher then wakes the language server. The server rereads more files than you touched.

I did not trace the editor this time. That cut needs a different knife entirely.

This note stops at process-local apply work. Editor guts can wait for another night.

Then I wrapped a remote call in the same clock. I did not do that to crown a vendor.

I wanted generation on the same printed axis. First byte and last byte stay separate.

import os
import time
import urllib.request

url = os.environ["CODE_ENDPOINT"]  # your endpoint, not a brand claim
body = b'{"prompt":"fixture only"}'
req = urllib.request.Request(
    url,
    data=body,
    headers={"Content-Type": "application/json"},
    method="POST",
)
start = time.perf_counter()
with urllib.request.urlopen(req, timeout=60) as resp:
    first = time.perf_counter()
    payload = resp.read()
done = time.perf_counter()
print(
    f"ttfb_ms={(first - start) * 1000:.1f} "
    f"body_ms={(done - first) * 1000:.1f} "
    f"bytes={len(payload)}"
)

If your endpoint streams tokens as chunks, listen. Close the stream span on the last chunk.

First byte is a different story altogether. Do not merge those two into one bar.

I almost merged them on the first draft. That merge would have repeated a popular mistake.

First token is a headline for dashboards. Last token is the actual completed job.

The waterfall I kept contains seven named rows. dns_and_connect sits at the far left.

first_byte follows if the handshake was cheap. stream_body holds the model while it talks.

json_decode and path writes live on my laptop. write_and_fsync is the span people skip.

post_write_idle catches formatter and watcher wakeups. Only two of those rows live remotely.

The rest live right beside your chair. That split is the whole practical lesson.

How do you decide what to fix next? I used a crude fence instead of statistics.

If generation is over two thirds, change remotes. If apply is over one third, stop shopping models.

That fence is not a scientific result. It exists to stop a common panic buy.

Do not swap GPUs for a slow open call. Have you bought compute to hide a flush?

I watched one run invert my whole plan. I was ready to switch the remote server.

Apply was already eating about forty percent. The writes were not flushed to disk.

The next run dropped that bar again. So I forced fsync inside the measurement path.

The write bar came back immediately then. Hidden buffering remains a very talented liar.

Have you timed a write that never landed? Python write can return before disk agrees.

Your editor may still wait on the file. Your tests may wait on mtime updates.

Those clocks disagree with each other on purpose. Measurement has to pick a clock and stay.

The command I keep beside the script is plain. I save the ASCII output somewhere boring.

python apply_waterfall.py --fixture fixture.json --fsync > waterfall.txt

Then I argue with it again tomorrow morning. Feelings fade. The file still has bars.

If you add formatting, give it a named span. If you add tests, give them a named span.

Do not hide extra work under a misc label. Misc is where slow code goes to sulk.

I tried a repo formatter once during apply. It walked far beyond the touched files.

The waterfall grew a brand new villain. Generation still looked small beside that walk.

Should you fsync in the assistant happy path? Probably not for an interactive coding loop.

Assistants usually want the write to return. Crash safety is a separate product choice.

I fsync during measurement on purpose only. I do not fsync the daily happy path.

Mixing those modes ruined an earlier graph. The graph I kept is the honest one.

Limitations are not a polite footnote here. This method ignores editor thread internals on purpose.

It ignores GPU scheduling on the remote host. It ignores prompt cache warmth between turns.

It also ignores neighbors on a free server. Shared queues move without warning or apology.

One capture is not a climate report. Do not rank vendors with this teaching script.

Who should skip this whole approach then? Anyone shipping a latency SLA tomorrow morning.

Anyone without a fixture they actually trust. Anyone hoping a chart will pick models.

If you need capacity numbers, run load tests. If you need quality, read the produced diff.

This waterfall answers one rude question only. The rude question is still worth asking weekly.

Is the model slow or is apply theatrical? Most weeks the local theater still wins.

I still like remote generation for draft work. I still use a free server while iterating timers.

Paying for tokens to debug disk is comedy. The graph I kept is not pretty at all.

It is a monospace waterfall in a text file. I open it when someone says models got worse.

Sometimes they are right about the remote side. Sometimes the node_modules tree simply grew overnight.

Sometimes the formatter started checking ignored files. The waterfall does not care about our feelings.

Run the script against your own fixture patch. Change the files until a span surprises you.

Keep the first graph that argues back. That surprise is the artifact worth saving.

Need a free remote target for generation spans? MonkeyCode's free server covered my contrast runs.

I will not turn that into a slogan. The open source repo is there if you want the same contrast.

What would I refuse to add next week? A live dashboard with three green arrows.

A public ranking table of unnamed models. A weekly score that pretends at climate.

Those extras invite a very fake precision. I may add one more span later on.

lsp_idle could come from a watcher log. Not today though, and not in this gist.

Today the disk was already loud enough. That is the whole performance note then.

Keep the waterfall when arguments get loud. Blame apply when the bar earns blame.

Leave the model alone when it already finished.

I Saved the Stacked Bar

Dakota Lin — Wed, 16 Sep 2026 04:07:22 +0000

Local context packing stole the agent clock today. The remote model sat behind a fat snapshot. Comment threads still grade models like racehorses. I kept a stacked bar instead.

Why trust a vibe when wall time has names? I split one coding pass into four stages. Walk the tree. Pack the prompt. Ride the wire. Apply the patch.

The graph I saved was rude. Bytes moved first. Tokens waited second. That order embarrassed my earlier hunches.

This is a lab method, not a trophy benchmark. I did not crown a model. I timed my own laptop.

The debate this week sounds loud on DEV. Some folks say AI already codes better. Others say vibe work is not engineering. Both skip the clock.

I care about the clock. An agent loop is a pipeline. Pipelines hide fat stages under a single spinner.

Here is the analogy I use. You do not blame the chef for a slow dinner. You check the grocery run first. Context packing is that grocery run.

I wanted a graph I could keep. Not a screenshot of a chat. A CSV I can diff after I change ignore rules.

The artifact is a small profiler. It walks a repo. It records bytes and milliseconds. It prints a stacked bar you can paste.

I label every number as sample output. Your tree will disagree. That disagreement is the point.

Start with a dirty fixture. Copy a tiny service. Leave node_modules in place on purpose. Then run the walker twice.

Run it naive. Run it with ignores. Naive walks feel fast in your head. They are not.

Stat calls add up like gravel in a shoe. One extra directory is nothing. A thousand extra files become the story.

I used Python because the stdlib is enough. No new agent framework. No fake distributed trace.

Save this as pack_profile.py. Point it at a real tree. Read the bar before you blame the model.

#!/usr/bin/env python3
"""Lab profiler for agent context packing. Sample timings only."""
from __future__ import annotations

import csv
import os
import time
from dataclasses import dataclass, field
from pathlib import Path

SKIP_DIR_NAMES = {".git", "node_modules", "dist", ".venv", "__pycache__"}
TEXT_SUFFIXES = {".py", ".ts", ".js", ".md", ".json", ".toml", ".yml", ".yaml"}

@dataclass
class Stage:
    name: str
    ms: float = 0.0
    bytes: int = 0

@dataclass
class Report:
    stages: list[Stage] = field(default_factory=list)

    def add(self, name: str, ms: float, nbytes: int = 0) -> None:
        self.stages.append(Stage(name, ms, nbytes))

def should_skip_dir(name: str, use_ignores: bool) -> bool:
    if not use_ignores:
        return name in {".git"}
    return name in SKIP_DIR_NAMES

def walk_and_pack(root: Path, use_ignores: bool, cap_bytes: int) -> tuple[int, int, float]:
    t0 = time.perf_counter()
    total = 0
    files = 0
    chunks: list[bytes] = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if not should_skip_dir(d, use_ignores)]
        for name in filenames:
            path = Path(dirpath) / name
            if path.suffix.lower() not in TEXT_SUFFIXES:
                continue
            try:
                data = path.read_bytes()
            except OSError:
                continue
            files += 1
            if total >= cap_bytes:
                continue
            take = data[: max(0, cap_bytes - total)]
            chunks.append(take)
            total += len(take)
    packed = b"\n".join(chunks)
    ms = (time.perf_counter() - t0) * 1000
    return len(packed), files, ms

def wire_stub(payload_bytes: int) -> float:
    """Local serialization stand-in. Not a model measurement."""
    t0 = time.perf_counter()
    blob = b"x" * min(payload_bytes, 1_000_000)
    _ = hash(blob)
    return (time.perf_counter() - t0) * 1000

def apply_stub(root: Path) -> tuple[int, float]:
    t0 = time.perf_counter()
    patch = b"# lab patch marker\n"
    target = root / ".pack_profile_apply.txt"
    target.write_bytes(patch)
    ms = (time.perf_counter() - t0) * 1000
    return len(patch), ms

def bar(ms: float, total: float, width: int = 24) -> str:
    if total <= 0:
        return " " * width
    n = int(round(width * (ms / total)))
    n = min(width, max(0, n))
    return "#" * n + "." * (width - n)

def write_csv(path: Path, report: Report) -> None:
    with path.open("w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["stage", "ms", "bytes"])
        for s in report.stages:
            w.writerow([s.name, f"{s.ms:.3f}", s.bytes])

def main() -> None:
    root = Path(os.environ.get("PACK_ROOT", ".")).resolve()
    use_ignores = os.environ.get("PACK_IGNORES", "1") != "0"
    cap = int(os.environ.get("PACK_CAP", "750000"))
    packed, files, walk_ms = walk_and_pack(root, use_ignores, cap)
    report = Report()
    report.add("walk_pack", walk_ms, packed)
    report.add("wire_local_stub", wire_stub(packed), packed)
    apply_bytes, apply_ms = apply_stub(root)
    report.add("apply_stub", apply_ms, apply_bytes)
    total = sum(s.ms for s in report.stages) or 1.0
    print(f"root={root}")
    print(f"files_touched={files} packed_bytes={packed} ignores={use_ignores}")
    print("SAMPLE GRAPH ONLY. This is not a model benchmark.")
    for s in report.stages:
        print(f"{s.name:16} {s.ms:8.2f} ms  {bar(s.ms, total)}  {s.bytes} B")
    out = Path("pack_graph.csv")
    write_csv(out, report)
    print(f"wrote {out}")

if __name__ == "__main__":
    main()

Run the naive pass first. Watch the file count jump. Then turn ignores back on.

python3 pack_profile.py
PACK_ROOT=. PACK_IGNORES=0 PACK_CAP=750000 python3 pack_profile.py
PACK_ROOT=. PACK_IGNORES=1 PACK_CAP=750000 python3 pack_profile.py
cp pack_graph.csv pack_graph_ignores.csv

Want CPU names, not just bars? Keep a profile too. cProfile is enough for this lab.

python3 -m cProfile -o pack.prof pack_profile.py
python3 -c "import pstats; p=pstats.Stats('pack.prof'); p.sort_stats('cumtime').print_stats(20)"

If py-spy is on the box, record a picture. I still keep the CSV. Pictures rot. Rows diff.

py-spy record -o pack.svg -- python3 pack_profile.py

Sample output looks like this on my fixture. Treat it as a shape, not a score. Your milliseconds will move.

SAMPLE GRAPH ONLY. This is not a model benchmark.
walk_pack         412.17 ms  ################....  750000 B
wire_local_stub     6.02 ms  ....................  750000 B
apply_stub          1.14 ms  ....................  18 B

See the punchline? The walk ate the bar. The stub wire was a footnote. Apply was a shrug.

That graph is what I kept. Not a model leaderboard. Not a chat screenshot. A stacked bar that survived the debate.

People ask the wrong first question. They ask which model is smarter. I ask which stage owns the seconds.

If packing owns the seconds, a fancier model will still wait. It waits on your snapshot. It waits on your ignores.

Cap bytes on purpose. Unlimited packing is a fantasy. Agents choke on whole monorepos the way pockets choke on bricks.

I cap in the script because production prompts cap too. The cap is a flashlight. It shows which files won the lottery.

Change one ignore rule. Diff the CSV. Did walk_pack shrink? Did packed bytes drop? That is the experiment.

diff -u pack_graph.csv pack_graph_ignores.csv || true

Do not stop at local stubs forever. A stub cannot catch queue delay. It cannot catch TLS setup. It cannot catch a cold server.

When I needed a real wire stage, I used a throwaway endpoint. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which I treated as a lab lane for that slice only.

I did not use it to grade intelligence. I used it to give the wire a heartbeat. Packing still had to win or lose on my disk.

Hook the wire later if you want. Time connect, headers, and first byte yourself. Keep those rows next to walk_pack. Then the bar stays honest.

A fair test plan is boring. That is why it works. Same fixture. Same cap. One changed ignore. One changed endpoint.

Warm the disk once. Discard that run. Record the next three. Median the milliseconds. Do not publish the warm-up.

Refuse to mix variables. Do not change the model and the ignore file together. You will not know which lever moved.

If walk_pack still dominates, stop tuning prompts. Fix the grocery run. Add ignores. Narrow suffixes. Cap harder.

If the live wire dominates, then talk servers. Until then, the model is a suspect with an alibi.

I have watched teams invert this. They swap models weekly. They never weigh the snapshot. The spinner stays. The story changes clothes.

The stacked bar does not care about clothes. It cares about milliseconds. It cares about bytes.

Limitations are not fine print. They are the method. This script does not measure token quality. It does not measure correctness.

The wire stub is local hashing. It is not network truth. The apply stub writes one marker file. It is not git apply.

Binary files are skipped. Generated folders are skipped only when ignores are on. Symlink storms can still lie.

Who should not use this approach? Anyone selling a latency SLA from these rows. Anyone profiling a huge monorepo with ignores off. Anyone who needs a courtroom trace of a vendor model.

Also skip it if you will not keep the CSV. A one-off print is a vibe. A saved bar is evidence.

I still like arguments about engineering taste. Taste does not replace a clock. The clock does not replace taste either.

Keep both. Start with the grocery run. Save the stacked bar. Then decide whether the model even got a turn.

If you want a disposable lane for the live wire later, the free model access and free server option was enough for that lab slice. Pack the tree first anyway.