DEV Community: Dakota Lin

I Profiled the Agent. Rebuild Ate the Clock.

Dakota Lin — Wed, 23 Sep 2026 16:52:41 +0000

The language model was not my real bottleneck. Prompt rebuild ate the clock on every later round. I spent days blaming inference like a fool.

Have you ever tuned the model and missed the copy? I wanted one graph I could actually keep. A zoo of dashboards would have hidden the crossing.

I plotted rebuild time against a stubbed model nap. This is a lab note, not a customer war story. I did not harvest production traces for this.

I wrote a harness you can run tonight. The plot is an agent loop with tools. Each tool returns a chunky JSON payload on purpose.

Then the loop concatenates history into the next prompt. After that step it calls the model again. In the stub, the model is just sleep.

On the wire, the model is a network hop. The span names stay identical either way though. Why named spans and not a single timer?

Averages already lied to me enough last month. I wanted stackable intervals with very boring names. I clocked serialize, tool, rebuild, and model separately.

They all land in one boring CSV file. The CSV is the graph I kept. Here is the core of that harness.

#!/usr/bin/env python3
"""Agent-loop span profiler. Stub model. Rebuild grows."""
from __future__ import annotations

import csv
import json
import os
import time
from dataclasses import dataclass

@dataclass
class RoundRow:
    n: int
    serialize_ms: float
    tool_ms: float
    rebuild_ms: float
    model_ms: float
    prompt_chars: int


def fake_tool(round_n: int) -> tuple[dict, float]:
    t0 = time.perf_counter()
    time.sleep(0.005)  # 5ms stand-in for a local tool
    payload = {
        "round": round_n,
        "files": [
            {"path": f"src/mod_{i}.py", "preview": "x" * 2000}
            for i in range(50)
        ],
        "log": "ok" * 400,
    }
    return payload, (time.perf_counter() - t0) * 1000.0


def serialize_tool_result(payload: dict) -> tuple[str, float]:
    t0 = time.perf_counter()
    text = json.dumps(payload, separators=(",", ":"))
    return text, (time.perf_counter() - t0) * 1000.0


def rebuild_prompt(history: list[str], tool_text: str) -> tuple[str, float]:
    t0 = time.perf_counter()
    history.append("TOOL:" + tool_text)
    prompt = ""
    # Quadratic on purpose: a join people actually write.
    for i in range(len(history)):
        prompt = "\n".join(history[: i + 1])
    return prompt, (time.perf_counter() - t0) * 1000.0


def stub_model(prompt: str) -> tuple[str, float]:
    t0 = time.perf_counter()
    time.sleep(0.040)  # ruler, not a model benchmark
    _ = len(prompt)
    return "call_tool", (time.perf_counter() - t0) * 1000.0


def run_loop(rounds: int, model_fn) -> list[RoundRow]:
    history = ["SYS:you are a coding agent", "USER:find the leak"]
    rows: list[RoundRow] = []
    for n in range(1, rounds + 1):
        payload, tool_ms = fake_tool(n)
        tool_text, ser_ms = serialize_tool_result(payload)
        prompt, rebuild_ms = rebuild_prompt(history, tool_text)
        _, model_ms = model_fn(prompt)
        rows.append(
            RoundRow(n, ser_ms, tool_ms, rebuild_ms, model_ms, len(prompt))
        )
    return rows


def write_csv(path: str, rows: list[RoundRow]) -> None:
    with open(path, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(
            ["round", "serialize_ms", "tool_ms", "rebuild_ms",
             "model_ms", "prompt_chars"]
        )
        for r in rows:
            w.writerow([
                r.n,
                f"{r.serialize_ms:.3f}",
                f"{r.tool_ms:.3f}",
                f"{r.rebuild_ms:.3f}",
                f"{r.model_ms:.3f}",
                r.prompt_chars,
            ])


def ascii_graph(rows: list[RoundRow]) -> str:
    lines = ["round  rebuild#  model.", "-----  --------  ------"]
    for r in rows:
        rb = max(1, int(round(r.rebuild_ms)))
        md = max(1, int(round(r.model_ms / 2)))
        lines.append(f"{r.n:5d}  {'#' * rb}  {'.' * md}")
    return "\n".join(lines)

I run the stub path with a plain command. Nothing fancy. No extra flags.

python agent_spans.py

Wire main to run_loop(12, stub_model) and write agent_spans.csv. The stub sleeps for forty milliseconds each round. That sleep is a ruler, not a benchmark.

Please do not quote it as model speed. What should you watch on a quiet machine? Rebuild starts tiny, almost like a rounding error.

Then the tool payload lands and history snowballs. By later rounds the copy work gets rude. The snowball is the whole point here.

If rebuild crosses that forty millisecond ruler, keep the graph. That crossing is the only picture I kept. Everything else was noise I threw away.

I also wrap the same loop with cProfile. Spans tell a timeline, and functions tell a culprit. json.dumps often walks into the spotlight first here.

import cProfile
import io
import pstats

def profile_loop() -> None:
    pr = cProfile.Profile()
    pr.enable()
    run_loop(12, stub_model)
    pr.disable()
    buf = io.StringIO()
    stats = pstats.Stats(pr, stream=buf)
    stats.sort_stats("cumulative").print_stats(15)
    print(buf.getvalue())

Did you expect the model function to win? I did. It lost on the stub path.

That embarrassment is why I kept the picture. Export pstats if you want a flame-style view. A wide plateau sits under rebuild_prompt on stub runs.

The stub_model bar stays thin and honest. I still prefer the stacked CSV over a flame. It pastes into notes without a graphics stack.

The insult survives the copy-paste just fine. Sample stub output will look locally noisy. Treat the shape as the lesson, not the digits.

round  rebuild#  model.
-----  --------  ------
    1  ##        ....................
    4  #####     ....................
    8  ##########  ....................
   12  ##################  ....................

Your bars will move. That is expected. Pin the payload size if you need a fair fight. Inflate the tool payload until the bars actually fight.

Now comes the remote path with the same four spans. The model function POSTs the prompt over HTTP. I do not wrap a vendor SDK in this note.

I pointed this harness at MonkeyCode because free model access and a free server option let the loop leave my laptop without a GPU box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not publishing their latency, model names, or quotas. I am not calling this a product benchmark either. Point the client at any URL you actually control.

import urllib.request

def remote_model(prompt: str) -> tuple[str, float]:
    url = os.environ.get("AGENT_MODEL_URL", "").strip()
    if not url:
        raise SystemExit("set AGENT_MODEL_URL to an endpoint you control")
    body = json.dumps({"prompt": prompt}).encode("utf-8")
    req = urllib.request.Request(url, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=30) as resp:
        raw = resp.read()
    return raw.decode("utf-8", errors="replace")[:200], (time.perf_counter() - t0) * 1000.0

Set AGENT_MODEL_URL and run the same CSV path. Compare stub bars against wire bars on your graph. Keep the picture and throw away vendor adjectives.

export AGENT_MODEL_URL="http://127.0.0.1:8080/prompt"
python -c "from agent_spans import run_loop, remote_model, write_csv, ascii_graph
rows = run_loop(8, remote_model)
write_csv('agent_spans_wire.csv', rows)
print(ascii_graph(rows))"

Watch the first remote round with extra suspicion. TLS and DNS can steal that opening lap. Queue delay on a shared box can steal more.

After warmup, rebuild may still be the growth term. Why? You resend the whole history every round. Streaming would change this story quite a bit.

This harness does not stream a single token. If your real agent streams, instrument the decoder. Do not borrow my stub graph for that case.

The quadratic join is a microscope, not advice. Real history should be appended, not rebuilt from prefixes. I left the slow join in so the graph has teeth.

Swap it for a single "\n".join(history) after you see the crossing. Then run the CSV again on the same machine. Did the rebuild bar collapse like a cheap tent?

If it did, you just caught the copy. If it did not, stare at json.dumps next. Huge tool payloads serialize like wet cement.

This method will not find GPU kernel stalls. It will not find tokenizer weirdness on its own. cProfile also perturbs the thing you measure.

Laptop clocks are noisy and fans make them worse. Run it thrice before you believe a crossing. Pin CPU frequency if you are being precious.

A free shared server adds queue time you cannot see. Without server-side traces you must not blame the model. You can only blame the round trip you measured.

Skip this if you already have tracing in the agent. Skip this if you need legal vendor comparisons. Skip this if your bottleneck is inside CUDA.

I would also skip it for tiny prompts. A twenty token chat will not show rebuild. Inflate the tool payload until the bars fight.

That inflation is a lab trick, not production. Real tools should return less, not more. The harness is a microscope, not a product.

Clone the idea, not my sleep constant. Change the payload. Change the join. Keep one graph.

When rebuild crosses your ruler, you have a bug. Fix the copy. Then go bother inference again.

I Kept the Waterfall. The Client Paid.

Dakota Lin — Tue, 22 Sep 2026 15:11:17 +0000

The remote model rarely owned my wall clock. Local client work stole those seconds from me instead. I kept one waterfall graph after seeing that.

Blame is cheap when a spinner hangs on you. Everyone points at weights and batch size first. I did the same, then I finally measured.

Have you ever watched a slow model that barely ran? I have, on one quiet evening at the desk. The GPU was not the villain that night.

A coding loop is a relay race, not a cannon shot. Prompt build hands off to JSON encoding next. JSON then hands off to the wire itself.

The client is a kitchen line during dinner. The model is only the oven at the back. You can still wait all night on chopping.

The wire hands off to parse, then a tool. Each baton pass looks innocent in isolation today. The stack of those passes is the real tax.

I needed a picture a skeptic would actually trust. Means lie when the tail gets fat fast. One waterfall tells the true order of pain.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I pointed this harness at MonkeyCode's free model access and free server. I am not selling a quota story here.

The product was just the remote side of this. The lesson still lives in the client. Remove the name and the method still holds.

What I actually recorded

I did not start with perf on a kernel. I wrapped the Python client in named spans. Each span got a name, a start, and an end.

Wall clock beats CPU time for this bug class. You can burn no cores and still wait. Waiting is still user pain in this loop.

So I treated the assistant like a tiny distributed system. It was one process with many sequential hops. There was no magic hiding in the stack.

The harness below is a lab notebook, not a production claim. Run it on your own machine later. Keep the graph that you actually measured yourself.

# span_waterfall.py
from __future__ import annotations

import json
import os
import time
import urllib.request
from dataclasses import dataclass, field
from typing import List


@dataclass
class Span:
    name: str
    start: float
    end: float = 0.0
    detail: str = ""

    @property
    def ms(self) -> float:
        return (self.end - self.start) * 1000.0


@dataclass
class Trace:
    spans: List[Span] = field(default_factory=list)

    def start(self, name: str, detail: str = "") -> Span:
        span = Span(name=name, start=time.perf_counter(), detail=detail)
        self.spans.append(span)
        return span

    def stop(self, span: Span) -> None:
        span.end = time.perf_counter()

    def t0(self) -> float:
        return self.spans[0].start if self.spans else 0.0

That is the whole religion for this note. Named intervals sit on a monotonic clock only. Nothing else gets to argue with the bars.

I then added a painter for the review. ASCII is enough when the fight is order. SVG can wait for a prettier dashboard later.

def render_ascii(trace: Trace, width: int = 56) -> str:
    if not trace.spans:
        return "(empty trace)"
    t0 = trace.t0()
    t1 = max(s.end for s in trace.spans)
    total = max(t1 - t0, 1e-9)
    lines = [f"total {total * 1000:.1f} ms  |  {len(trace.spans)} spans"]
    lines.append("span".ljust(16) + " waterfall".ljust(width + 2) + " ms")
    for s in trace.spans:
        a = int(((s.start - t0) / total) * width)
        b = max(int((s.ms / 1000.0 / total) * width), 1)
        bar = (" " * a + "#" * b)[:width].ljust(width)
        lines.append(f"{s.name[:16].ljust(16)}|{bar}| {s.ms:7.1f}")
    return "\n".join(lines)

Look at that bar across the page. The gap before the hash is queue time. The hash is work, or wait dressed as work.

Which of those hashes is the model, really? Only the HTTP span talks to remote weights. The rest is your process talking to itself.

The experiment I actually ran

I faked a two-tool coding turn on purpose. First turn asks for a file from disk. Second turn tries to answer with context.

That shape matches how agents chew a repo. It also multiplies the quiet client tax. A single model call will hide that tax.

Plug your base URL in through the environment. Do not hardcode a fantasy path tonight. I did not invent a vendor API here.

def post_json(url: str, payload: dict, timeout: float = 60.0) -> dict:
    raw = json.dumps(payload).encode("utf-8")
    req = urllib.request.Request(
        url,
        data=raw,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read().decode("utf-8"))


def run_turn(trace: Trace, url: str, prompt: str, tools: int) -> None:
    s = trace.start("build_prompt", detail=str(len(prompt)))
    body = {"prompt": prompt, "max_tokens": 64}
    trace.stop(s)

    s = trace.start("json_dumps")
    _ = json.dumps(body)
    trace.stop(s)

    s = trace.start("http_post")
    try:
        _resp = post_json(url, body)
    except Exception as exc:
        _resp = {"error": type(exc).__name__}
    trace.stop(s)

    s = trace.start("json_parse")
    _ = json.dumps(_resp)
    trace.stop(s)

    for i in range(tools):
        s = trace.start(f"tool_{i}")
        time.sleep(0.05)  # labeled stand-in for a small local tool
        trace.stop(s)

The sleep is labeled as a stand-in tool. Swap it for a real tool if you want blood. Do not ship the sleep into production code.

Then I drive two turns and print the graph. That printed graph is the artifact I keep. I throw away every chart that cannot show order.

def main() -> None:
    url = os.environ.get("REMOTE_URL", "http://127.0.0.1:9/does-not-exist")
    trace = Trace()
    run_turn(trace, url, "find the handler", tools=1)
    run_turn(trace, url, "now patch it", tools=1)
    s = trace.start("render")
    picture = render_ascii(trace)
    trace.stop(s)
    print(picture)


if __name__ == "__main__":
    main()

Run it like a grown-up with captured stdout. Do not screenshot a spinner and call it data. A spinner has no owner and no duration.

python3 span_waterfall.py
REMOTE_URL="$REMOTE_URL" python3 span_waterfall.py
python3 span_waterfall.py > /tmp/waterfall.txt
wc -l /tmp/waterfall.txt

Example output below is illustrative from a dry run. Your numbers will move on your network. Steal the layout, not these fake milliseconds.

total 312.4 ms  |  11 spans
span            waterfall                                                ms
build_prompt    |##                                                      |     4.1
json_dumps      |  #                                                     |     0.8
http_post       |   ################                                     |   140.2
json_parse      |                   #                                    |     1.1
tool_0          |                    ###                                 |    50.4
build_prompt    |                       ##                               |     3.9
json_dumps      |                         #                              |     0.7
http_post       |                          ################              |   138.6
json_parse      |                                          #             |     1.0
tool_0          |                                           ###          |    50.3
render          |                                              #         |     0.6

See the two HTTP blocks in that print? That is the model budget, maybe, if the post worked. See the tools sitting between those HTTP blocks?

I even dumped JSON twice by accident. The second dump hid inside post_json itself. The waterfall cannot split a function you did not wrap.

I almost optimized the model first last week. Would you have done the same thing too? The graph killed that idea on contact.

The bottleneck that survived the graph

JSON was not the villain on short prompts. It will be later, with a fat repo dump. Watch the dump span when context swells hard.

The serial tool was the quiet tax. Fifty milliseconds feels like nothing at review time. Two tools and two turns eat a lunch.

Network failure is also a span, by design. A refused connection still draws a bar. That bar is honesty, not shame, for once.

When the http_post bar dominates, I tune the remote path. When the tool bars dominate, I stop blaming tokens. When build_prompt grows, I treat prompt assembly as the leak.

I kept that ASCII waterfall in the ticket. I threw out the pie chart the same hour. Pies hide order, and order is the plot.

A slow second call after a slow tool is a story. A mean of eighty milliseconds is only a shrug. I needed the story, not the shrug.

What I changed after the picture

I overlapped nothing at the start of this. I only named the crime on the page. Naming is cheaper than threads and queues.

Then I cached the prompt header in memory. Then I refused to dump the same file twice. Then I batched the cheap local tools together.

Did the model get faster after those cuts? No, and I honestly did not care then. The user-facing pause shrank because the client stopped napping between posts.

That is the whole punchline of the note. Remote free inference can be fast enough here. Your relay race may still be the problem.

A remote URL was a convenient far end. I did not need a local GPU theater. The spans would look the same against localhost.

If you try this, keep the graph that argues with you. Delete the graph that only flatters the client.

Who should not take this path

Do not use this harness as a model benchmark. It measures your client, on purpose, every time. Vendor leaderboards will laugh at these client bars.

Do not ship the time.sleep tool to production. It is a labeled stand-in for a local tool. Replace it or go home with empty hands.

If you cannot send prompts off-box, skip remote. Run those same spans against localhost instead. The waterfall still teaches without leaving the machine.

If you need hard SLAs, this note is not a contract. It is a debug habit for messy loops. Habits do not sign your uptime pages.

If you are tuning kernels, you are in another movie. Bring perf, not these ASCII bars, with you. We are not the same audience tonight.

The graph I still keep

I keep the waterfall with two HTTP hashes. I keep the gaps that sit between them. I keep the rule that follows that picture.

Client spans come first, model spans come second. When someone says the model got slower, I ask. Where is the span that actually grew today?

Show me the bar or drop the debate. That sounds harsh in a calm standup. It saved me from another fake optimization week.

Short sentences keep me honest on timers too. A long paragraph hides a missing timer easily. A missing timer hides a missing owner too.

So here is the practice I still run. Wrap the loop and print the bars. Keep the ugly one that argues back.

The pretty mean can leave the ticket. The waterfall has to stay in the review. I will not debate without that picture.

I Chased Tokens Per Second. First Token Won.

Dakota Lin — Mon, 21 Sep 2026 13:21:46 +0000

Tokens per second looks like a clean scientific score. It often hides the stall your users actually feel. I stopped averaging the whole stream this week.

Why did the demo feel snappy while the chart looked sad? Why did the next prompt reverse that story? The wait sat between accept and first token.

I wanted a graph I could actually keep. I did not want another standup wallpaper chart. I wanted a plot that survived a skeptical teammate.

So I treated streaming like a packet capture. I timed the wait and every later chunk. The method stays boring on purpose, and that helps.

A long completion can post a fat throughput number. The user still stared at a blank bubble. Did your pretty score measure work, or waiting?

I ran this as a local lab notebook. The numbers below are local script fixtures only. Please treat them as examples, not production truth.

The bottleneck I actually wanted

I used to log total elapsed time. Then I divided elapsed time by chunk count. That ratio feels like honest throughput.

That ratio is still a blended cocktail, though. Queue delay, warmup, and sampler loops hide inside it. One slow first token poisons the sip.

Later chunks can sprint and still look heroic. The average never files a complaint. Want a better question for the next incident?

Ask when the first byte actually moved. Then ask whether later bytes arrived steadily. Those two clocks split stall from engine.

I kept a heatmap of inter-chunk gaps. The average line was a polite little liar. A single long hole felt like a freeze.

Users do not experience means on a chat bubble. They experience the hole in the drip. That hole is the whole argument here.

A tiny probe you can rerun

This probe is labeled as a local experiment. It records start, first token, and finish. It also records gaps between streamed chunks.

I used a plain HTTP streaming client. No vendor SDK sat in that path. Point it at any OpenAI-style chat endpoint.

# stream_probe.py
# Labeled experiment: local clocks, not a vendor benchmark.

import json
import os
import sys
import time
import urllib.request


def percentile(values, p):
    if not values:
        return None
    ordered = sorted(values)
    idx = min(len(ordered) - 1, int(round((p / 100.0) * (len(ordered) - 1))))
    return ordered[idx]


def iter_sse_lines(resp):
    buf = b""
    while True:
        piece = resp.read(256)
        if not piece:
            if buf:
                yield buf
            break
        buf += piece
        while b"\n" in buf:
            line, buf = buf.split(b"\n", 1)
            yield line


def probe(url, payload, timeout=120):
    data = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    token = os.environ.get("API_TOKEN")
    if token:
        headers["Authorization"] = "Bearer " + token
    req = urllib.request.Request(url, data=data, headers=headers, method="POST")
    t0 = time.perf_counter()
    first = None
    chunks = 0
    gaps = []
    last = t0
    text = []

    with urllib.request.urlopen(req, timeout=timeout) as resp:
        for raw in iter_sse_lines(resp):
            now = time.perf_counter()
            line = raw.decode("utf-8", errors="replace").strip()
            if not line.startswith("data:"):
                continue
            body = line[5:].strip()
            if body == "[DONE]":
                break
            try:
                obj = json.loads(body)
            except json.JSONDecodeError:
                continue
            delta = (
                obj.get("choices", [{}])[0]
                .get("delta", {})
                .get("content")
            )
            if not delta:
                continue
            if first is None:
                first = now
            gaps.append(now - last)
            last = now
            chunks += 1
            text.append(delta)

    end = time.perf_counter()
    ttft = None if first is None else (first - t0)
    gen = 0.0 if first is None else (end - first)
    tps = 0.0 if gen <= 0 else (chunks / gen)
    later = gaps[1:]
    return {
        "ttft_s": ttft,
        "total_s": end - t0,
        "chunks": chunks,
        "tps_after_first": tps,
        "gap_p50_s": percentile(later, 50),
        "gap_p99_s": percentile(later, 99),
        "chars": sum(len(x) for x in text),
        "gaps_s": later,
    }


if __name__ == "__main__":
    url = sys.argv[1]
    prompt = sys.argv[2] if len(sys.argv) > 2 else "Explain tail latency in one paragraph."
    payload = {
        "model": os.environ.get("MODEL", "local"),
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "max_tokens": 256,
    }
    result = probe(url, payload)
    printable = {k: v for k, v in result.items() if k != "gaps_s"}
    print(json.dumps(printable, indent=2))

Save it as stream_probe.py and run it from a shell. You should use the same prompt every time. Change one variable per run, or you will learn noise.

python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions
python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions "Count from 1 to 40."

Warm the process with a throwaway prompt first. Then capture five runs of the real prompt. Discarding the cold start keeps the graph honest.

Sample fixture output from that local run follows. Do not read it as a product claim. Do not paste it into a vendor table.

{
  "ttft_s": 1.82,
  "total_s": 4.11,
  "chunks": 94,
  "tps_after_first": 41.0,
  "gap_p50_s": 0.018,
  "gap_p99_s": 0.240,
  "chars": 612
}

See the split in those three clocks? Throughput after first token looks perfectly fine. The user still paid almost two silent seconds.

Would you ship that graph to a product manager? I would not ship that blended score anyway. I would ship those two split clocks instead.

The experiment I kept

I repeated the same prompt five times. I discarded the first run as a warmup. I plotted TTFT right beside gap P99.

Throughput wandered around and taught me almost nothing. TTFT explained the thinking pause in the bubble. Gap P99 explained that mid-sentence freeze well.

I also logged chunk sizes on the side. Some servers drip one token at a time. Some servers dump a whole phrase at once.

If your client paints on every chunk, burst still stutters. The wire itself can still be perfectly fine. Then the renderer gets bored and then flooded.

That is why I refuse a single score now. A single score is just a blender. You cannot unmix waiting from working later on.

I wrote the five runs into a CSV. Then I sketched TTFT against gap P99 next. That pair is the graph I still have open.

# keep_graph.py
# Proposal: turn probe JSON lines into a tiny CSV.

import json
import sys

print("run,ttft_s,gap_p99_s,tps_after_first,chunks")
for i, line in enumerate(sys.stdin, 1):
    line = line.strip()
    if not line:
        continue
    row = json.loads(line)
    print(
        f"{i},{row['ttft_s']:.3f},{row['gap_p99_s']:.3f},"
        f"{row['tps_after_first']:.1f},{row['chunks']}"
    )

Pipe the probe into it after each run. Please do not average those ugly rows away. The ugly run is the one users will feel.

python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions >> runs.jsonl
python3 keep_graph.py < runs.jsonl

Look at run three before you trust run two. If TTFT jumps while later gaps stay flat, the engine was waiting, not writing. If later gaps jump while TTFT stays calm, the stall moved into the stream.

That split is the whole profile. Everything else is interior decoration. Keep the pair that argues with the demo.

Where a nearby free endpoint helps

I needed a cheap loop for the probe. Paid APIs punish retries and those broken parsers. You want to break the client on purpose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I pointed the probe at MonkeyCode for that loop. Free model access and a free server option kept the retries local. I am not naming models, quotas, or hardware here.

I am naming a setup I could restart without paperwork. The clocks remain the subject, not the vendor. If you already have a compatible URL, the same script works.

If you need a nearby loop like that, start with that free setup. Then keep the TTFT plot instead of a throughput trophy.

What the heatmap taught me

I dumped gaps into a histogram in the terminal. I skipped Grafana and the twelve-panel shrine. A mean line would have clapped for the wrong room.

def ascii_hist(gaps, width=40):
    if not gaps:
        return
    lo, hi = min(gaps), max(gaps)
    span = max(hi - lo, 1e-9)
    buckets = [0] * 10
    for g in gaps:
        i = min(9, int((g - lo) / span * 10))
        buckets[i] += 1
    peak = max(buckets) or 1
    for i, n in enumerate(buckets):
        bar = "#" * int(n / peak * width)
        edge = lo + (span / 10.0) * i
        print(f"{edge * 1000:7.1f} ms | {bar}")

Call it on the later gaps after each run. Skip the first gap if you already print TTFT. The first gap is the wait you already named.

The mass sat near twenty milliseconds in my fixture. Then a lonely tail jumped past two hundred. That lonely tail is the freeze users see.

The mean never voted the tail into office. I used to worship those total time bars. This week I kept the gap histogram instead.

Is the model slow, or is the stream polite then silent? That question is the whole debugging job here. Those simple throughput averages cannot answer that question.

What I changed after the plot

I stopped printing tokens per second in the demo script. I printed TTFT and gap P99 beside the answer. The room talked about those two clocks immediately.

I also delayed first paint until the first real chunk. A spinning glyph for 1.8 seconds felt honest. A frozen empty bubble just felt broken.

On the server side I looked at write flushing next. A buffered SSE stream merges polite tokens into late bursts. Your client then draws a stall that the model did not cause.

None of that work required a new model. It required a graph that argued with me. The probe is a flashlight, not a leaderboard.

I also stopped comparing cold runs with warm runs. Mixing them rebuilds the blender I just smashed. Cold start belongs on its own axis, beside queue depth.

If you only change the prompt, say so on the plot. If you only change the client flush, say that too. Silent legend changes are how throughput becomes folklore.

Limitations, because the script is rude

This probe counts chunks, not true tokenizer tokens. A chunk may hold a word or a clause. Do not publish it as paper-grade throughput work.

It also ignores TLS, DNS, and proxy buffers. A nearby server still hides those extra costs. A public edge will not hide them at all.

HTTP line buffering can merge events on you. Your P99 gap may be the codec itself. Flush settings and runtime buffering both still matter.

Who should skip this whole trick today? Anyone shipping an SLO from one laptop run should stop. Anyone comparing vendors with this tiny fixture should stop too.

Skip it for token-accurate billing math as well. Also skip it if your product is pure batch. If you only need a file, first-token wait is vanity.

Measure job time and failure rate instead here. Do not use it to crown a free server as faster. Localhost quietly removes the ocean sitting in between.

That ocean is where your real users live. A local heatmap is a rehearsal, not opening night. Bring the same clocks to a client near those users.

The graph I still have open

I did not keep the old throughput chart. I kept TTFT versus gap P99 for one prompt. When TTFT jumps, I look at cold start and queue.

When gap P99 jumps, I look at sampler and write path. I re-run the probe after every quick fix. If the first token does not move, you optimized the wrong room.

If throughput soars and the bubble still blinks, you already know why. Please keep the graph that argues with you. Then throw away the one that claps.

I Kept the CDF. The Mean Clocked Out.

Dakota Lin — Sun, 20 Sep 2026 11:28:18 +0000

The mean never caught the stall I measured. I kept the CDF instead of it.

One average can hide a whole outage today. You already know that ugly fact. Do you still paste the mean into Slack?

I did that. Then a retry wrote the real story.

I was timing a small coding-assistant HTTP client loop. Not a cluster and not a formal lab. Just a script that sent prompts and waited. The sticky note said two hundred milliseconds even. The room still felt much slower than that. Why did those two clocks disagree so hard?

The mean is a tourist with a short itinerary. It visits the easy hours right after lunch. It leaves before the tail even sits down.

So I threw out the single cheerful number. I kept every sample in a simple list. Then I drew a CDF in the terminal.

The trap I walked into first

I logged only total wall time per call. I averaged the blob without shame. I shipped a screenshot with a tiny smile.

That screenshot lied with a perfectly straight face. P50 sat near the mean like a cousin. P99 lived on another planet entirely. Did I even record time to first byte?

No. I recorded a hug from start to end. Handshake, queue, tokens, and JSON parse. One blob. Completely useless for a later argument.

I wanted shape from this round of timing. Not another glossy dashboard panel. Shape you can read from a chair.

The fixture, not a fairy tale

I did not point this harness at production traffic. I built a local liar on purpose. The server sleeps a little on most calls. Sometimes it sleeps a lot instead.

That is the whole joke of the fixture. Your model endpoint does this under load too. Queue. Cold start. Retry. Name the goblin after you see the cliff.

Label this clearly for anyone reading along. These numbers come from localhost only. They teach a shape. They are not a vendor benchmark.

# cdf_fixture.py
# Local bimodal latency fixture. Not a product claim.
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import random
import time

class Handler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        return

    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(n)
        # Most calls are cheap. A few are not.
        if random.random() < 0.08:
            time.sleep(random.uniform(0.35, 0.90))
        else:
            time.sleep(random.uniform(0.04, 0.09))
        body = b'{"ok":true,"n":16}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run it and leave it in a corner quietly.

python cdf_fixture.py

The client that keeps every sample

Averages die inside a fold. A list does not die. I wrote a client that stores every latency sample. Then it prints a tiny CDF.

No pandas. No hosted dashboard in this note. Just ranks you can argue with.

# cdf_client.py
import json
import time
import urllib.request

URL = "http://127.0.0.1:8765/work"
N = 200

def one():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=5) as resp:
        _ = resp.read()
    return (time.perf_counter() - t0) * 1000.0

def cdf(samples):
    xs = sorted(samples)
    n = len(xs)
    marks = [50, 90, 95, 99, 100]
    out = []
    for p in marks:
        i = min(n - 1, max(0, int(round(p / 100.0 * n)) - 1))
        if p == 100:
            i = n - 1
        out.append((p, xs[i]))
    return out, xs

if __name__ == "__main__":
    for _ in range(20):
        one()
    samples = [one() for _ in range(N)]
    mean = sum(samples) / len(samples)
    table, xs = cdf(samples)
    print(f"n={len(samples)} mean={mean:.1f}ms min={xs[0]:.1f} max={xs[-1]:.1f}")
    for p, v in table:
        bar = "#" * max(1, int(v / 10))
        print(f"p{p:>3} {v:7.1f}ms {bar}")

Run that against the fixture on the same machine.

python cdf_client.py

You will see a mean that looks almost polite. Then p99 walks in very late. Same machine. Same code. Different story.

Does that surprise you after all these years? It should not surprise you now. It still does, every single time.

Why eight percent owns the graph

The fixture sends eight percent of calls down the long sleep. That is not a rounding error in review. That is a person staring at a spinner.

Think of a bus schedule that is usually fine. One bus in twelve never arrives on time. Do you quote the average wait to your team?

I printed nearest-rank percentiles because they stay honest enough. Sort the list. Pick a rank. Stop smoothing the pain away.

p = 99
n = len(xs)
i = min(n - 1, max(0, round(p / 100 * n) - 1))

People fight about percentile estimators late at night. Nearest-rank is good for a terminal. If you need Hyndman-Fan later, fine. First keep the samples in order.

The mean is a weighted hug of the easy buses. Eight percent still owns the street. Your users ride the street, not the hug.

The graph I actually kept

I do not keep pretty Grafana permalinks around. I keep the terminal block. Something like this showed up in my scrollback later.

Your numbers will move a bit. The cliff should not move.

n=200 mean=112.4ms min=41.2 max=887.0
p 50    68.1ms ######
p 90    84.4ms ########
p 95   402.7ms ########################################
p 99   741.0ms ##########################################################################
p100   887.0ms ########################################################################################

Those bars are from one local run of this fixture. Treat them as a postcard, not a contract. Look at p50 sitting near the mean. Look at p95 falling off a cliff. Would you still ship the mean after that?

I kept that block in the note. I deleted the average from the same note. The cliff is the product you actually shipped. Users do not experience means at all. They experience the bus that never came.

Clocks, warmup, and other small lies

I used time.perf_counter on purpose here. time.time can jump without asking you. NTP is not your friend during a microbench. Have you ever watched a negative duration appear?

Warm the client before you trust the bins. The first calls pay import cost. They pay DNS cache fills. They pay code objects getting hot.

for _ in range(20):
    one()
samples = [one() for _ in range(N)]

Throw those twenty away without mourning them. They are not your distribution today. They are the kettle coming up to temperature.

Garbage collection can steal a bin too. If p99 spikes once, look at GC. A CDF with one poisoned sample still talks. It may tell the wrong story though.

I also printed a crude hash bar per percentile. That is not science class. It is a wall I can see from a chair.

First byte, then the rest of the body

I stopped asking if it was fast. I asked who was waiting on me. I added a second clock for first byte. Not the full body. First byte.

def one_ttfb():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=5) as resp:
        ttfb = (time.perf_counter() - t0) * 1000.0
        _ = resp.read()
        total = (time.perf_counter() - t0) * 1000.0
    return ttfb, total

On this fixture they move together closely. That is the point of the liar. On a real stream they split apart. First byte can look just fine. The rest can crawl home slowly. Then you know which hallway to walk at night.

I also toggled connection reuse on the client. urllib can hide a cold socket. Want a nastier tail in the CDF? Force a new connection every time.

import http.client

def one_cold():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    t0 = time.perf_counter()
    conn = http.client.HTTPConnection("127.0.0.1", 8765, timeout=5)
    conn.request(
        "POST",
        "/work",
        body=payload,
        headers={"Content-Type": "application/json"},
    )
    resp = conn.getresponse()
    _ = resp.read()
    conn.close()
    return (time.perf_counter() - t0) * 1000.0

Cold sockets add jitter even on localhost today. On a remote name they add DNS lookups. They add TLS. They add a handshake you never budgeted for. The mean still shrugs at you. The CDF does not shrug.

Change one knob. Keep the new postcard. That is the whole experiment.

Where a free model and free server fit

I needed a quiet box and a second pair of eyes. Not a chorus. A scratch loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access on the CDF printer. I used the free server option to run the fixture. That is the whole cameo in this note. The harness still works if you delete those sentences.

The model did not invent the bins for me. I still ranked every sample by hand. You should rank them too. Do not let a chat window average your night.

What this will not save you from

This is not load testing in any serious sense. Two hundred sequential calls are only a sketch. They are not a soak or a region test. They still are not your written production SLO.

Do not use a localhost CDF to pick a vendor. Do not paste these bars into a pricing fight. Do not trust a mean from a warm laptop either.

Skip this if traces already give you real histograms. Skip this if you only need token clocks. Skip this when you cannot steer the server sleep. A CDF of pure noise is still noise.

Also skip it if you will not keep raw samples. Binned vanity is just a mean in a costume. People are arguing about tests for models again. Fine. Start with the client you actually ship. If your timer is a single average, the debate is already lost.

The rule I taped to the monitor

If you can only keep one picture, keep the tail. Ask the mean to sit down for a minute. Ask p99 who hurt it last.

Then change one thing only after that. Reuse the socket. Cut the retry storm. Drop a giant tool schema. Run the client again after the change. Keep the new CDF beside the old one.

Did the cliff move after the change? Good. You learned a real thing. Did only the mean move a little? You learned nothing useful tonight.

I still catch myself averaging on busy days. Then I look at that old terminal block again. The mean had clocked out early. The CDF stayed for the whole shift.

I Split the Wait. The Wire Ate P99.

Dakota Lin — Sat, 19 Sep 2026 09:32:00 +0000

The generate call was not the slow part. The wait around it was.

I stopped saying the model felt slow. That sentence is a shrug. It hides DNS, TLS, queueing, and decode together.

You ever stare at a spinner and blame tokens? I did, for too long. Then I split the wait into clocks I could name. P99 jumped out of the handshake. The body looked almost polite.

This is not a take on AI slogans. I do not need another slogan. I need a phase I can point at. If you cannot name the phase, you are guessing.

I work in a tight loop. Prompt, remote generate, apply a patch, run one test. The happy path looks fine in demos. The ugly path lives in p99. That is the graph I kept.

A mean latency is a lullaby. It soothes a standup. It also hides the stall that kills flow. I wanted a CDF, not a vibe. Fifty samples first. Then two hundred. I watched p50 and p99 diverge. That gap is the whole story.

Think of a subway commute. Average wait looks civil on paper. One jammed train ruins the morning. Remote inference has that jammed train. It is often TLS or a cold socket. Sometimes it is a queue you never see.

Local metal will not show this tax. A laptop model skips the wire. That is why a remote path belongs in this lab. You need the network in the picture. Hide the wire and you will blame tokens forever.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option. A free remote path is enough to put the wire on the chart. Point this harness at that option, or at any URL you already trust.

I am not selling a bake-off. I am selling a habit. Stamp the phases. Keep the CDF. Change one variable. Blame tokens last.

I picked a small coding task on purpose. Explain a failing test. Ask for a patch. Apply it. Run pytest on one file. I did not chase quality scores. Quality is another lab. This lab is time. If the patch is nonsense, the test phase still counts. Failed tests are still clocks.

Each run wrote one JSON line. Phase stamps. Status code. Byte count. Test exit. No model names in the log. The log is about clocks, not brands. I ran a dry-run first. Then a live endpoint. The mock taught me my client was sloppy. Connection reuse was off. That bug would have blamed the model. Sound familiar?

I use two tools, not one dashboard. curl splits the handshake. Python keeps the CDF. urllib will not give you honest DNS on every box. I refuse to invent that column. curl -w still can. So I let curl name TLS. I let Python name the tail across many loops.

Save the Python as split_wait.py. Stdlib only. No extra install.

#!/usr/bin/env python3
"""Phase-split timer for a generate call plus a local test.

Dry-run prints a shape. Live mode needs --url.
This is a lab harness, not a vendor benchmark.
"""
from __future__ import annotations

import argparse
import json
import random
import statistics
import sys
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass


@dataclass
class Sample:
    connect_ms: float
    ttfb_ms: float
    body_ms: float
    apply_ms: float
    pytest_ms: float
    bytes_in: int
    status: int
    dry_run: bool


def pct(xs, q):
    if not xs:
        return 0.0
    ys = sorted(xs)
    i = min(len(ys) - 1, max(0, int(round((q / 100) * (len(ys) - 1)))))
    return ys[i]


def dry_sample():
    # Labeled synthetic shape for the article. Not a measurement.
    def jitter(mid, spread):
        return max(0.5, random.gauss(mid, spread))

    ttfb = jitter(130, 35)
    if random.random() < 0.05:
        ttfb += 180  # fake tail, dry-run only
    return Sample(
        connect_ms=jitter(18, 8),
        ttfb_ms=ttfb,
        body_ms=jitter(95, 20),
        apply_ms=jitter(7, 2),
        pytest_ms=jitter(90, 15),
        bytes_in=2048,
        status=200,
        dry_run=True,
    )


def live_sample(url, payload, timeout):
    t0 = time.perf_counter()
    req = urllib.request.Request(url, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            ttfb = time.perf_counter()
            data = resp.read()
            t1 = time.perf_counter()
            status = getattr(resp, "status", 200)
    except urllib.error.URLError as exc:
        raise SystemExit(f"request failed: {exc}") from exc

    t_apply0 = time.perf_counter()
    time.sleep(0.005)  # stand-in for patch apply
    apply_ms = (time.perf_counter() - t_apply0) * 1000

    t_py0 = time.perf_counter()
    time.sleep(0.02)  # replace with subprocess pytest
    pytest_ms = (time.perf_counter() - t_py0) * 1000

    return Sample(
        connect_ms=(ttfb - t0) * 1000,
        ttfb_ms=(ttfb - t0) * 1000,
        body_ms=(t1 - ttfb) * 1000,
        apply_ms=apply_ms,
        pytest_ms=pytest_ms,
        bytes_in=len(data),
        status=status,
        dry_run=False,
    )


def summarize(path):
    rows = []
    with open(path, encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    if not rows:
        print("no rows")
        return
    phases = ["connect_ms", "ttfb_ms", "body_ms", "apply_ms", "pytest_ms"]
    print(f"n={len(rows)}  dry_run={rows[0].get('dry_run')}")
    print(f"{'phase':<12} {'p50_ms':>8} {'p99_ms':>8}")
    for p in phases:
        xs = [r[p] for r in rows]
        print(f"{p:<12} {pct(xs, 50):8.1f} {pct(xs, 99):8.1f}")


def main():
    ap = argparse.ArgumentParser(description="Split wait time. Blame tokens last.")
    ap.add_argument("--url")
    ap.add_argument("--n", type=int, default=50)
    ap.add_argument("--out", default="wait.jsonl")
    ap.add_argument("--dry-run", action="store_true")
    ap.add_argument("--summarize")
    ap.add_argument("--timeout", type=float, default=30.0)
    args = ap.parse_args()
    if args.summarize:
        summarize(args.summarize)
        return
    payload = json.dumps({"prompt": "lab: do not send secrets"}).encode()
    with open(args.out, "w", encoding="utf-8") as out:
        for i in range(args.n):
            if args.dry_run or not args.url:
                sample = dry_sample()
            else:
                sample = live_sample(args.url, payload, args.timeout)
            out.write(json.dumps(asdict(sample)) + "\n")
            print(f"wrote {i + 1}/{args.n}", file=sys.stderr)
    summarize(args.out)


if __name__ == "__main__":
    main()

Run the dry-run until the file format bores you. Then point it at a real URL.

python split_wait.py --dry-run --n 50 --out wait.jsonl
python split_wait.py --summarize wait.jsonl

Live mode is one flag more. Keep secrets out of the payload.

python split_wait.py --url "$ENDPOINT" --n 50 --out wait.jsonl

The summarize command prints a CDF slice. P50 and p99 side by side. I keep that pair. The mean can wait in the hallway.

Handshake detail still belongs to curl. Python told me the tail existed. Curl told me where the handshake sat. I keep this next to the harness.

curl -s -o /dev/null \
  -w "dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n" \
  "$ENDPOINT"

Run that twenty times on a cold client. Then run a reused session. Compare the tls field. You will feel the subway metaphor in your hands.

Here is a dry-run shape from the harness. Treat it as a shape, not a benchmark. I generated it with --dry-run. It is not a product claim. It is not my laptop's truth either.

n=50  dry_run=True
phase          p50_ms   p99_ms
connect_ms       17.4     41.2
ttfb_ms         128.0    312.7
body_ms          94.1    148.6
apply_ms          6.8     12.1
pytest_ms        89.4    129.0

Look at TTFB p99 in that shape. That is the jammed train. Tokens did not do that. Body did not do that. Pytest did not do that. I kept that table anyway, as a reminder of the form. Live numbers have to come from your endpoint. Mine will not travel.

What did I change after I trusted the split? First I enabled keep-alive. One session. Many calls. Connect tails shrank. P50 barely moved. P99 fell hard. That is the lesson in one move. Averages hide reuse. Tails reveal it.

Then I stopped opening a new client per prompt. Sounds obvious in a code review. Watch your agent loop anyway. Many wrappers build a fresh client every turn. They look clean. They tax the handshake. The spinner still smiles.

I added a cheap probe too. A tiny POST with no real prompt. If the probe is already fat, skip the model blame. Fix the path. I also logged bytes. A fat prompt can look like a slow decode. It is often upload. Split TTFB from body or you will misread it.

I do not keep a pretty dashboard for this. I keep a two-line CDF. P50 as a calm line. P99 as a rude one. When they kiss, the path is honest. When they split, I hunt the phase with the bigger gap. For a cold remote path, the gap often sits before the first byte. Decode can win on another day. Then the CDF would say so. I would believe the CDF. I would not believe the spinner.

Here is the decision I actually use after the chart settles. If p99 lives in connect or TLS, reuse the client. If p99 lives in TTFB, suspect queueing or a cold remote. If p99 lives in body, then you may talk about tokens. If p99 lives in pytest, stop tuning prompts. Fix the test. That last one stings. I have tuned prompts while the suite did extra I/O. The chart called me out.

Free remote inference is a spotlight, not a trophy. A free server puts the wire in your lap. You see handshake tax. You see cold starts. You see what a laptop model will never show. Spend the free samples on the tail. Do not spend them on another debate about whether AI coding counts as engineering. The clocks do not care.

This split does not grade models. It grades your wait. Do not use it to rank vendors. Do not use it as an SLA. Do not paste secrets into the payload. A free remote path is still a remote path. Redact prompts. Use a dummy repo. If your company forbids egress, stop. This is not for you.

Do not treat dry-run numbers as evidence. They teach shape. Live numbers are yours. If you need guaranteed latency, this is the wrong lab. If you need eval scores, this is the wrong lab. If you just want to ship a patch tonight, skip the CDF. Write the patch.

The method also assumes you can name phases. If your SDK hides HTTP, you will blame the SDK. That is still useful. Wrap lower. If you cannot log bytes, you will confuse upload with decode. Log bytes.

I will not quote a speedup factor. I already learned that lesson elsewhere. I will not name models I did not pin. I will not invent quotas. I will not pretend a free server is permanent. Availability changes. The split still works on the next URL.

I will not tell you AI coding is engineering. I will not tell you it is not. I will tell you where my seconds went. You can pick the slogan later. Bring a CDF to that argument or stay home.

The spinner was a liar. The CDF was not. Split the wait. Keep p99. Change one variable. Blame tokens last.

I Timed the Packer. The Model Waited.

Dakota Lin — Fri, 18 Sep 2026 07:52:48 +0000

The real bottleneck was prompt packing, not generation. I blamed the remote model for a full week. Does that confession sound a little too familiar?

I kept one graph after all the noise. It plotted packed bytes against later cited bytes. The model waited on my packer almost every run.

This week's feeds keep arguing about vibe coding. I still want a profile before any loud verdict. Can we call it engineering without honest timestamps?

I pointed the client at MonkeyCode's free server and free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The open-source project states a ten-million-token free allowance.

I am not ranking vendors in this short note. I am timing my own packing client instead. Brand names will rot. The file walker will not.

Here is the failure I kept seeing locally. The assistant asked for one tiny numeric patch. My client still shipped half the tree anyway.

Why do we keep doing that to ourselves? We fear a missing import more than latency. Then we rename the wait as model slowness.

Think of a weekend bag versus a steamer trunk. The model asked for a toothbrush. I mailed the garage.

I built a packing-efficiency harness in plain Python. Treat every line as a proposal you can run. Point it at a fixture, never at production secrets.

The harness records five facts on each request. Files scanned. Files packed. Bytes packed. Bytes later cited. Client wait until first byte.

Cited bytes are a messy proxy. I know that already. They still beat a vibes-only dashboard, right?

# pack_profile.py — proposal harness, not a production tracer
from __future__ import annotations

import argparse, json, re, time
from pathlib import Path

IGNORE = {".git", "node_modules", "dist", "__pycache__", ".venv"}
TEXT_OK = {".py", ".md", ".toml", ".txt", ".json"}

def walk(root: Path) -> list[Path]:
    out: list[Path] = []
    for p in root.rglob("*"):
        if not p.is_file():
            continue
        if any(part in IGNORE for part in p.parts):
            continue
        if p.suffix.lower() in TEXT_OK:
            out.append(p)
    return sorted(out)

def pack(paths: list[Path], query: str, allow: set[str] | None) -> str:
    chunks: list[str] = [f"Query:\n{query}\n"]
    for p in paths:
        rel = str(p.as_posix())
        if allow and not any(a in rel for a in allow):
            continue
        try:
            body = p.read_text(encoding="utf-8", errors="replace")
        except OSError:
            continue
        chunks.append(f"\n# FILE {rel}\n{body}")
    return "".join(chunks)

def cited_bytes(prompt: str, reply: str) -> int:
    hits = 0
    for m in re.finditer(r"# FILE (\S+)", prompt):
        name = Path(m.group(1)).name
        if name and name in reply:
            start = m.start()
            nxt = prompt.find("# FILE ", start + 1)
            block = prompt[start: nxt if nxt != -1 else len(prompt)]
            hits += len(block.encode("utf-8"))
    return hits

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--root", required=True)
    ap.add_argument("--query", required=True)
    ap.add_argument("--allow", default="")
    ap.add_argument("--reply-file", default="")  # paste a saved reply for the proxy
    args = ap.parse_args()
    root = Path(args.root)
    t0 = time.perf_counter()
    scanned = walk(root)
    allow = {x for x in args.allow.split(",") if x}
    prompt = pack(scanned, args.query, allow or None)
    packed_paths = [p for p in scanned if not allow or any(a in str(p) for a in allow)]
    pack_ms = (time.perf_counter() - t0) * 1000
    reply = Path(args.reply_file).read_text() if args.reply_file else ""
    row = {
        "scanned": len(scanned),
        "packed": len(packed_paths),
        "packed_bytes": len(prompt.encode("utf-8")),
        "cited_bytes": cited_bytes(prompt, reply) if reply else 0,
        "pack_ms": round(pack_ms, 1),
    }
    print(json.dumps(row, indent=2))
    Path("pack_last.json").write_text(json.dumps(row))

if __name__ == "__main__":
    main()

Run the walker against a throwaway fixture first. Do not aim it at a customer tree. Secrets do not belong in a packing experiment.

python pack_profile.py \
  --root ./fixture_repo \
  --query "fix the off-by-one in total.py" \
  --reply-file ./sample_reply.txt

The fixture stays tiny on purpose. Two modules and one test file. You should watch the packer swallow comments and junk docs.

# fixture_repo/total.py
def total(xs):
    n = 0
    for x in xs:
        n = n + x
    return n  # off-by-one lives in the test, not here

# fixture_repo/total_test.py
from total import total

def test_total():
    assert total([1, 2, 3]) == 6

# fixture_repo/NOTES.md
This file is bait. A hungry packer will ship it.

Illustrative output has this shape only. Do not quote it as a study.

# fat pack, no allow list
{"scanned": 8, "packed": 8, "packed_bytes": 17240, "cited_bytes": 640, "pack_ms": 14.2}

# tight pack, allow=total.py,total_test.py
{"scanned": 8, "packed": 2, "packed_bytes": 1880, "cited_bytes": 580, "pack_ms": 3.1}

See the first row clearly? Eight files packed, almost nothing cited. The second row follows one boring allow list. Same question. Less luggage on the wire.

I kept that pair as the graph. Two dots. One lesson. Stop shipping noise the reply never names.

How did I draw it without a dashboard product? I printed a tiny CSV. Then I plotted it with a short script.

printf '%s\n' 'packed_bytes,cited_bytes,pack_ms' \
  '17240,640,14.2' \
  '1880,580,3.1' > pack.csv
python plot_pack.py

# plot_pack.py — labeled example, not a claimed production chart
import csv
from pathlib import Path

rows = list(csv.DictReader(Path("pack.csv").open()))
print("packed -> cited (pack_ms)")
for r in rows:
    pb = int(float(r["packed_bytes"]))
    cb = int(float(r["cited_bytes"]))
    ms = r["pack_ms"]
    bar = "#" * max(1, pb // 800)
    cit = "." * max(1, cb // 800)
    print(f"{bar}\n{cit}  {pb}->{cb} bytes  {ms} ms")

The plot is not pretty. I still kept it. Pretty charts hide the overpack every time.

Network time still matters on a free remote hop. I did not pretend it vanished overnight. Packing waste moved first on my client, though.

Want a sanity check before you trust one fat wait number? Split DNS, TLS, and first byte. One blob named "generation" will lie to you.

# ttfb_probe.py — proposal only, swap in your real endpoint
import socket, ssl, time

host = "example.invalid"  # replace; do not paste secrets
port = 443

def timed(fn):
    t = time.perf_counter()
    fn()
    return (time.perf_counter() - t) * 1000

dns_ms = timed(lambda: socket.getaddrinfo(host, port, type=socket.SOCK_STREAM))
print({"dns_ms": round(dns_ms, 1)})

I also logged which paths entered the prompt. The list shamed me immediately. README, changelog, a screenshot path. None of them appeared in the reply.

So I added a cheap allow list after the first graph. Tests plus the module named in the query. Nothing else unless the reply asked.

That rule will miss a real import someday. I accept that miss on purpose. I would rather miss once than wait on every run.

Is this still vibe coding if I time it? Maybe it is. I just refuse to vibe the clock itself.

A coding agent on a free server is a flashlight. The packer decides where you point it. A bright light on the wrong wall still wastes the night.

If you work in a secret-heavy repo, skip remote packing. Keep the tree on your side of the door. A free server is not an access-control story.

If you need a signed SLA, this note is not yours. A free server can be busy without warning. I am not promising queue times or hardware.

If your monorepo has no ignore file, stop here. You will profile a disaster you already know. Fix the ignore file before you blame tokens.

Client clocks drift under load. Repeat the run three times at least. Throw away the first warm-up. Caches lie in both directions.

Do not ship this harness into CI as a gate. It measures packing waste on the client. It does not measure correctness of the patch.

The testing debate this week missed this dull layer. People ask if models outgrow the tests. I ask if our prompts outgrow the bug.

A test that never runs because the packer is fat still fails. Slow feedback is a test failure. You just named it generation, didn't you?

I now refuse to tune temperature before I tune includes. Temperature is a spicy leftover knob. Includes are the actual meal on the table.

First-token time is not the same as useful time. A fast hello with a useless tree is still waste. Did your last trace even show the packed byte count?

I keep the packing graph next to the patch. If cited bytes stay tiny, I cut the prompt. If cited bytes jump, I inspect the allow list.

That is the whole method, said plainly. Profile the suitcase. Then talk about the model. Anything else is theater with a spinner.

If you already have a free remote coding endpoint, run the harness on a fixture tonight. Keep the ugly graph. Drop the trunk.

I Forced Fsync. The Model Looked Fine.

Dakota Lin — Thu, 17 Sep 2026 05:54:59 +0000

The remote model was not my bottleneck today. The work after the last token was. I kept that waterfall because prettier charts lied.

Why did first token feel like the whole wait? Product dashboards adore that one shiny latency number. Editors feel a later and much heavier stall.

I ran a small local experiment on purpose. I wanted one graph I could actually defend.

Generation received one span on that graph. Apply received five distinct spans beside it.

The setup stayed boring on purpose here. I used a fixture payload and a timer.

I used a printer for the waterfall. There was no production traffic in this run.

I am not claiming a speedup number. I still needed a remote generator for contrast.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access here. I also used its free server option. Those were the remote side of contrast runs.

They were not a benchmark laboratory for ranking. I will not freeze model names in this note.

I will not publish quotas that drift weekly. Marketing pages change faster than a fixture patch.

The question stayed simpler than any launch post. Where did wall clock actually go then?

Could I see it without buying a tracer? I wrote a Python timer around apply.

It calls perf_counter for each named span. It records those spans in arrival order.

Then it prints a compact ASCII waterfall.

#!/usr/bin/env python3
"""apply_waterfall.py — local spans around patch apply."""
from __future__ import annotations

import argparse
import json
import os
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable


@dataclass
class Span:
    name: str
    start: float
    end: float = 0.0

    @property
    def ms(self) -> float:
        return (self.end - self.start) * 1000.0


@dataclass
class Clock:
    spans: list[Span] = field(default_factory=list)

    def measure(self, name: str, fn: Callable[[], Any]) -> Any:
        span = Span(name, time.perf_counter())
        try:
            return fn()
        finally:
            span.end = time.perf_counter()
            self.spans.append(span)

    def print_waterfall(self) -> None:
        if not self.spans:
            print("no spans")
            return
        t0 = self.spans[0].start
        t1 = max(span.end for span in self.spans)
        width = 48
        total = t1 - t0 or 1e-9
        print(f"{'span':<18} {'ms':>8}  waterfall")
        for span in self.spans:
            left = int(((span.start - t0) / total) * width)
            bar = max(1, int((span.ms / 1000.0 / total) * width))
            print(f"{span.name:<18} {span.ms:8.1f}  {' ' * left}{'#' * bar}")
        print(f"{'total':<18} {total * 1000:8.1f}")


def apply_files(root: Path, files: list[dict], do_fsync: bool) -> None:
    for item in files:
        path = root / item["path"]
        path.parent.mkdir(parents=True, exist_ok=True)
        with path.open("w", encoding="utf-8") as handle:
            handle.write(item["content"])
            handle.flush()
            if do_fsync:
                os.fsync(handle.fileno())


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--fixture", type=Path, required=True)
    parser.add_argument("--root", type=Path, default=Path("./_apply_tmp"))
    parser.add_argument("--fsync", action="store_true")
    args = parser.parse_args()

    clock = Clock()
    args.root.mkdir(parents=True, exist_ok=True)
    raw = clock.measure("load_fixture", lambda: args.fixture.read_text(encoding="utf-8"))
    files = clock.measure("json_decode", lambda: json.loads(raw))
    label = "write_and_fsync" if args.fsync else "write_nofsync"
    clock.measure(label, lambda: apply_files(args.root, files, args.fsync))
    clock.measure("post_write_idle", lambda: time.sleep(0.02))
    clock.print_waterfall()
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

I fed the timer a tiny JSON fixture. Two files landed in a throwaway root.

Nothing in that fixture payload was clever. That boredom was the entire measurement point.

[
  {"path": "hello.py", "content": "def hello():\n    return \"ok\"\n"},
  {"path": "notes.txt", "content": "notes\none more line\n"}
]

Sample output from my machine follows below. Treat these numbers as one local capture. Do not paste them into a vendor slide.

$ python apply_waterfall.py --fixture fixture.json
span                     ms  waterfall
load_fixture            0.4  #
json_decode             0.8  #
write_nofsync           3.9  ##
post_write_idle        20.1       ###########
total                  25.2

I ran it again with fsync forced on. The write span stopped pretending to be cheap.

$ python apply_waterfall.py --fixture fixture.json --fsync
span                     ms  waterfall
load_fixture            0.4  #
json_decode             0.7  #
write_and_fsync        41.6  #####################
post_write_idle        20.0            ##########
total                  62.7

Look at that write span on the second run. Generation would have looked almost polite beside it. Does that match the story in your gut?

Mine did not match that story at all. I had blamed the free server first.

I had blamed the model right after. The disk was laughing under both takes.

Real editors make the same stall worse. A write wakes the recursive file watcher.

The watcher then wakes the language server. The server rereads more files than you touched.

I did not trace the editor this time. That cut needs a different knife entirely.

This note stops at process-local apply work. Editor guts can wait for another night.

Then I wrapped a remote call in the same clock. I did not do that to crown a vendor.

I wanted generation on the same printed axis. First byte and last byte stay separate.

import os
import time
import urllib.request

url = os.environ["CODE_ENDPOINT"]  # your endpoint, not a brand claim
body = b'{"prompt":"fixture only"}'
req = urllib.request.Request(
    url,
    data=body,
    headers={"Content-Type": "application/json"},
    method="POST",
)
start = time.perf_counter()
with urllib.request.urlopen(req, timeout=60) as resp:
    first = time.perf_counter()
    payload = resp.read()
done = time.perf_counter()
print(
    f"ttfb_ms={(first - start) * 1000:.1f} "
    f"body_ms={(done - first) * 1000:.1f} "
    f"bytes={len(payload)}"
)

If your endpoint streams tokens as chunks, listen. Close the stream span on the last chunk.

First byte is a different story altogether. Do not merge those two into one bar.

I almost merged them on the first draft. That merge would have repeated a popular mistake.

First token is a headline for dashboards. Last token is the actual completed job.

The waterfall I kept contains seven named rows. dns_and_connect sits at the far left.

first_byte follows if the handshake was cheap. stream_body holds the model while it talks.

json_decode and path writes live on my laptop. write_and_fsync is the span people skip.

post_write_idle catches formatter and watcher wakeups. Only two of those rows live remotely.

The rest live right beside your chair. That split is the whole practical lesson.

How do you decide what to fix next? I used a crude fence instead of statistics.

If generation is over two thirds, change remotes. If apply is over one third, stop shopping models.

That fence is not a scientific result. It exists to stop a common panic buy.

Do not swap GPUs for a slow open call. Have you bought compute to hide a flush?

I watched one run invert my whole plan. I was ready to switch the remote server.

Apply was already eating about forty percent. The writes were not flushed to disk.

The next run dropped that bar again. So I forced fsync inside the measurement path.

The write bar came back immediately then. Hidden buffering remains a very talented liar.

Have you timed a write that never landed? Python write can return before disk agrees.

Your editor may still wait on the file. Your tests may wait on mtime updates.

Those clocks disagree with each other on purpose. Measurement has to pick a clock and stay.

The command I keep beside the script is plain. I save the ASCII output somewhere boring.

python apply_waterfall.py --fixture fixture.json --fsync > waterfall.txt

Then I argue with it again tomorrow morning. Feelings fade. The file still has bars.

If you add formatting, give it a named span. If you add tests, give them a named span.

Do not hide extra work under a misc label. Misc is where slow code goes to sulk.

I tried a repo formatter once during apply. It walked far beyond the touched files.

The waterfall grew a brand new villain. Generation still looked small beside that walk.

Should you fsync in the assistant happy path? Probably not for an interactive coding loop.

Assistants usually want the write to return. Crash safety is a separate product choice.

I fsync during measurement on purpose only. I do not fsync the daily happy path.

Mixing those modes ruined an earlier graph. The graph I kept is the honest one.

Limitations are not a polite footnote here. This method ignores editor thread internals on purpose.

It ignores GPU scheduling on the remote host. It ignores prompt cache warmth between turns.

It also ignores neighbors on a free server. Shared queues move without warning or apology.

One capture is not a climate report. Do not rank vendors with this teaching script.

Who should skip this whole approach then? Anyone shipping a latency SLA tomorrow morning.

Anyone without a fixture they actually trust. Anyone hoping a chart will pick models.

If you need capacity numbers, run load tests. If you need quality, read the produced diff.

This waterfall answers one rude question only. The rude question is still worth asking weekly.

Is the model slow or is apply theatrical? Most weeks the local theater still wins.

I still like remote generation for draft work. I still use a free server while iterating timers.

Paying for tokens to debug disk is comedy. The graph I kept is not pretty at all.

It is a monospace waterfall in a text file. I open it when someone says models got worse.

Sometimes they are right about the remote side. Sometimes the node_modules tree simply grew overnight.

Sometimes the formatter started checking ignored files. The waterfall does not care about our feelings.

Run the script against your own fixture patch. Change the files until a span surprises you.

Keep the first graph that argues back. That surprise is the artifact worth saving.

Need a free remote target for generation spans? MonkeyCode's free server covered my contrast runs.

I will not turn that into a slogan. The open source repo is there if you want the same contrast.

What would I refuse to add next week? A live dashboard with three green arrows.

A public ranking table of unnamed models. A weekly score that pretends at climate.

Those extras invite a very fake precision. I may add one more span later on.

lsp_idle could come from a watcher log. Not today though, and not in this gist.

Today the disk was already loud enough. That is the whole performance note then.

Keep the waterfall when arguments get loud. Blame apply when the bar earns blame.

Leave the model alone when it already finished.

I Saved the Stacked Bar

Dakota Lin — Wed, 16 Sep 2026 04:07:22 +0000

Local context packing stole the agent clock today. The remote model sat behind a fat snapshot. Comment threads still grade models like racehorses. I kept a stacked bar instead.

Why trust a vibe when wall time has names? I split one coding pass into four stages. Walk the tree. Pack the prompt. Ride the wire. Apply the patch.

The graph I saved was rude. Bytes moved first. Tokens waited second. That order embarrassed my earlier hunches.

This is a lab method, not a trophy benchmark. I did not crown a model. I timed my own laptop.

The debate this week sounds loud on DEV. Some folks say AI already codes better. Others say vibe work is not engineering. Both skip the clock.

I care about the clock. An agent loop is a pipeline. Pipelines hide fat stages under a single spinner.

Here is the analogy I use. You do not blame the chef for a slow dinner. You check the grocery run first. Context packing is that grocery run.

I wanted a graph I could keep. Not a screenshot of a chat. A CSV I can diff after I change ignore rules.

The artifact is a small profiler. It walks a repo. It records bytes and milliseconds. It prints a stacked bar you can paste.

I label every number as sample output. Your tree will disagree. That disagreement is the point.

Start with a dirty fixture. Copy a tiny service. Leave node_modules in place on purpose. Then run the walker twice.

Run it naive. Run it with ignores. Naive walks feel fast in your head. They are not.

Stat calls add up like gravel in a shoe. One extra directory is nothing. A thousand extra files become the story.

I used Python because the stdlib is enough. No new agent framework. No fake distributed trace.

Save this as pack_profile.py. Point it at a real tree. Read the bar before you blame the model.

#!/usr/bin/env python3
"""Lab profiler for agent context packing. Sample timings only."""
from __future__ import annotations

import csv
import os
import time
from dataclasses import dataclass, field
from pathlib import Path

SKIP_DIR_NAMES = {".git", "node_modules", "dist", ".venv", "__pycache__"}
TEXT_SUFFIXES = {".py", ".ts", ".js", ".md", ".json", ".toml", ".yml", ".yaml"}

@dataclass
class Stage:
    name: str
    ms: float = 0.0
    bytes: int = 0

@dataclass
class Report:
    stages: list[Stage] = field(default_factory=list)

    def add(self, name: str, ms: float, nbytes: int = 0) -> None:
        self.stages.append(Stage(name, ms, nbytes))

def should_skip_dir(name: str, use_ignores: bool) -> bool:
    if not use_ignores:
        return name in {".git"}
    return name in SKIP_DIR_NAMES

def walk_and_pack(root: Path, use_ignores: bool, cap_bytes: int) -> tuple[int, int, float]:
    t0 = time.perf_counter()
    total = 0
    files = 0
    chunks: list[bytes] = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if not should_skip_dir(d, use_ignores)]
        for name in filenames:
            path = Path(dirpath) / name
            if path.suffix.lower() not in TEXT_SUFFIXES:
                continue
            try:
                data = path.read_bytes()
            except OSError:
                continue
            files += 1
            if total >= cap_bytes:
                continue
            take = data[: max(0, cap_bytes - total)]
            chunks.append(take)
            total += len(take)
    packed = b"\n".join(chunks)
    ms = (time.perf_counter() - t0) * 1000
    return len(packed), files, ms

def wire_stub(payload_bytes: int) -> float:
    """Local serialization stand-in. Not a model measurement."""
    t0 = time.perf_counter()
    blob = b"x" * min(payload_bytes, 1_000_000)
    _ = hash(blob)
    return (time.perf_counter() - t0) * 1000

def apply_stub(root: Path) -> tuple[int, float]:
    t0 = time.perf_counter()
    patch = b"# lab patch marker\n"
    target = root / ".pack_profile_apply.txt"
    target.write_bytes(patch)
    ms = (time.perf_counter() - t0) * 1000
    return len(patch), ms

def bar(ms: float, total: float, width: int = 24) -> str:
    if total <= 0:
        return " " * width
    n = int(round(width * (ms / total)))
    n = min(width, max(0, n))
    return "#" * n + "." * (width - n)

def write_csv(path: Path, report: Report) -> None:
    with path.open("w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["stage", "ms", "bytes"])
        for s in report.stages:
            w.writerow([s.name, f"{s.ms:.3f}", s.bytes])

def main() -> None:
    root = Path(os.environ.get("PACK_ROOT", ".")).resolve()
    use_ignores = os.environ.get("PACK_IGNORES", "1") != "0"
    cap = int(os.environ.get("PACK_CAP", "750000"))
    packed, files, walk_ms = walk_and_pack(root, use_ignores, cap)
    report = Report()
    report.add("walk_pack", walk_ms, packed)
    report.add("wire_local_stub", wire_stub(packed), packed)
    apply_bytes, apply_ms = apply_stub(root)
    report.add("apply_stub", apply_ms, apply_bytes)
    total = sum(s.ms for s in report.stages) or 1.0
    print(f"root={root}")
    print(f"files_touched={files} packed_bytes={packed} ignores={use_ignores}")
    print("SAMPLE GRAPH ONLY. This is not a model benchmark.")
    for s in report.stages:
        print(f"{s.name:16} {s.ms:8.2f} ms  {bar(s.ms, total)}  {s.bytes} B")
    out = Path("pack_graph.csv")
    write_csv(out, report)
    print(f"wrote {out}")

if __name__ == "__main__":
    main()

Run the naive pass first. Watch the file count jump. Then turn ignores back on.

python3 pack_profile.py
PACK_ROOT=. PACK_IGNORES=0 PACK_CAP=750000 python3 pack_profile.py
PACK_ROOT=. PACK_IGNORES=1 PACK_CAP=750000 python3 pack_profile.py
cp pack_graph.csv pack_graph_ignores.csv

Want CPU names, not just bars? Keep a profile too. cProfile is enough for this lab.

python3 -m cProfile -o pack.prof pack_profile.py
python3 -c "import pstats; p=pstats.Stats('pack.prof'); p.sort_stats('cumtime').print_stats(20)"

If py-spy is on the box, record a picture. I still keep the CSV. Pictures rot. Rows diff.

py-spy record -o pack.svg -- python3 pack_profile.py

Sample output looks like this on my fixture. Treat it as a shape, not a score. Your milliseconds will move.

SAMPLE GRAPH ONLY. This is not a model benchmark.
walk_pack         412.17 ms  ################....  750000 B
wire_local_stub     6.02 ms  ....................  750000 B
apply_stub          1.14 ms  ....................  18 B

See the punchline? The walk ate the bar. The stub wire was a footnote. Apply was a shrug.

That graph is what I kept. Not a model leaderboard. Not a chat screenshot. A stacked bar that survived the debate.

People ask the wrong first question. They ask which model is smarter. I ask which stage owns the seconds.

If packing owns the seconds, a fancier model will still wait. It waits on your snapshot. It waits on your ignores.

Cap bytes on purpose. Unlimited packing is a fantasy. Agents choke on whole monorepos the way pockets choke on bricks.

I cap in the script because production prompts cap too. The cap is a flashlight. It shows which files won the lottery.

Change one ignore rule. Diff the CSV. Did walk_pack shrink? Did packed bytes drop? That is the experiment.

diff -u pack_graph.csv pack_graph_ignores.csv || true

Do not stop at local stubs forever. A stub cannot catch queue delay. It cannot catch TLS setup. It cannot catch a cold server.

When I needed a real wire stage, I used a throwaway endpoint. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which I treated as a lab lane for that slice only.

I did not use it to grade intelligence. I used it to give the wire a heartbeat. Packing still had to win or lose on my disk.

Hook the wire later if you want. Time connect, headers, and first byte yourself. Keep those rows next to walk_pack. Then the bar stays honest.

A fair test plan is boring. That is why it works. Same fixture. Same cap. One changed ignore. One changed endpoint.

Warm the disk once. Discard that run. Record the next three. Median the milliseconds. Do not publish the warm-up.

Refuse to mix variables. Do not change the model and the ignore file together. You will not know which lever moved.

If walk_pack still dominates, stop tuning prompts. Fix the grocery run. Add ignores. Narrow suffixes. Cap harder.

If the live wire dominates, then talk servers. Until then, the model is a suspect with an alibi.

I have watched teams invert this. They swap models weekly. They never weigh the snapshot. The spinner stays. The story changes clothes.

The stacked bar does not care about clothes. It cares about milliseconds. It cares about bytes.

Limitations are not fine print. They are the method. This script does not measure token quality. It does not measure correctness.

The wire stub is local hashing. It is not network truth. The apply stub writes one marker file. It is not git apply.

Binary files are skipped. Generated folders are skipped only when ignores are on. Symlink storms can still lie.

Who should not use this approach? Anyone selling a latency SLA from these rows. Anyone profiling a huge monorepo with ignores off. Anyone who needs a courtroom trace of a vendor model.

Also skip it if you will not keep the CSV. A one-off print is a vibe. A saved bar is evidence.

I still like arguments about engineering taste. Taste does not replace a clock. The clock does not replace taste either.

Keep both. Start with the grocery run. Save the stacked bar. Then decide whether the model even got a turn.

If you want a disposable lane for the live wire later, the free model access and free server option was enough for that lab slice. Pack the tree first anyway.

The Agent Loop Was Fast. The Trace Lied.

Dakota Lin — Mon, 14 Sep 2026 21:45:03 +0000

Your agent loop is probably faster than you think. Mine looked slow for a week. Then I moved one timer and the slowness evaporated.

Here is the mistake. You wrap the entire round trip in one clock. The number comes back huge, so you blame the model and start shopping for a smaller one. Wrong layer. A model round trip holds two clocks, and one timer smears them together.

So stop timing. Start tracing.

Two clocks hide inside one box

Time-to-first-token is queue plus prefill. Everything after that first token is decode. If TTFT dominates, you have a scheduling problem. If decode dominates, you have an output-length problem. Those two fixes share nothing. Which one is yours? A total will never tell you.

Then there is a third clock nobody wraps. The tool call. Bash, tests, a git status. Those block your loop while doing zero inference. How often is your slow model really a slow subprocess?

You need three spans, not one timer.

The harness

Each span appends one JSONL line. Extra fields ride along on the record.

# trace.py
import json, os, time, httpx
from contextlib import contextmanager

TRACE = 'spans.jsonl'

@contextmanager
def span(name, **fields):
    rec = {'span': name, 'start_ns': time.perf_counter_ns(), **fields}
    try:
        yield rec                     # add keys to rec before the block ends
    finally:
        rec['dur_ms'] = (time.perf_counter_ns() - rec['start_ns']) / 1e6
        with open(TRACE, 'a') as f:
            f.write(json.dumps(rec) + '\n')

The streaming call is where the split happens. Mark the first content chunk, then keep counting.

def round_trip(messages, tool):
    with span('prompt_build') as s:
        payload = {'model': MODEL, 'messages': messages, 'stream': True,
                   'stream_options': {'include_usage': True}}
        s['payload_bytes'] = len(json.dumps(payload))

    text = ''
    with span('model_round_trip') as s:
        t0 = time.perf_counter_ns()
        with httpx.stream('POST', BASE + '/chat/completions', json=payload,
                          headers={'Authorization': 'Bearer ' + KEY},
                          timeout=120.0) as r:
            r.raise_for_status()
            for line in r.iter_lines():
                if not line.startswith('data: '):
                    continue
                body = line[6:]
                if body == '[DONE]':
                    break
                chunk = json.loads(body)
                delta = chunk['choices'][0]['delta'].get('content')
                if delta and 'ttft_ms' not in s:
                    s['ttft_ms'] = (time.perf_counter_ns() - t0) / 1e6
                text += delta or ''
                if chunk.get('usage'):
                    s['prompt_tokens'] = chunk['usage']['prompt_tokens']
                    s['out_tokens'] = chunk['usage']['completion_tokens']

    with span('tool_exec', tool=tool['name']) as s:
        ok = tool['fn'](text)
        s['ok'] = bool(ok)
    return text, ok

One ttft_ms per round trip is enough. You are looking for a ratio, not a thesis.

The graph you keep

Do not keep a dashboard. Keep one bar stack and one ratio line. Run this after thirty or forty turns.

# report.py
import collections, json, statistics

spans = [json.loads(l) for l in open('spans.jsonl')]
by = collections.defaultdict(list)
for s in spans:
    by[s['span']].append(s['dur_ms'])

top = max(statistics.median(v) for v in by.values())
for name, xs in sorted(by.items(), key=lambda kv: -statistics.median(kv[1])):
    xs.sort()
    p50, p95 = statistics.median(xs), xs[int(len(xs) * 0.95) - 1]
    bar = '#' * int(40 * p50 / top)
    print(f'{name:17} n={len(xs):4d} p50={p50:8.1f}ms p95={p95:8.1f}ms {bar}')

rt = [s for s in spans if s['span'] == 'model_round_trip' and s.get('ttft_ms')]
q = sum(s['ttft_ms'] for s in rt)
d = sum(s['dur_ms'] - s['ttft_ms'] for s in rt)
print(f'queue+prefill {q:8.0f}ms  decode {d:8.0f}ms  ratio {q/d:.2f}')

Illustrative sample output from the script above, not a measurement of your loop:

model_round_trip  n=  62 p50=  1840.2ms p95=  9110.4ms ########################################
tool_exec         n=  58 p50=   118.7ms p95=   640.1ms ##
prompt_build      n=  62 p50=     0.9ms p95=     3.2ms
queue+prefill    68400ms  decode    49600ms  ratio 1.38

Read the ratio first. Above 1 means waiting outweighs generating. Below 0.2 means you are paying to write long answers. Then read the p50 gap between spans. If tool_exec sits in your top two, your model was never the problem.

One caveat worth repeating. A client-side TTFT includes your own network round trip. You cannot split scheduler delay from transit without server timestamps. Treat it as a proxy and stay honest about it.

What each shape tells you to fix

Trace shape	Likely cause	First change
Ratio above 1, p50 growing per turn	Context resends the whole history	Hash the history, send a diff
p50 flat, p95 on `model_round_trip` huge	Retries and timeouts	Cap retries, set an explicit timeout
`tool_exec` p50 over 1s	Blocking shell or test run	Run tools concurrently, cache results
`prompt_build` in your top three	Re-serializing large payloads	Build once, reuse the bytes

Now the metric that actually matters. Round trips are motion. Accepted edits are progress. Divide your token usage by accepted edits, meaning patches that survive review or pass a test. If a fix cut p50 in half but doubled tokens per accepted edit, you made the loop cheaper and the work worse.

Where does MonkeyCode fit here? Nowhere in the loop logic, honestly. It fits as a place to run the harness. The operator states that model access is free within a 10M-token allowance and that a free server option is available, which is enough to collect a few thousand spans without your laptop fan becoming a variable. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That allowance is an iteration budget, not an infinite one. The arithmetic is yours to run: 10,000,000 divided by your own tokens per accepted edit gives a rough ceiling on how many useful turns you get. I will not guess that number for you, because it depends entirely on how chatty your prompt is.

Limitations, and who should skip this

Client spans cannot separate network from server scheduling. If you need that, you need server-side tracing instead.

Appending a JSONL line per span is real IO. Buffer it, or sample every tenth turn in production. Do not leave this in a hot path.

A single client in a single region is not an SLO rig. It is also not a benchmark, no matter how nice the bars look. And if your real cost is human review time, no trace will show it, because the clock you forgot to wrap is the one sitting in a chair.

Finally, skip all of this if you already know your bottleneck. Instrumentation you do not read is just overhead with good intentions.

Point the harness at any endpoint you have credentials for and run it for an hour. If you want somewhere free to spend your first thousand spans, MonkeyCode's free tier is one option, but start the trace before you judge it. You might find out the model was fine all along.

MonkeyCode provides free models that can run this workflow.

I Profiled the Spinner. Git Status Won.

Dakota Lin — Sun, 13 Sep 2026 20:17:36 +0000

The freeze was never the remote model at all. Your coding spinner just needed a simple villain. I kept a stall graph instead of vibes.

You ever watch an AI pane freeze hard? You blame tokens because the badge says thinking. I blamed tokens too, then I measured spans.

Git status won the bar fight cleanly. Decode time looked almost polite on the graph. This note is about stalls, not model quality.

I keep a waterfall of the coding loop. You can run the probe on tonight's repo. No extra cluster has to join this.

A spinner has only one facial expression. Slow still looks like slow from here. So we invent a story about tokens.

The free server must be cold tonight, right? The model must be thinking very hard? Those stories sound neat and still fail.

I treat the assistant like any other worker. I give every wait a stubborn name. Then I keep the graph beside the patch.

If decode owns only a thin slice, I stop yelling at tokens for sport. The loop looks clever in product demos. It is still a small queue.

Read files, build a prompt, call a server. Stream text, apply a patch, then repeat. Which step owns the freeze on your machine? Ask the graph. Do not ask the spinner.

I record five spans on every probe run. Workspace walk, git status, prompt build. Request wait, first token, then decode.

Then I print them as a single bar chart. Think of a subway delay board at night. One train is late down the line.

The board still shows every station in order. You want the red station, not the line. That is the whole method in one picture.

Here is the recorder I keep as a scratch file. Treat this as a local probe only. It does not praise a vendor yet. The timings it prints are illustrative sample output, not a lab claim.

# stall_graph.py — local probe, labeled sample harness
from __future__ import annotations

import json
import os
import subprocess
import time
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import Iterator, List


@dataclass
class Span:
    name: str
    start: float
    end: float = 0.0

    @property
    def ms(self) -> float:
        return (self.end - self.start) * 1000.0


@dataclass
class StallGraph:
    spans: List[Span] = field(default_factory=list)

    @contextmanager
    def span(self, name: str) -> Iterator[None]:
        s = Span(name=name, start=time.perf_counter())
        try:
            yield
        finally:
            s.end = time.perf_counter()
            self.spans.append(s)

    def dump(self) -> str:
        total = sum(s.ms for s in self.spans) or 1.0
        lines = []
        for s in self.spans:
            width = max(1, int(s.ms / total * 40))
            bar = "#" * width
            lines.append(f"{s.name:16} {s.ms:8.1f} ms  {bar}")
        return "\n".join(lines)


def walk_workspace(root: str, limit: int = 400) -> list[str]:
    out: list[str] = []
    skip = {".git", "node_modules", ".venv", "dist", "target"}
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if d not in skip]
        for name in filenames:
            if name.endswith((".py", ".ts", ".go", ".rs")):
                out.append(os.path.join(dirpath, name))
                if len(out) >= limit:
                    return out
    return out


def git_status(root: str) -> str:
    try:
        p = subprocess.run(
            ["git", "status", "--porcelain", "-uall"],
            cwd=root,
            capture_output=True,
            text=True,
            timeout=8,
        )
        return p.stdout
    except (FileNotFoundError, subprocess.TimeoutExpired):
        return ""


def assemble_prompt(files: list[str], status: str, question: str) -> str:
    parts = [f"STATUS:\n{status}\n", f"Q:\n{question}\n"]
    for path in files[:12]:
        try:
            with open(path, "r", encoding="utf-8", errors="ignore") as handle:
                parts.append(f"FILE {path}\n{handle.read()[:2000]}\n")
        except OSError:
            continue
    return "\n".join(parts)


def fake_remote(prompt: str) -> str:
    # Stand-in only. Swap this for a real HTTP call later.
    time.sleep(0.05)
    _ = hash(prompt) % 97
    time.sleep(0.12)
    return "ok"


def run_probe(root: str) -> None:
    g = StallGraph()
    with g.span("workspace_walk"):
        files = walk_workspace(root)
    with g.span("git_status"):
        status = git_status(root)
    with g.span("prompt_build"):
        prompt = assemble_prompt(
            files, status, "Where is the N+1 query?"
        )
    with g.span("request_wait"):
        time.sleep(0.01)  # DNS / connect stand-in
    with g.span("first_token"):
        _ = fake_remote(prompt)[:1]
    with g.span("decode"):
        _ = fake_remote(prompt)
    print(g.dump())
    print("---")
    print(json.dumps({s.name: round(s.ms, 1) for s in g.spans}, indent=2))


if __name__ == "__main__":
    run_probe(os.getcwd())

Run it on a dirty repo first tonight. Then run it on a tiny clean tree. Compare the bars like a crime scene.

python stall_graph.py

The block below is fake shape, not my laptop. Your numbers will move with disk heat. A dirty monorepo often shows this kind of cartoon.

workspace_walk     180.0 ms  ############
git_status         420.0 ms  ############################
prompt_build        90.0 ms  ######
request_wait        25.0 ms  #
first_token         40.0 ms  ##
decode              80.0 ms  #####

See the joke hiding in those bars? Tokens look shy while git looks loud. The spinner still writes thinking on its face. Who taught that badge such a sloppy word?

I needed a remote that skipped the credit card. I also needed that remote to stay boring. Boring helps a probe stay honest.

You want a stable far side for contrast. You want the near side to confess first.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I pointed the far spans at MonkeyCode for that contrast. Free model access and a free server option. That is why the project appears in this note. I am not ranking vendors here at all. I am holding the network still on purpose.

Point request_wait and decode at that server. Keep the local spans brutally honest. If you need a quiet far side, that free server option is enough to run the probe.

Do not paste secrets into the prompt body. Do not ship production keys with a probe. This is a stall graph, not a load test.

A thin client can look like the sketch below. Wire your own URL and your own auth. Do not treat the snippet as production code.

import json
import urllib.request


def remote_complete(prompt: str, url: str) -> str:
    body = json.dumps({"prompt": prompt, "stream": False}).encode()
    req = urllib.request.Request(
        url,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return resp.read().decode("utf-8", errors="replace")

Wrap the call until headers actually arrive. Then wrap the body read as decode. If you only store one blob, split it. A single span hides the handshake every time.

Why use a free server for the far side? Because the live question is local I/O. Pay for a huge box and the story bends.

You start hunting glamorous model wins again. The graph goes quiet and polite. I wanted the opposite mood for once. Cheap remote. Loud local. Let the stall speak.

If git status dominates the printed bars, stop asking porcelain on every keystroke. Cache it. Watch the index instead of polling. Skip -uall on huge trees.

That flag walks untracked files like a tourist. It is a trap inside a monorepo. You already knew that in your bones. The spinner made you forget the lesson.

If workspace walk dominates the same graph, stop crawling from the repo root. Pass the open file and the diff. Pass a tight pathspec and nothing else.

The model cannot read a forest faster than disk. It never could, even on a good day. A glob from root is not intelligence. It is just a cold walk.

If prompt build dominates, cap the file bytes. Hash the includes and refuse rereads. Prompt build is CPU plus disk in a costume. It still wears a model badge.

If request wait finally dominates, then talk network. DNS, TLS, idle connections, cold start. Then the free server is on the hook. Until that bar grows, leave the remote alone.

If decode dominates, you may have a model wait. Or you asked the stream for a novel. Still measure it. Do not guess it.

I rerun the probe three boring ways. Same question. Same laptop. I refuse one lucky run.

First comes a clean git worktree with no junk. Second comes a dirty tree with untracked build output. Third comes that dirty tree with node_modules still visible.

That third run is the horror movie version. You should watch it once and keep the printout. I keep bars. I do not keep vibes.

If the bars do not move after a fix, the fix is theater and you know it. Ship the graph with the patch. Future you will thank present you.

Here is a pytest sketch that guards the probe. It asserts shape, not a magic speed. Label this unexecuted until you run it. These tests will not prove production latency. They only keep the recorder from rotting.

# test_stall_shape.py
import os
from stall_graph import StallGraph, git_status, walk_workspace


def test_spans_are_named():
    g = StallGraph()
    with g.span("git_status"):
        git_status(os.getcwd())
    assert g.spans[0].name == "git_status"
    assert g.spans[0].ms >= 0.0


def test_walk_respects_limit(tmp_path):
    for i in range(50):
        (tmp_path / f"f{i}.py").write_text("x=1\n")
    files = walk_workspace(str(tmp_path), limit=10)
    assert len(files) == 10

This probe lies when the assistant is a black box. You cannot span what you cannot wrap. If file crawl lives inside a closed binary, you only see one blob.

Then you need OS tools, not this script. fs_usage, perf, procmon, whatever you trust. That is a different sport with heavier shoes.

This probe also lies under tiny toy repos. Everything looks like decode in a sandbox. Of course it does with nothing else to eat. Do not publish that graph as gospel.

Do not use this method to rank models. Free access is not a bake-off on quality. Token taste is another axis I ignored. If you need evals, walk away.

Do not use this for SLA theater either. There are no percentiles hiding in the script. There is no jitter model and no region story. One laptop. One waterfall. That is scope.

Skip this if you do not own the client. You cannot instrument a closed pane with Python. You would be guessing, and guessing feeds the spinner.

Skip it if your freeze is local GPU inference. That needs another graph and other clocks. I am talking about a coding loop that calls a server. That is the only scene.

I paste the last waterfall into the ticket. Not a screenshot of a lying spinner. The bars, the names, one blunt sentence.

Name the span that dwarfed decode, then the patch. Then a second graph after the change. If the red station moves, we learned something. If it stays, we stop arguing with folklore.

AI coding talk loves agents and long loops this week. A loop is still a wait with extra makeup. If-statements in a trench coat still call git status.

They still walk trees and pack bloated prompts. Measure that part before the costume parade. I did not check whether the model got smarter.

I checked whether the spinner lied to me. The stall graph did not lie back. Bring a dirty repo and run the recorder. Keep the graph.

I Logged Occupancy. It Never Left One.

Dakota Lin — Sat, 12 Sep 2026 18:22:49 +0000

The model was not my real bottleneck today. My agent loop was the stall. Occupancy never left one on the graph.

I had blamed the endpoint for a sluggish demo. The cursor spun and I narrated GPU pain. I had no occupancy line, only a story.

Does that spinning cursor feel like a GPU? Or is your client just being polite? I wanted a graph I could keep.

Think of a single-lane bridge at rush hour. Cars can be quick and still crawl. My tool loop was that bridge, one car forever.

Agent demos love this costume. One tool, then another, then a recap. Each call waits like a queued checkout.

The wall clock looks like a tired model. The occupancy graph looks like a shy client. I instrumented the client, not the box.

I needed a cheap second lane for the rerun. I pointed the same tracer at MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The tracer did not care which host I hit. That was the whole point of the harness. Local mock first, shared lane later.

The graph I actually kept

I stopped collecting vibes and started four stamps. Request start, first byte, last byte, in-flight count. Occupancy is a counter with a clock beside it.

Increment on send. Decrement when the body ends. Sample the counter while calls run.

If occupancy stays at one, concurrency is fan fiction. The model can still be perfectly fine. You are watching a polite queue you built.

Here is the tracer I ran against a mock first. It wraps any HTTP generate-style call you already have.

# occupancy_trace.py
from __future__ import annotations

import json
import threading
import time
from dataclasses import dataclass, field, asdict
from typing import Callable, List

@dataclass
class Span:
    name: str
    t0: float
    t_first: float | None = None
    t1: float | None = None
    bytes_in: int = 0

    @property
    def wait_first_ms(self) -> float:
        if self.t_first is None:
            return -1.0
        return (self.t_first - self.t0) * 1000.0

    @property
    def total_ms(self) -> float:
        if self.t1 is None:
            return -1.0
        return (self.t1 - self.t0) * 1000.0


@dataclass
class OccupancySample:
    t: float
    in_flight: int


class OccupancyLog:
    def __init__(self) -> None:
        self._lock = threading.Lock()
        self.in_flight = 0
        self.samples: List[OccupancySample] = []
        self.spans: List[Span] = []
        self.t_origin = time.perf_counter()

    def _stamp(self) -> None:
        t = time.perf_counter() - self.t_origin
        self.samples.append(OccupancySample(t=t, in_flight=self.in_flight))

    def start(self, name: str) -> Span:
        with self._lock:
            self.in_flight += 1
            self._stamp()
            span = Span(name=name, t0=time.perf_counter())
            self.spans.append(span)
            return span

    def first_byte(self, span: Span) -> None:
        if span.t_first is None:
            span.t_first = time.perf_counter()

    def done(self, span: Span, n: int = 0) -> None:
        span.t1 = time.perf_counter()
        span.bytes_in += n
        with self._lock:
            self.in_flight -= 1
            self._stamp()

    def dump_csv(self, path: str) -> None:
        with open(path, "w", encoding="utf-8") as f:
            f.write("t_s,in_flight\n")
            for s in self.samples:
                f.write(f"{s.t:.4f},{s.in_flight}\n")

    def ascii_graph(self, width: int = 48) -> str:
        if not self.samples:
            return "(empty occupancy log)"
        peak = max(s.in_flight for s in self.samples) or 1
        lines = ["occupancy over wall time (each char ~ a sample)"]
        for s in self.samples:
            bar = "#" * max(1, int(round((s.in_flight / peak) * width)))
            lines.append(f"{s.t:6.3f}s | {s.in_flight} | {bar}")
        return "\n".join(lines)

Why wrap the client like this? Because server dashboards lie about your loop. They see arrivals. They do not see your serial habits.

Run a local mock before you touch any shared box. You want a known stall you control. Otherwise you will mythologize queue noise.

# mock_generate.py
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import time

class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        body = json.loads(self.rfile.read(n) or b"{}")
        delay = float(body.get("delay_s", 0.25))
        text = body.get("text", "ok")
        time.sleep(delay)  # labeled mock stall, not a model
        payload = json.dumps({"text": text}).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

def serve(port: int = 8765):
    httpd = ThreadingHTTPServer(("127.0.0.1", port), Handler)
    httpd.serve_forever()

if __name__ == "__main__":
    serve()

Start that in one terminal. Keep it ugly and local. You are measuring the loop, not a vendor.

python mock_generate.py

The driver below plays a tiny agent. Three tools, always in a row. That is the costume I kept seeing.

# serial_loop.py
import json
import urllib.request
from occupancy_trace import OccupancyLog

URL = "http://127.0.0.1:8765"
log = OccupancyLog()

def generate(name: str, text: str, delay_s: float) -> dict:
    span = log.start(name)
    payload = json.dumps({"text": text, "delay_s": delay_s}).encode()
    req = urllib.request.Request(
        URL,
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        log.first_byte(span)
        raw = resp.read()
    log.done(span, n=len(raw))
    # local parse sits inside the "model" story unless you split it
    t_parse0 = __import__("time").perf_counter()
    out = json.loads(raw)
    parse_ms = (__import__("time").perf_counter() - t_parse0) * 1000.0
    print(f"parse {name}: {parse_ms:.2f} ms")
    return out

def serial_agent():
    generate("plan", "plan tools", 0.20)
    generate("tool_search", "search hits", 0.35)
    generate("tool_fetch", "fetch body", 0.35)
    generate("recap", "final answer", 0.20)

if __name__ == "__main__":
    serial_agent()
    log.dump_csv("occupancy.csv")
    print(log.ascii_graph())
    for s in log.spans:
        print(
            f"{s.name:12} first={s.wait_first_ms:7.1f}ms "
            f"total={s.total_ms:7.1f}ms"
        )

Sample output from that local mock, not a cloud scoreboard. Treat it as a fixture, not a benchmark.

parse plan: 0.12 ms
parse tool_search: 0.08 ms
parse tool_fetch: 0.09 ms
parse recap: 0.11 ms
occupancy over wall time (each char ~ a sample)
 0.000s | 1 | #
 0.201s | 0 | #
 0.201s | 1 | #
 0.552s | 0 | #
 0.552s | 1 | #
 0.903s | 0 | #
 0.903s | 1 | #
 1.104s | 0 | #
plan         first=  200.4ms total=  201.1ms
tool_search  first=  350.2ms total=  350.9ms
tool_fetch   first=  350.1ms total=  350.8ms
recap        first=  200.3ms total=  201.0ms

See the occupancy column? It never leaves one. Four mountains, four valleys, zero overlap. That is a single-lane bridge with good manners.

Would a bigger model fix this graph? It would paint the same teeth. Taller teeth, same gaps, same story.

What I changed after the graph

I did not tune temperature. I changed the shape of the loop. Independent tools can share a moment. Dependent tools cannot.

Search and fetch were coupled in my first script. They were coupled by habit, not by data. Habit is a bottleneck that wears a model badge.

The second driver fires independent work together. Occupancy should climb if the mock allows it. The mock is threaded on purpose.

# overlap_loop.py
import json
import threading
import urllib.request
from occupancy_trace import OccupancyLog

URL = "http://127.0.0.1:8765"
log = OccupancyLog()

def generate(name: str, text: str, delay_s: float) -> dict:
    span = log.start(name)
    payload = json.dumps({"text": text, "delay_s": delay_s}).encode()
    req = urllib.request.Request(
        URL, data=payload,
        headers={"Content-Type": "application/json"}, method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        log.first_byte(span)
        raw = resp.read()
    log.done(span, n=len(raw))
    return json.loads(raw)

def overlap_agent():
    generate("plan", "plan tools", 0.20)
    batch = [
        threading.Thread(target=generate, args=("tool_a", "a", 0.35)),
        threading.Thread(target=generate, args=("tool_b", "b", 0.35)),
    ]
    for t in batch:
        t.start()
    for t in batch:
        t.join()
    generate("recap", "final answer", 0.20)

if __name__ == "__main__":
    overlap_agent()
    log.dump_csv("occupancy_overlap.csv")
    print(log.ascii_graph())

Expected local shape, still a fixture. Occupancy should tick to two during the tool batch. Recap still waits, because recap depends on both.

 0.000s | 1 | ##############
 0.201s | 0 | ##############
 0.201s | 1 | ##############
 0.202s | 2 | ################################
 0.552s | 1 | ##############
 0.553s | 0 | ##############
 0.553s | 1 | ##############
 0.754s | 0 | ##############

Did the mock get faster? No. The loop stopped pretending it was one car. Wall time dropped because overlap existed.

Plot the CSV if you hate ASCII. A two-line Python snippet is enough.

# plot_occupancy.py  (optional; needs matplotlib)
import csv
import sys

xs, ys = [], []
with open(sys.argv[1], newline="") as f:
    for row in csv.DictReader(f):
        xs.append(float(row["t_s"]))
        ys.append(int(row["in_flight"]))

try:
    import matplotlib.pyplot as plt
except ImportError:
    print("no matplotlib; print pairs instead")
    for x, y in zip(xs, ys):
        print(x, y)
else:
    plt.step(xs, ys, where="post")
    plt.xlabel("wall seconds")
    plt.ylabel("in-flight calls")
    plt.title("occupancy (client truth)")
    plt.ylim(0, max(ys) + 1)
    plt.savefig(sys.argv[2] if len(sys.argv) > 2 else "occupancy.png")

python plot_occupancy.py occupancy.csv occupancy.png
python plot_occupancy.py occupancy_overlap.csv occupancy_overlap.png

Keep both PNGs next to the agent. When someone says the model got worse, open the occupancy line first. Ask whether in-flight ever moved.

A small decision matrix I still use

I keep this beside the graph because advice without a gate becomes folklore. It is a client checklist, not a vendor scorecard.

What the graph shows	What I try next	What I refuse to do
Occupancy stuck at 1	Overlap independent tools	Buy a "faster" model first
Occupancy climbs, first-byte still huge	Look at queue, DNS, TLS	Blame decode tokens
Occupancy climbs, totals shrink	Keep the parallel shape	Rewrite the whole agent
Occupancy climbs, errors bloom	Cap in-flight, add jitter	Hammer a free shared lane

How do you know tools are independent? If tool B does not need tool A's bytes, they can overlap. If B needs A's id, you still have a bridge.

I still serialize the recap. Recap is a merge, not a hero. Merges wait. That wait should look like occupancy dropping to one.

Pointing the same tracer at a shared lane

After the mock told the truth, I reused the wrapper. Same stamps, different URL. MonkeyCode stayed a lane, not the lesson.

I will not paste shared-lane timings here. Those numbers move with neighbors you cannot see. Occupancy remains a client fact even when the lane is busy.

If occupancy stays one on a free server, your loop is still serial. If occupancy climbs and waits explode, you found queueing. Those are different diseases.

Do not treat a free server as a private GPU. Do not shove secrets into a shared prompt box. Do not turn retries into a stampede.

A free lane is useful for shaping the client. It is a poor SLA. It is a worse compliance story.

Limitations, said plainly

This tracer does not see kernel scheduling. It does not see GPU SM occupancy. It sees your process, your sockets, your manners.

First-byte includes queue you do not own. Last-byte includes network you do not own. Occupancy is still the cleanest lie detector I have for loops.

Threading helps the mock. It will not magically make dependent tools honest. Deadlocks are still available if you lock around recap.

urllib is blocking. That is deliberate for a teaching harness. Your production client might be async and still serial by await order.

Who should skip this approach? Anyone with a hard p99 contract. Anyone who cannot put prompts on a shared box. Anyone hoping a graph replaces an SLO.

Also skip it if you only have one call. Occupancy of one is then correct. You need a loop before this graph gets spicy.

The question I leave on the graph

When the demo crawls, who is in flight? If the answer is always one, stop dressing the loop as a model.

I still keep the CSV. I still print the ASCII teeth. The picture is ruder than a dashboard screenshot.

If you want a spare lane for the same tracer, MonkeyCode's free model access and free server option is the one I pointed at after the mock. The occupancy line remains yours either way.

I Split the Wait Into Three Clocks

Dakota Lin — Fri, 11 Sep 2026 15:54:23 +0000

One wall-clock number lied about the model. It blamed generation for every stall I felt. The graph I kept made the split obvious.

You feel a pause and name it inference. Do you actually know that? I didn't until three timestamps existed.

I still watch coding agents do this live. Someone wraps await complete() in a timer. They screenshot the total and shrug. They call the remote path dead after one run.

A single elapsed time is a blended drink. Queue time dissolves inside that glass. Network time dissolves beside it quietly. Your JSON parse dissolves last, smiling.

Generation drinks the insult alone. That is the whole measurement bug. We argue about models with dirty clocks. Then we twist the wrong knob for days.

I wanted a graph I could keep overnight. Not a vibe about intelligence. Not a badge on a vendor card. Three clocks on every call, always.

I stamp t_send when bytes leave me. I stamp t_first on the first inbound byte. I stamp t_last when the stream goes quiet. Anything after t_last belongs to me.

Does that sound too simple for production work? It is painfully simple on purpose. People skip simple measurements every single week. Complexity makes a gorgeous hiding place.

Think about a late dinner at home. You blame the oven immediately, right? You shopped after preheating the rack though. Then you plated for twenty slow minutes.

The oven was never the whole story. Remote inference gets that same unfair blame. Shared free paths get it even worse. Noise turns into mythology by morning.

I needed names for each wait. I use three names, only three. I call them queue-ish, generation-ish, and mine. They are nicknames, not laboratory physics.

Queue-ish is t_first minus t_send. It includes DNS lookup time. It includes the TLS handshake. It includes server admission. It includes a proxy that hoards bytes.

Generation-ish is t_last minus t_first. It includes real token emission. It also includes server-side buffering. A stream flag can still dump once.

Mine is work after t_last returns. Logging frameworks love this slice. Schema validators love it too. Pretty printers rewrite the world slowly.

Why three clocks, not ten spans? Ten clocks never get kept. Three clocks survive a working afternoon. The CSV stays small enough to open.

Here is the harness I keep locally. Treat every line as a labeled example. This snippet is unexecuted in this article. Point it at your own URL.

# labeled example: three-clock stream harness, not a vendor scoreboard
import csv, json, sys, time, urllib.request
from pathlib import Path

url, prompt, token = sys.argv[1], sys.argv[2], sys.argv[3]
body = json.dumps({
    "stream": True,
    "messages": [{"role": "user", "content": prompt}],
}).encode()

req = urllib.request.Request(
    url,
    data=body,
    method="POST",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {token}",
        "Accept": "text/event-stream",
    },
)

t_send = time.perf_counter()
cpu0 = time.process_time()
t_first = t_last = None
chunks = nbytes = 0
gaps = []
last = t_send

with urllib.request.urlopen(req, timeout=120) as resp:
    while True:
        piece = resp.read(256)
        now = time.perf_counter()
        if not piece:
            break
        if t_first is None:
            t_first = now
        gaps.append(now - last)
        last = now
        t_last = now
        chunks += 1
        nbytes += len(piece)

t_done = time.perf_counter()
cpu1 = time.process_time()
if t_first is None:
    raise SystemExit("no bytes returned; cannot split the wait")

queue_s = t_first - t_send
gen_s = t_last - t_first
mine_s = t_done - t_last
cpu_s = cpu1 - cpu0
max_gap = max(gaps) if gaps else 0.0

row = {
    "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    "queue_s": f"{queue_s:.4f}",
    "gen_s": f"{gen_s:.4f}",
    "mine_s": f"{mine_s:.4f}",
    "cpu_s": f"{cpu_s:.4f}",
    "chunks": chunks,
    "bytes": nbytes,
    "max_gap_s": f"{max_gap:.4f}",
}
path = Path("wait_split.csv")
new = not path.exists()
with path.open("a", newline="") as f:
    w = csv.DictWriter(f, fieldnames=row.keys())
    if new:
        w.writeheader()
    w.writerow(row)
print(row)

Curl still earns a permanent seat beside it. HTTP hiding places show up there first. DNS and TLS often fake a slow model. I check those before I touch prompts.

# labeled example: ask curl to name the handshake
curl -sN -o /tmp/stream.bin \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  --data '{"stream":true,"messages":[{"role":"user","content":"ping"}]}' \
  -w "dns:%{time_namelookup} tcp:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n" \
  "$URL"

I append one CSV row per call. I keep the file beside the service. I refuse to trust memory tomorrow morning. Graphs beat hallway anecdotes when something pages.

Then I draw the split as stacked bars. A tiny script is enough here. Queue-ish gets one color. Generation-ish gets another. Mine gets a color that looks guilty.

# labeled example: draw the CSV you actually kept
import csv
from pathlib import Path

rows = list(csv.DictReader(Path("wait_split.csv").open()))
print("n=", len(rows))
for name in ("queue_s", "gen_s", "mine_s", "cpu_s"):
    vals = [float(r[name]) for r in rows]
    avg = sum(vals) / len(vals)
    print(f"{name} avg={avg:.3f} max={max(vals):.3f}")

Read those bars like a mechanic would. Do not read them like a launch blog. One sample is weather on a window. A week of samples is climate.

I use a tiny decision matrix here. It keeps my hands off the wrong knob. I taped a copy above the terminal. Yes, that part is sincere.

Dominant slice	What it usually is	What I refuse to tune
queue-ish	admission, DNS, TLS, proxy buffer	temperature, max tokens, prompt poetry
generation-ish, steady gaps	actual decode / long prompt	JSON pretty printers, extra log lines
mine, or CPU rising during read	client parse, tools, tracing	model shopping for its own sake
one cliff, then silence	someone buffered the "stream"	anything that assumes live tokens

If queue-ish dominates, I stop decoding. Tokens do not exist yet. I warm the connection instead. I reuse the session. I inspect the admission path.

If generation-ish dominates with steady gaps, tokens are real. Prompt size belongs in that fight. Packing belongs there too. That fight is a different note.

If mine dominates, I inspect my process. Receiving already finished cleanly. I was still narrating a wait. The model had left the building.

If chunk gaps look violently spiky, someone buffered. The stream flag was ceremonial, honestly. First byte arrived late on purpose. The graph shows a cliff, not rain.

I also watch cpu_s next to wall time. If CPU climbs during the so-called wait, I lied. I was parsing inside the read loop. I was pretty-printing tokens as they landed.

That bug felt like generation drag. It was me, talking too much. Logging each token to disk is expensive. So is coloring a terminal on every chunk.

I pointed this loop at a free remote path. I needed retries without burning a paid key.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access fit that loop. A free server option fit it too. I used both as a reachable target. I did not crown them with latency medals.

Could I have used any other URL? Yes, obviously I could. The harness does not care about brands. The lesson is the split itself.

What broke my first graphs badly? Proxies did. They delayed first byte for assembly. My queue-ish bar went cartoonish overnight. I thought the model was queued. The proxy was writing a novel.

TLS handshakes polluted queue-ish as well. Cold starts look like heavy congestion. I logged curl -w beside the Python client. The two stories had to rhyme.

Clock resolution lied on tiny replies. A thirty millisecond generation is just noise. Do not rank models on that jitter. Rank your parser instead, every time.

I commit the CSV on bad weeks only. Not forever, and not as marketing. Just long enough to compare two Mondays. The graph I kept is the argument.

Who should ignore this whole note? Anyone selling an SLA from it. This is a debug habit. It is not a bakeoff protocol. It will not survive a legal review.

Skip it if your API never streams. You will invent a first-byte fairy tale. The response is one fat chunk then. Three clocks collapse into two shrugs.

Skip it if you need quality evaluation. Tokens per second is not correctness. A fast wrong answer still loses users. Keep this graph next to tests.

Shared free capacity stays noisy on purpose. Neighbors exist whether you like them. One slow call proves almost nothing. I keep a week or I stay quiet.

I still catch myself cheating daily. I glance at total wall time. I invent a story in one breath. Then I open the CSV and stop.

The core move stays boring on purpose. Split the wait into three clocks. Name the biggest slice out loud. Fix only that slice this round.

Repeat until the graph looks dull. Dull is the actual goal here. Drama is usually a measurement bug. When all three slices shrink, ship.

If you already have an endpoint, run the harness. Keep the CSV next to the code. Argue with the bars, not with me.