I Kept the Waterfall. The Client Paid.

The remote model rarely owned my wall clock. Local client work stole those seconds from me instead. I kept one waterfall graph after seeing that.

Blame is cheap when a spinner hangs on you. Everyone points at weights and batch size first. I did the same, then I finally measured.

Have you ever watched a slow model that barely ran? I have, on one quiet evening at the desk. The GPU was not the villain that night.

A coding loop is a relay race, not a cannon shot. Prompt build hands off to JSON encoding next. JSON then hands off to the wire itself.

The client is a kitchen line during dinner. The model is only the oven at the back. You can still wait all night on chopping.

The wire hands off to parse, then a tool. Each baton pass looks innocent in isolation today. The stack of those passes is the real tax.

I needed a picture a skeptic would actually trust. Means lie when the tail gets fat fast. One waterfall tells the true order of pain.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I pointed this harness at MonkeyCode's free model access and free server. I am not selling a quota story here.

The product was just the remote side of this. The lesson still lives in the client. Remove the name and the method still holds.

What I actually recorded

I did not start with perf on a kernel. I wrapped the Python client in named spans. Each span got a name, a start, and an end.

Wall clock beats CPU time for this bug class. You can burn no cores and still wait. Waiting is still user pain in this loop.

So I treated the assistant like a tiny distributed system. It was one process with many sequential hops. There was no magic hiding in the stack.

The harness below is a lab notebook, not a production claim. Run it on your own machine later. Keep the graph that you actually measured yourself.

# span_waterfall.py
from __future__ import annotations

import json
import os
import time
import urllib.request
from dataclasses import dataclass, field
from typing import List


@dataclass
class Span:
    name: str
    start: float
    end: float = 0.0
    detail: str = ""

    @property
    def ms(self) -> float:
        return (self.end - self.start) * 1000.0


@dataclass
class Trace:
    spans: List[Span] = field(default_factory=list)

    def start(self, name: str, detail: str = "") -> Span:
        span = Span(name=name, start=time.perf_counter(), detail=detail)
        self.spans.append(span)
        return span

    def stop(self, span: Span) -> None:
        span.end = time.perf_counter()

    def t0(self) -> float:
        return self.spans[0].start if self.spans else 0.0

That is the whole religion for this note. Named intervals sit on a monotonic clock only. Nothing else gets to argue with the bars.

I then added a painter for the review. ASCII is enough when the fight is order. SVG can wait for a prettier dashboard later.

def render_ascii(trace: Trace, width: int = 56) -> str:
    if not trace.spans:
        return "(empty trace)"
    t0 = trace.t0()
    t1 = max(s.end for s in trace.spans)
    total = max(t1 - t0, 1e-9)
    lines = [f"total {total * 1000:.1f} ms  |  {len(trace.spans)} spans"]
    lines.append("span".ljust(16) + " waterfall".ljust(width + 2) + " ms")
    for s in trace.spans:
        a = int(((s.start - t0) / total) * width)
        b = max(int((s.ms / 1000.0 / total) * width), 1)
        bar = (" " * a + "#" * b)[:width].ljust(width)
        lines.append(f"{s.name[:16].ljust(16)}|{bar}| {s.ms:7.1f}")
    return "\n".join(lines)

Look at that bar across the page. The gap before the hash is queue time. The hash is work, or wait dressed as work.

Which of those hashes is the model, really? Only the HTTP span talks to remote weights. The rest is your process talking to itself.

The experiment I actually ran

I faked a two-tool coding turn on purpose. First turn asks for a file from disk. Second turn tries to answer with context.

That shape matches how agents chew a repo. It also multiplies the quiet client tax. A single model call will hide that tax.

Plug your base URL in through the environment. Do not hardcode a fantasy path tonight. I did not invent a vendor API here.

def post_json(url: str, payload: dict, timeout: float = 60.0) -> dict:
    raw = json.dumps(payload).encode("utf-8")
    req = urllib.request.Request(
        url,
        data=raw,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read().decode("utf-8"))


def run_turn(trace: Trace, url: str, prompt: str, tools: int) -> None:
    s = trace.start("build_prompt", detail=str(len(prompt)))
    body = {"prompt": prompt, "max_tokens": 64}
    trace.stop(s)

    s = trace.start("json_dumps")
    _ = json.dumps(body)
    trace.stop(s)

    s = trace.start("http_post")
    try:
        _resp = post_json(url, body)
    except Exception as exc:
        _resp = {"error": type(exc).__name__}
    trace.stop(s)

    s = trace.start("json_parse")
    _ = json.dumps(_resp)
    trace.stop(s)

    for i in range(tools):
        s = trace.start(f"tool_{i}")
        time.sleep(0.05)  # labeled stand-in for a small local tool
        trace.stop(s)

The sleep is labeled as a stand-in tool. Swap it for a real tool if you want blood. Do not ship the sleep into production code.

Then I drive two turns and print the graph. That printed graph is the artifact I keep. I throw away every chart that cannot show order.

def main() -> None:
    url = os.environ.get("REMOTE_URL", "http://127.0.0.1:9/does-not-exist")
    trace = Trace()
    run_turn(trace, url, "find the handler", tools=1)
    run_turn(trace, url, "now patch it", tools=1)
    s = trace.start("render")
    picture = render_ascii(trace)
    trace.stop(s)
    print(picture)


if __name__ == "__main__":
    main()

Run it like a grown-up with captured stdout. Do not screenshot a spinner and call it data. A spinner has no owner and no duration.

python3 span_waterfall.py
REMOTE_URL="$REMOTE_URL" python3 span_waterfall.py
python3 span_waterfall.py > /tmp/waterfall.txt
wc -l /tmp/waterfall.txt

Example output below is illustrative from a dry run. Your numbers will move on your network. Steal the layout, not these fake milliseconds.

total 312.4 ms  |  11 spans
span            waterfall                                                ms
build_prompt    |##                                                      |     4.1
json_dumps      |  #                                                     |     0.8
http_post       |   ################                                     |   140.2
json_parse      |                   #                                    |     1.1
tool_0          |                    ###                                 |    50.4
build_prompt    |                       ##                               |     3.9
json_dumps      |                         #                              |     0.7
http_post       |                          ################              |   138.6
json_parse      |                                          #             |     1.0
tool_0          |                                           ###          |    50.3
render          |                                              #         |     0.6

See the two HTTP blocks in that print? That is the model budget, maybe, if the post worked. See the tools sitting between those HTTP blocks?

I even dumped JSON twice by accident. The second dump hid inside post_json itself. The waterfall cannot split a function you did not wrap.

I almost optimized the model first last week. Would you have done the same thing too? The graph killed that idea on contact.

The bottleneck that survived the graph

JSON was not the villain on short prompts. It will be later, with a fat repo dump. Watch the dump span when context swells hard.

The serial tool was the quiet tax. Fifty milliseconds feels like nothing at review time. Two tools and two turns eat a lunch.

Network failure is also a span, by design. A refused connection still draws a bar. That bar is honesty, not shame, for once.

When the http_post bar dominates, I tune the remote path. When the tool bars dominate, I stop blaming tokens. When build_prompt grows, I treat prompt assembly as the leak.

I kept that ASCII waterfall in the ticket. I threw out the pie chart the same hour. Pies hide order, and order is the plot.

A slow second call after a slow tool is a story. A mean of eighty milliseconds is only a shrug. I needed the story, not the shrug.

What I changed after the picture

I overlapped nothing at the start of this. I only named the crime on the page. Naming is cheaper than threads and queues.

Then I cached the prompt header in memory. Then I refused to dump the same file twice. Then I batched the cheap local tools together.

Did the model get faster after those cuts? No, and I honestly did not care then. The user-facing pause shrank because the client stopped napping between posts.

That is the whole punchline of the note. Remote free inference can be fast enough here. Your relay race may still be the problem.

A remote URL was a convenient far end. I did not need a local GPU theater. The spans would look the same against localhost.

If you try this, keep the graph that argues with you. Delete the graph that only flatters the client.

Who should not take this path

Do not use this harness as a model benchmark. It measures your client, on purpose, every time. Vendor leaderboards will laugh at these client bars.

Do not ship the time.sleep tool to production. It is a labeled stand-in for a local tool. Replace it or go home with empty hands.

If you cannot send prompts off-box, skip remote. Run those same spans against localhost instead. The waterfall still teaches without leaving the machine.

If you need hard SLAs, this note is not a contract. It is a debug habit for messy loops. Habits do not sign your uptime pages.

If you are tuning kernels, you are in another movie. Bring perf, not these ASCII bars, with you. We are not the same audience tonight.

The graph I still keep

I keep the waterfall with two HTTP hashes. I keep the gaps that sit between them. I keep the rule that follows that picture.

Client spans come first, model spans come second. When someone says the model got slower, I ask. Where is the span that actually grew today?

Show me the bar or drop the debate. That sounds harsh in a calm standup. It saved me from another fake optimization week.

Short sentences keep me honest on timers too. A long paragraph hides a missing timer easily. A missing timer hides a missing owner too.

So here is the practice I still run. Wrap the loop and print the bars. Keep the ugly one that argues back.

The pretty mean can leave the ticket. The waterfall has to stay in the review. I will not debate without that picture.