I Profiled the Client. Inference Was Idle.

The p99 never lived inside the model itself. It hid inside prompt rebuilds I kept ignoring. I wasted nights staring at token clocks anyway.

Sound familiar if you ship a coding agent now? We treat the model like the entire machine. Then a profiler opens and the story flips.

I started with the graph everyone keeps first. Wall clock versus generated tokens looked almost honest. Longer answers took longer, so I blamed inference.

Wouldn't you have made that same bad call? That graph lies by omission every single time. It never shows work before the request leaves.

It skips the walk across your working tree. It skips JSON stitching that grows every turn. Boring client work never makes a pretty dashboard.

I kept a second graph after that night. Phases, not tokens: assemble, wait, then apply. The wait bar shrank while assemble kept climbing.

The tracer

This is a local tracing recipe you can rerun. Treat it as a lab loop, not production gospel. I used Python because the client already spoke it.

You do not need a GPU for this. You need a slow disk and a messy repo. That ugly combo is the entire point here.

# labeled lab tracer — rerun against your own tree
from __future__ import annotations

import os
import time
from collections import defaultdict
from contextlib import contextmanager

TIMING: dict[str, list[float]] = defaultdict(list)


@contextmanager
def span(name: str):
    t0 = time.perf_counter()
    try:
        yield
    finally:
        TIMING[name].append(time.perf_counter() - t0)


def dump_timeline(turn: int) -> None:
    print(f"--- turn {turn} ---")
    for name, samples in TIMING.items():
        ms = samples[-1] * 1000
        bar = "#" * max(1, int(ms / 20))
        print(f"{name:12} {ms:8.1f}ms {bar}")

Wire those spans around the three real phases. Do not wrap the HTTP call only, ever. That single wrap is how the first graph lied.

def assemble_prompt(root: str) -> str:
    chunks: list[str] = []
    with span("assemble"):
        for dirpath, _, files in os.walk(root):
            for name in files:
                if not name.endswith((".py", ".ts", ".go")):
                    continue
                path = os.path.join(dirpath, name)
                with open(path, "r", encoding="utf-8", errors="ignore") as fh:
                    chunks.append(fh.read())
    return "\n".join(chunks)


def wait_on_model(prompt: str) -> str:
    # labeled stand-in; swap for your completion client
    with span("wait"):
        time.sleep(0.05)
        return "// patch placeholder\n"


def apply_patch(diff: str) -> None:
    with span("apply"):
        _ = diff.splitlines()

Run it across several turns and keep the bars. That printout is the graph I still keep. A checked-in timeline diffs cleanly in git.

python tracer.py --root . --turns 8

Watch assemble on turn one without any mercy. Then watch the same bar on turn eight. Did it fall, or did it climb like ivy?

The climb

I expected wait to dominate after the first walk. Caching should make assemble cheap after that, right? The lab run did not agree with me.

Every turn rebuilt the prompt from disk again. The agent had no snapshot of the tree. It walked files like a tourist with paper maps.

Each tool result bloated the next stuffed prompt. Assemble grew while wait stayed almost embarrassingly flat. I had been tuning the wrong stick.

Here is a sample timeline from one lab run. Treat every number as illustration, not a claim. Your disk and tree will disagree with mine.

--- turn 1 ---
assemble        412.0ms ####################
wait             51.2ms ##
apply             8.4ms #

--- turn 4 ---
assemble        903.7ms #############################################
wait             48.9ms ##
apply            11.1ms #

--- turn 8 ---
assemble       1884.2ms ##############################################################################################
wait             55.0ms ##
apply            14.6ms #

See the shape sitting there in plain text? Inference is a thin stick on the page. Context rebuild is the freight train behind it.

Why do we keep falling for token dashboards? They are pretty and they feel expensive. File walks are boring, so they hide the p99.

The snapshot

I did not switch models as the first move. I snapshotted the prompt and walked the tree once. Then I fed only dirty files plus an index.

import hashlib


def fingerprint(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(65536), b""):
            h.update(block)
    return h.hexdigest()


class PromptSnapshot:
    def __init__(self) -> None:
        self.files: dict[str, tuple[str, str]] = {}

    def refresh(self, root: str) -> str:
        with span("assemble"):
            dirty: list[str] = []
            seen: set[str] = set()
            for dirpath, _, files in os.walk(root):
                for name in files:
                    if not name.endswith((".py", ".ts", ".go")):
                        continue
                    path = os.path.join(dirpath, name)
                    seen.add(path)
                    fp = fingerprint(path)
                    old = self.files.get(path)
                    if old is None or old[0] != fp:
                        with open(path, "r", encoding="utf-8", errors="ignore") as fh:
                            text = fh.read()
                        self.files[path] = (fp, text)
                        dirty.append(path)
            for path in list(self.files):
                if path not in seen:
                    del self.files[path]
                    dirty.append(path)
            index = "\n".join(sorted(self.files))
            payload = "\n".join(self.files[p][1] for p in dirty if p in self.files)
            return index + "\n\n" + payload

Rerun the tracer after the snapshot finally lands. The assemble bar should collapse after turn one. If it does not, your dirty detector is lying.

Hash the file bytes, not the mtime stamp. Networked disks lie about mtime under concurrent writes. I learned that lesson the slow way twice.

The wait bar might rise a little after this. Shorter prompts can change what the model does. That is a different experiment on a different graph.

A free wait bar

I still needed a real completion call sometimes. Local fakes hide the true wait shape. Paid APIs punish you for looping a tracer.

That loop is the method, not a demo. Burning paid tokens to time a file walk feels backwards. So I parked wait on a free endpoint.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. I pointed the wait span there while chasing assemble.

The product is not the profiler in this writeup. It is just a bench where the wait bar can live. I am not publishing model names, quotas, or hardware.

Those numbers go stale before the ink dries. Time your own client against your own tree. Swap the fake wait for your HTTP client.

Keep the span names honest or the graph lies again. The next block is a tiny stand-in client. Replace the URL with whatever you actually run.

import urllib.request


def real_wait(prompt: str, url: str) -> str:
    body = prompt.encode("utf-8")
    req = urllib.request.Request(url, data=body, method="POST")
    req.add_header("Content-Type", "text/plain")
    with span("wait"):
        with urllib.request.urlopen(req, timeout=60) as resp:
            return resp.read().decode("utf-8", errors="replace")

If wait explodes while assemble stays tiny, stop. Then you earned a server-side story at last. Most of my traces never got that far.

Who should skip this

This method misses GPU kernels entirely, by design. It misses tokenizer internals and shared queue delay. It only sees what the client thread can time.

It also punishes huge monorepos on spinning disks. That pain is useful and not universal. A one-file script will look almost boring.

Do not use this if you need energy numbers. Do not use it to crown a model vendor. Do not paste it onto a sales slide.

The graph is a client autopsy, nothing else. Snapshotting file text can leak secrets into logs. Redact, cap sizes, and skip .env files.

Skip node_modules and every build artifact too. I should not have to write that reminder. Treat secret files as radioactive in every trace.

If your agent already streams packed context from a daemon, leave. You paid the walk cost once already. Profile that daemon instead of this client.

The graph I keep

I keep the phase graph, not the token graph. Tokens still matter when the bill arrives. They do not explain why the UI stalled.

Ask the ugly question before you buy more context. Is the model waiting on your file walk? Or are you actually waiting on the model?

Those two waits feel identical in a chat box. They are not identical once a profiler speaks. Run the tracer on a dirty repo tonight.

Keep the bars close to the failing turn. If assemble wins, stop tuning inference now. Tune the snapshot until the train leaves the page.

If wait wins, then talk about servers. A free server path is optional on purpose. The tracer still works against a stubbed wait.

Use a live endpoint when the wait bar must be real. That is why a free endpoint showed up here. Park the loop, then actually read the graph.