I Logged Occupancy. It Never Left One.

The model was not my real bottleneck today. My agent loop was the stall. Occupancy never left one on the graph.

I had blamed the endpoint for a sluggish demo. The cursor spun and I narrated GPU pain. I had no occupancy line, only a story.

Does that spinning cursor feel like a GPU? Or is your client just being polite? I wanted a graph I could keep.

Think of a single-lane bridge at rush hour. Cars can be quick and still crawl. My tool loop was that bridge, one car forever.

Agent demos love this costume. One tool, then another, then a recap. Each call waits like a queued checkout.

The wall clock looks like a tired model. The occupancy graph looks like a shy client. I instrumented the client, not the box.

I needed a cheap second lane for the rerun. I pointed the same tracer at MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The tracer did not care which host I hit. That was the whole point of the harness. Local mock first, shared lane later.

The graph I actually kept

I stopped collecting vibes and started four stamps. Request start, first byte, last byte, in-flight count. Occupancy is a counter with a clock beside it.

Increment on send. Decrement when the body ends. Sample the counter while calls run.

If occupancy stays at one, concurrency is fan fiction. The model can still be perfectly fine. You are watching a polite queue you built.

Here is the tracer I ran against a mock first. It wraps any HTTP generate-style call you already have.

# occupancy_trace.py
from __future__ import annotations

import json
import threading
import time
from dataclasses import dataclass, field, asdict
from typing import Callable, List

@dataclass
class Span:
    name: str
    t0: float
    t_first: float | None = None
    t1: float | None = None
    bytes_in: int = 0

    @property
    def wait_first_ms(self) -> float:
        if self.t_first is None:
            return -1.0
        return (self.t_first - self.t0) * 1000.0

    @property
    def total_ms(self) -> float:
        if self.t1 is None:
            return -1.0
        return (self.t1 - self.t0) * 1000.0


@dataclass
class OccupancySample:
    t: float
    in_flight: int


class OccupancyLog:
    def __init__(self) -> None:
        self._lock = threading.Lock()
        self.in_flight = 0
        self.samples: List[OccupancySample] = []
        self.spans: List[Span] = []
        self.t_origin = time.perf_counter()

    def _stamp(self) -> None:
        t = time.perf_counter() - self.t_origin
        self.samples.append(OccupancySample(t=t, in_flight=self.in_flight))

    def start(self, name: str) -> Span:
        with self._lock:
            self.in_flight += 1
            self._stamp()
            span = Span(name=name, t0=time.perf_counter())
            self.spans.append(span)
            return span

    def first_byte(self, span: Span) -> None:
        if span.t_first is None:
            span.t_first = time.perf_counter()

    def done(self, span: Span, n: int = 0) -> None:
        span.t1 = time.perf_counter()
        span.bytes_in += n
        with self._lock:
            self.in_flight -= 1
            self._stamp()

    def dump_csv(self, path: str) -> None:
        with open(path, "w", encoding="utf-8") as f:
            f.write("t_s,in_flight\n")
            for s in self.samples:
                f.write(f"{s.t:.4f},{s.in_flight}\n")

    def ascii_graph(self, width: int = 48) -> str:
        if not self.samples:
            return "(empty occupancy log)"
        peak = max(s.in_flight for s in self.samples) or 1
        lines = ["occupancy over wall time (each char ~ a sample)"]
        for s in self.samples:
            bar = "#" * max(1, int(round((s.in_flight / peak) * width)))
            lines.append(f"{s.t:6.3f}s | {s.in_flight} | {bar}")
        return "\n".join(lines)

Why wrap the client like this? Because server dashboards lie about your loop. They see arrivals. They do not see your serial habits.

Run a local mock before you touch any shared box. You want a known stall you control. Otherwise you will mythologize queue noise.

# mock_generate.py
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import time

class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        body = json.loads(self.rfile.read(n) or b"{}")
        delay = float(body.get("delay_s", 0.25))
        text = body.get("text", "ok")
        time.sleep(delay)  # labeled mock stall, not a model
        payload = json.dumps({"text": text}).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

def serve(port: int = 8765):
    httpd = ThreadingHTTPServer(("127.0.0.1", port), Handler)
    httpd.serve_forever()

if __name__ == "__main__":
    serve()

Start that in one terminal. Keep it ugly and local. You are measuring the loop, not a vendor.

python mock_generate.py

The driver below plays a tiny agent. Three tools, always in a row. That is the costume I kept seeing.

# serial_loop.py
import json
import urllib.request
from occupancy_trace import OccupancyLog

URL = "http://127.0.0.1:8765"
log = OccupancyLog()

def generate(name: str, text: str, delay_s: float) -> dict:
    span = log.start(name)
    payload = json.dumps({"text": text, "delay_s": delay_s}).encode()
    req = urllib.request.Request(
        URL,
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        log.first_byte(span)
        raw = resp.read()
    log.done(span, n=len(raw))
    # local parse sits inside the "model" story unless you split it
    t_parse0 = __import__("time").perf_counter()
    out = json.loads(raw)
    parse_ms = (__import__("time").perf_counter() - t_parse0) * 1000.0
    print(f"parse {name}: {parse_ms:.2f} ms")
    return out

def serial_agent():
    generate("plan", "plan tools", 0.20)
    generate("tool_search", "search hits", 0.35)
    generate("tool_fetch", "fetch body", 0.35)
    generate("recap", "final answer", 0.20)

if __name__ == "__main__":
    serial_agent()
    log.dump_csv("occupancy.csv")
    print(log.ascii_graph())
    for s in log.spans:
        print(
            f"{s.name:12} first={s.wait_first_ms:7.1f}ms "
            f"total={s.total_ms:7.1f}ms"
        )

Sample output from that local mock, not a cloud scoreboard. Treat it as a fixture, not a benchmark.

parse plan: 0.12 ms
parse tool_search: 0.08 ms
parse tool_fetch: 0.09 ms
parse recap: 0.11 ms
occupancy over wall time (each char ~ a sample)
 0.000s | 1 | #
 0.201s | 0 | #
 0.201s | 1 | #
 0.552s | 0 | #
 0.552s | 1 | #
 0.903s | 0 | #
 0.903s | 1 | #
 1.104s | 0 | #
plan         first=  200.4ms total=  201.1ms
tool_search  first=  350.2ms total=  350.9ms
tool_fetch   first=  350.1ms total=  350.8ms
recap        first=  200.3ms total=  201.0ms

See the occupancy column? It never leaves one. Four mountains, four valleys, zero overlap. That is a single-lane bridge with good manners.

Would a bigger model fix this graph? It would paint the same teeth. Taller teeth, same gaps, same story.

What I changed after the graph

I did not tune temperature. I changed the shape of the loop. Independent tools can share a moment. Dependent tools cannot.

Search and fetch were coupled in my first script. They were coupled by habit, not by data. Habit is a bottleneck that wears a model badge.

The second driver fires independent work together. Occupancy should climb if the mock allows it. The mock is threaded on purpose.

# overlap_loop.py
import json
import threading
import urllib.request
from occupancy_trace import OccupancyLog

URL = "http://127.0.0.1:8765"
log = OccupancyLog()

def generate(name: str, text: str, delay_s: float) -> dict:
    span = log.start(name)
    payload = json.dumps({"text": text, "delay_s": delay_s}).encode()
    req = urllib.request.Request(
        URL, data=payload,
        headers={"Content-Type": "application/json"}, method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        log.first_byte(span)
        raw = resp.read()
    log.done(span, n=len(raw))
    return json.loads(raw)

def overlap_agent():
    generate("plan", "plan tools", 0.20)
    batch = [
        threading.Thread(target=generate, args=("tool_a", "a", 0.35)),
        threading.Thread(target=generate, args=("tool_b", "b", 0.35)),
    ]
    for t in batch:
        t.start()
    for t in batch:
        t.join()
    generate("recap", "final answer", 0.20)

if __name__ == "__main__":
    overlap_agent()
    log.dump_csv("occupancy_overlap.csv")
    print(log.ascii_graph())

Expected local shape, still a fixture. Occupancy should tick to two during the tool batch. Recap still waits, because recap depends on both.

 0.000s | 1 | ##############
 0.201s | 0 | ##############
 0.201s | 1 | ##############
 0.202s | 2 | ################################
 0.552s | 1 | ##############
 0.553s | 0 | ##############
 0.553s | 1 | ##############
 0.754s | 0 | ##############

Did the mock get faster? No. The loop stopped pretending it was one car. Wall time dropped because overlap existed.

Plot the CSV if you hate ASCII. A two-line Python snippet is enough.

# plot_occupancy.py  (optional; needs matplotlib)
import csv
import sys

xs, ys = [], []
with open(sys.argv[1], newline="") as f:
    for row in csv.DictReader(f):
        xs.append(float(row["t_s"]))
        ys.append(int(row["in_flight"]))

try:
    import matplotlib.pyplot as plt
except ImportError:
    print("no matplotlib; print pairs instead")
    for x, y in zip(xs, ys):
        print(x, y)
else:
    plt.step(xs, ys, where="post")
    plt.xlabel("wall seconds")
    plt.ylabel("in-flight calls")
    plt.title("occupancy (client truth)")
    plt.ylim(0, max(ys) + 1)
    plt.savefig(sys.argv[2] if len(sys.argv) > 2 else "occupancy.png")

python plot_occupancy.py occupancy.csv occupancy.png
python plot_occupancy.py occupancy_overlap.csv occupancy_overlap.png

Keep both PNGs next to the agent. When someone says the model got worse, open the occupancy line first. Ask whether in-flight ever moved.

A small decision matrix I still use

I keep this beside the graph because advice without a gate becomes folklore. It is a client checklist, not a vendor scorecard.

What the graph shows	What I try next	What I refuse to do
Occupancy stuck at 1	Overlap independent tools	Buy a "faster" model first
Occupancy climbs, first-byte still huge	Look at queue, DNS, TLS	Blame decode tokens
Occupancy climbs, totals shrink	Keep the parallel shape	Rewrite the whole agent
Occupancy climbs, errors bloom	Cap in-flight, add jitter	Hammer a free shared lane

How do you know tools are independent? If tool B does not need tool A's bytes, they can overlap. If B needs A's id, you still have a bridge.

I still serialize the recap. Recap is a merge, not a hero. Merges wait. That wait should look like occupancy dropping to one.

Pointing the same tracer at a shared lane

After the mock told the truth, I reused the wrapper. Same stamps, different URL. MonkeyCode stayed a lane, not the lesson.

I will not paste shared-lane timings here. Those numbers move with neighbors you cannot see. Occupancy remains a client fact even when the lane is busy.

If occupancy stays one on a free server, your loop is still serial. If occupancy climbs and waits explode, you found queueing. Those are different diseases.

Do not treat a free server as a private GPU. Do not shove secrets into a shared prompt box. Do not turn retries into a stampede.

A free lane is useful for shaping the client. It is a poor SLA. It is a worse compliance story.

Limitations, said plainly

This tracer does not see kernel scheduling. It does not see GPU SM occupancy. It sees your process, your sockets, your manners.

First-byte includes queue you do not own. Last-byte includes network you do not own. Occupancy is still the cleanest lie detector I have for loops.

Threading helps the mock. It will not magically make dependent tools honest. Deadlocks are still available if you lock around recap.

urllib is blocking. That is deliberate for a teaching harness. Your production client might be async and still serial by await order.

Who should skip this approach? Anyone with a hard p99 contract. Anyone who cannot put prompts on a shared box. Anyone hoping a graph replaces an SLO.

Also skip it if you only have one call. Occupancy of one is then correct. You need a loop before this graph gets spicy.

The question I leave on the graph

When the demo crawls, who is in flight? If the answer is always one, stop dressing the loop as a model.

I still keep the CSV. I still print the ASCII teeth. The picture is ruder than a dashboard screenshot.

If you want a spare lane for the same tracer, MonkeyCode's free model access and free server option is the one I pointed at after the mock. The occupancy line remains yours either way.