The Idle Gap Wore a Model Badge

The model did not steal my compile time. The idle gap did, wearing a model badge. I kept a wait histogram, not vibes.

Have you ever watched a purple flame chart? Did you trust the label on that bar? I did, and it lied without blinking.

Agent posts keep timing the model, then stopping. They skip the quiet seconds around each call. Those quiet seconds ate my whole build.

I still mix local compiles with a remote helper. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode sits in that loop as the helper, nothing else.

It offers free model access and a free server option. I treat both as a shared waiting room. I do not invent quotas, hardware, or uptime.

Strip the product name and the method still holds. You still need clocks that speak in spans. Vibes will dress a file walker as a model.

My old loop looked tidy on paper. Save. Ask the helper. Compile. Read. I named the middle bar inference, like a fool.

Why did that name stick so hard? The HTTP client blocked my shell. The compiler waited in the hallway, tapping one foot.

A single curl stopwatch cannot save you here. It folds walker time, queue time, and TLS time together. Then it stamps the pile as intelligence.

I wanted a clock that named the baton. Spans show who held it. Histograms show how often they dropped it.

I built a local fixture around a toy C target. Treat every millisecond below as fixture output. Do not read them as vendor proof.

The fixture walks source for prompt context. It compiles a tiny binary. It records waits that merely look like model time.

Here is the span logger I kept beside the Makefile.

# labeled local fixture — not a vendor benchmark
import json, time, os, contextlib
from pathlib import Path

LOG = Path("wait_hist.jsonl")

@contextlib.contextmanager
def span(name):
    t0 = time.perf_counter()
    err = None
    try:
        yield
    except Exception as e:
        err = repr(e)
        raise
    finally:
        dt = time.perf_counter() - t0
        rec = {"name": name, "ms": round(dt * 1000, 2), "err": err}
        with LOG.open("a") as f:
            f.write(json.dumps(rec) + "\n")
        print(f"{name:18s} {rec['ms']:8.2f} ms")

Short, boring, honest. Fancy tracers hide idle gaps in color. This file will not.

The next function pretends to prepare a prompt. It walks files and concatenates slices. It never calls a model.

SKIP_PARTS = {".git", "build", "dist", "node_modules", "vendor"}

def allowed(path: Path) -> bool:
    return not any(part in SKIP_PARTS for part in path.parts)

def assemble_context(root="src"):
    chunks = []
    with span("assemble_context"):
        for p in Path(root).rglob("*.c"):
            if not allowed(p):
                continue
            if p.stat().st_size > 200_000:
                continue
            chunks.append(p.read_text(errors="ignore")[:4000])
    return "\n".join(chunks)

See the trap yet? A slow walk looks like thinking. Your editor may do this before any token moves.

The badge still says AI. The disk said otherwise. Which story would you ship to a dashboard?

I paired the walker with a compile that should run early.

import subprocess

def compile_toy():
    with span("compile"):
        subprocess.check_call(["make", "-s", "toy"])

And a remote wait, labeled as a stand-in. No invented SLA. No model name.

import urllib.request

def ping_helper(payload: bytes):
    # Example only. Point HELPER_URL at your helper if you have one.
    url = os.environ.get("HELPER_URL", "http://127.0.0.1:9/")
    req = urllib.request.Request(
        url,
        data=payload[:2048],
        method="POST",
        headers={"Content-Type": "text/plain"},
    )
    with span("helper_wait"):
        try:
            urllib.request.urlopen(req, timeout=2)
        except Exception:
            time.sleep(0.4)  # fixture stand-in for queue delay

That sleep is a costume for a waiting room. A free shared box can stall. Your histogram must survive the stall.

The first driver was the hallway version. Context, helper, then compile. Human shaped. Machine hostile.

def run_serial():
    LOG.write_text("")
    ctx = assemble_context()
    ping_helper(ctx.encode())
    compile_toy()

The second driver starts compile while the helper waits. Context still happens. It no longer owns the hallway.

from concurrent.futures import ThreadPoolExecutor

def run_overlapped():
    LOG.write_text("")
    ctx = assemble_context()
    with ThreadPoolExecutor(max_workers=2) as pool:
        h = pool.submit(ping_helper, ctx.encode())
        c = pool.submit(compile_toy)
        h.result()
        c.result()

Is overlap always legal? No. If the helper rewrites a compile input, you race. I kept the helper read-only in this fixture.

No patch landed until make exited. Advice, not magic. Would you let a reviewer edit the source mid-cc?

The toy target is one file on purpose. Fat trees lie with extra confidence.

# Makefile
toy: src/toy.c
    $(CC) -O2 -o toy src/toy.c

These are the commands I typed, in order.

python3 -m py_compile spans.py
rm -f wait_hist.jsonl toy
python3 -c "from spans import run_serial; run_serial()"
cp wait_hist.jsonl hist_serial.jsonl
python3 -c "from spans import run_overlapped; run_overlapped()"
cp wait_hist.jsonl hist_overlap.jsonl
python3 hist.py hist_serial.jsonl
python3 hist.py hist_overlap.jsonl

hist.py folds jsonl into coarse buckets. No dashboard. No flame theater.

# hist.py — fixture summarizer
import json, collections, sys
from pathlib import Path

buckets = [(50, "0-50ms"), (200, "50-200ms"), (1000, "200ms-1s"), (1e9, ">1s")]
names = collections.defaultdict(list)
path = Path(sys.argv[1] if len(sys.argv) > 1 else "wait_hist.jsonl")

for line in path.read_text().splitlines():
    rec = json.loads(line)
    names[rec["name"]].append(rec["ms"])

for name, vals in names.items():
    print(name, "n=", len(vals), "max=", max(vals), "sum=", round(sum(vals), 1))
    counts = collections.Counter()
    for v in vals:
        for cap, label in buckets:
            if v <= cap:
                counts[label] += 1
                break
    print(" ", dict(counts))

Run serial, then overlapped. Keep both files. I dated mine. The graph is two sum lines, nothing prettier.

In this fixture, the walker often outran compile. Serial mode made helper_wait look huge. Overlap hid most of that wait behind cc.

The model never saw the stall. The hallway did. Do you still want to tune the prompt first?

Labels lie with a friendly accent. If your bar says inference, ask who walked. Ask who blocked make.

I once pointed the walker at the wrong root. It chewed generated code. The histogram spiked over one second, and I felt clever.

I thought I had tuned a prompt. I had tuned a glob. That is the whole comedy.

Picture a fire alarm in the lobby. You blame the kitchen. The toaster at reception was smoking.

I changed the loop with three dull rules. Compile first when the helper only comments. Overlap when it cannot touch inputs. Refuse overlap when a patch might land mid-compile.

No slogan. No badge. The idle gap hates that kind of boredom.

I also bounded the walker before any remote call. Generated trees stay out. Secrets stay out. The waiting room gets a postcard, not the house keys.

Already using free models on a free server? Steal the histogram, not another prompt. Let make run while the queue does queue things.

Was the change glamorous? No. The model did not get faster. My naming did.

This fixture is not a distributed trace. ThreadPoolExecutor will not save a heavy walk. A two-second timeout is a local choice, not physics.

Free servers can queue, throttle, or nap. I will not guess their current limits. I will not publish fixture milliseconds as proof of a product.

Those numbers rot when your tree changes. They exist to catch a costume. They are not a leaderboard.

Who should skip this? Air-gapped shops. Teams with a hard inference SLA. Anyone whose helper must patch files the compiler reads.

If your build is already fully local, you do not need a waiting room. If you will not keep the jsonl, you are back to vibes.

I kept the file beside the Makefile. That is the trick. A histogram you delete is a story you already forgot.

When a bar turns purple, I ask three questions. Who walked? Who waited? Who compiled?

If those answers collapse into the model, I start over. The idle gap loves a costume. This month the costume is agents.

The histogram does not care about costumes. It only cares who held the baton. Would you still tune the prompt first?