I Subtracted the Queue. The Fast Model Wasn't.

The number on your latency dashboard is not a model score. It is a traffic report wearing a lab coat.

I keep watching people rank free endpoints by time-to-first-token as if TTFT were a property of weights. It isn't. Not when the box is shared. Not when your so-called eval is standing in the same line as everyone else's autocomplete. You are scoring the parking lot. Then you publish a leaderboard. Then you act shocked when Tuesday disagrees with Thursday.

What happens when the tests we use to measure models get slower than the models? The tests start measuring the hallway.

I wanted a splitter I could run twice and trust. Wait on one side. Generation on the other. If wait dominates, I throw the run out. If generation dominates, I keep it. That is the whole method. It is boring on purpose. Boring is how you stop lying to yourself.

The lab for this is not a secret GPU in a closet. MonkeyCode is an open source project with free model access and a free server option, which is exactly the shape of infrastructure that makes queue contamination loud. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not naming a model, a quota, a SKU, or a duration I cannot verify. I am naming a measurement error you can reproduce on any shared endpoint, including that one.

Picture the failure. You send the same prompt ten times. Naive wall clock swears run 3 was a genius and run 7 was a brick. Split the timestamps and run 3 sat in queue for a blink while run 7 waited for a bus. Tokens per second never moved. Your ranking did. Are you still averaging those ten numbers like they mean the same thing?

I treat three clocks as sacred. t_submit is the instant the client writes the last request byte. t_first is the instant the first response byte lands. t_done is the instant the stream closes. Wait is first minus submit. Work is done minus first. Contamination is wait divided by wait plus work. If that ratio is high, the run is weather, not intelligence.

You want code, not a sermon. Fine. This harness is the artifact. The fixture traces are labeled samples, not a product benchmark I pretended to run.

# queue_split.py — labeled fixture + splitter. Not a vendor scorecard.
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Trace:
    run_id: str
    t_submit: float
    t_first: float
    t_done: float
    prompt_hash: str
    streaming: bool
    cold: bool

    @property
    def wait_s(self) -> float:
        return max(0.0, self.t_first - self.t_submit)

    @property
    def work_s(self) -> float:
        return max(0.0, self.t_done - self.t_first)

    @property
    def wall_s(self) -> float:
        return max(0.0, self.t_done - self.t_submit)

    @property
    def contamination(self) -> float:
        total = self.wait_s + self.work_s
        return 0.0 if total == 0 else self.wait_s / total

    def verdict(self, discard_at: float = 0.6) -> str:
        if not self.streaming:
            return "UNSPLITTABLE"
        if self.contamination >= discard_at:
            return "DISCARD"
        if self.contamination >= 0.2:
            return "WORK_ONLY"
        return "WALL_OK"


# Illustrative traces only. Same prompt_hash on purpose.
FIXTURE = [
    Trace("clean", 0.00, 0.12, 1.40, "p1", True, False),   # short wait, steady work
    Trace("queued", 0.00, 2.10, 3.38, "p1", True, False),  # long wait, SAME work interval
    Trace("slow", 0.00, 0.15, 3.05, "p1", True, False),    # short wait, actually slow generate
    Trace("buffered", 0.00, 2.80, 2.81, "p1", False, True), # non-streaming: clocks collapse
]


def rank(traces, key):
    return [t.run_id for t in sorted(traces, key=key)]


if __name__ == "__main__":
    streamed = [t for t in FIXTURE if t.streaming]
    print("naive wall rank:", rank(streamed, lambda t: t.wall_s))
    print("work-only rank:", rank(streamed, lambda t: t.work_s))
    for t in FIXTURE:
        print(
            f"{t.run_id:9} wait={t.wait_s:.2f}s work={t.work_s:.2f}s "
            f"contam={t.contamination:.2f} -> {t.verdict()}"
        )

Run it locally. No credentials. No cloud required for the lie detector itself.

python queue_split.py

You should see the reversal immediately. Naive wall clock ranks clean, then slow, then queued. Work-only ranking puts clean and queued in the same neighborhood and leaves slow as the actual drag. That flip is the bug. buffered comes back UNSPLITTABLE because a one-shot body gives you a single thud, not two clocks. If your eval client does that, stop calling it TTFT. You do not have TTFT. You have a door slam.

I pipe real calls through the same dataclass. The HTTP client is deliberately ugly. Environment variables so this article does not invent a hostname, a model id, or a token budget.

# live_probe.py — unexecuted template. Fill env. Do not treat output as a brand score.
import hashlib, json, os, time, urllib.request
from queue_split import Trace

URL = os.environ["EVAL_URL"]          # your endpoint, not mine
BODY = os.environ.get("EVAL_BODY", '{"prompt":"ping","stream":true}')

def stream_once(run_id: str, cold: bool) -> Trace:
    payload = BODY.encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t_submit = time.perf_counter()
    t_first = None
    with urllib.request.urlopen(req, timeout=120) as resp:
        while True:
            chunk = resp.read(64)
            now = time.perf_counter()
            if not chunk:
                break
            if t_first is None:
                t_first = now
        t_done = time.perf_counter()
    if t_first is None:
        t_first = t_done
    digest = hashlib.sha256(payload).hexdigest()[:12]
    streaming = b"stream" in payload and b"true" in payload.lower()
    return Trace(run_id, t_submit, t_first, t_done, digest, streaming, cold)

if __name__ == "__main__":
    rows = [stream_once(f"r{i}", cold=(i == 0)) for i in range(10)]
    print(json.dumps([r.__dict__ | {
        "wait_s": r.wait_s, "work_s": r.work_s,
        "contamination": r.contamination, "verdict": r.verdict(),
    } for r in rows], indent=2))

export EVAL_URL="http://127.0.0.1:8080/v1/stream"   # local stand-in
export EVAL_BODY='{"prompt":"ping","stream":true}'
python live_probe.py

Do I always get a streaming socket? No. If the free server buffers the whole completion, t_first and t_done collapse, and wait swallows work. Then the splitter cannot save you. That is a limitation, not a footnote. Non-streaming free servers make this method weaker, not stronger. If your loop expects one JSON blob, you are timing a black box. Split nothing. Say so out loud.

How do I decide keep versus discard? I do not worship a magic cutoff. I look at the batch. If contamination sits under about 0.2, I treat wall clock as mostly work. If it wanders from 0.2 to 0.6, I report work time and I print the wait column in the same breath so nobody can hide. If a run clears 0.6, I discard it for ranking. I still log it. Hidden discards are how people fake a calm night.

Is 0.6 science? No. It is a fence I can defend in a code review. You can move the fence. You cannot skip the column. Would you ship a unit test that asserts elapsed < 2.0 on a machine you do not own? Then why is your model eval doing that?

I also refuse to mix cold and warm. The first call after a pause is a different animal. I tag it. I do not average it with the next nine. People love n=10 until you ask which one was the kettle boiling. A free server after idle is a library that has to turn the lights on. That is not a regression in the book.

Where this holds together: identical prompts, streaming responses, a client you control, and a reason to compare two setups on the same night. Where it falls apart: tool-call agents whose "first token" is an opening brace, judges that treat latency as quality, batches that span a traffic cliff, and any protocol that cannot show you the first byte. A free server is a city street. Rush hour is not a model regression. Off-peak is not a breakthrough.

Should you use this if you are publishing a public model card? Only as a hygiene check, never as the headline metric. Should you use this if you are shopping dedicated hardware? No. Buy a stopwatch that is not shared. Should you use this if your eval is a chatbot screenshot and a vibe? Please stop. The method will not fix a missing clock.

I still like free inference. I like it the way I like a public library. Useful. Crowded. A terrible place to time how fast you read. An open source project plus a free server is a gift for iteration. It is a trap for stopwatch science. Hold both thoughts in the same head.

If you already have a slot on that free server, do not start with a blog graph. Start with one prompt, ten streamed calls, and the wait column. If wait moves and work does not, your old dashboard was theatre. That is the only invitation I have.