Measure Warmth Before You Measure Speed

A warm remote call usually beats a cold local model. Average latency hides that difference. Measure warmth before you measure speed.

Most local-first comparisons sample steady state. That sample describes a machine that has already paid its startup cost. Your laptop is rarely in that state when you need it.

The lid closed. The kernel reclaimed memory. Another process took the GPU. The next call pays the full load again, and the developer feels it as a hang.

So the interesting number is not tokens per second. It is time to first token, split by warm and cold. Those two numbers can differ by an order of magnitude, and they route differently.

The probe that produces both numbers

The script below measures time to first token against any streaming chat-completions endpoint you control. Point it at a local runtime, then at a remote one. Keep the same prompt length for both.

# probe_warmth.py — time to first chunk, warm and cold
import json, os, statistics, time, urllib.request

ENDPOINT = os.environ["PROBE_ENDPOINT"]   # full chat-completions URL
MODEL    = os.environ["PROBE_MODEL"]
KEY      = os.environ.get("PROBE_KEY", "")

def first_chunk_ms(prompt: str) -> float:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=180) as resp:
        for line in resp:
            if line.startswith(b"data:"):
                return (time.perf_counter() - t0) * 1000
    raise RuntimeError("no stream chunk received")

def sample(label: str, n: int, prompt: str) -> dict:
    xs = [first_chunk_ms(prompt) for _ in range(n)]
    return {"label": label, "n": n,
            "min_ms": round(min(xs)),
            "p50_ms": round(statistics.median(xs)),
            "max_ms": round(max(xs))}

if __name__ == "__main__":
    import sys
    label, n = sys.argv[1], int(sys.argv[2])
    print(json.dumps(sample(label, n, "Name three colors. One word each.")))

Run the cold sample immediately after your local runtime starts. Run the warm sample after three throwaway calls. Append both lines to one log file.

# placeholder commands: substitute your own runtime start/stop
$LOCAL_STOP_CMD && $LOCAL_START_CMD
python probe_warmth.py cold 5 >> warmth.jsonl
python probe_warmth.py warm 20 >> warmth.jsonl

Five cold samples is enough to see the shape. One cold sample is a story, not a measurement.

What the two lines tell you

Look at p50_ms for warm, and min_ms for cold. That cold minimum is your best case after an eviction. Your worst case is the max_ms you will never fully control.

Now compare the cold minimum against the remote warm p50_ms. If cold local is larger, the local path is only defensible for reasons other than speed. Privacy is such a reason. Being on a train is another.

This is where a warm remote tier earns its place. During the cold window, the remote call answers first. The operator describes a free model allowance and a free server option for exactly that kind of low-stakes traffic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free tier as a buffer, not as the foundation. Free capacity is shared, so sample it at the hour you actually work. An idle Tuesday morning tells you nothing about a Monday deadline.

Keep-alive is a budget, not a setting

A warm model is a rental. You pay memory and power to hold it. Keeping it warm forever is how laptops start swapping during unrelated builds.

So convert warmth into a number you can argue about: the idle window. If the next call lands inside the window, keep local. If it lands outside, the model is probably gone anyway.

# pseudocode — not executed; wire your own measurements in
def route(cold_min_ms, remote_p50_ms, secret, offline,
          idle_s, window_s, budget_ms=1500):
    if offline:
        return "local"                    # nothing else exists
    if secret:
        return "local"                    # egress is the wrong cost
    if idle_s > window_s and cold_min_ms > budget_ms:
        return "remote"                   # cold local is the slow path
    if cold_min_ms < remote_p50_ms:
        return "local"
    return "remote"

Set window_s from your own log, not from a blog post. If you never idle longer than the window, the local model is effectively always warm. Then the whole debate shrinks.

Secrets and offline override everything

Some payloads must not leave the machine. For those, warmth is irrelevant and the local path is mandatory. That constraint belongs to a payload class, not to the whole application.

Offline is the same kind of constraint. It is a state you check, not a property you assume. A route that ignores it fails at the worst moment.

Everything else is negotiable. A color-picking prompt, a summary of public documentation, a lint suggestion: none of these justify a two-second stall behind a cold model.

The useful framing is a small table you can defend in review.

Condition	Route	Why
Secret payload	local	no egress allowed
Offline	local	no network exists
Cold, non-secret, bursty	remote	local pays load cost per burst
Warm, non-secret, steady	local	no per-token bill
Large context, non-secret	measure	transfer may cost more than load

Fill the last row with your own log. Bandwidth and payload size decide it, and neither is constant.

Who should not use this

If your traffic is uniform and always warm, this workflow adds nothing. If you cannot restart your local runtime, you cannot get a cold sample. Skip it.

If you handle regulated data, do not route anything to a remote tier to save two seconds. The classification work is worth more than the latency win.

And if your team cannot read the log, the log will not change the routing. Put the numbers in the pull request, next to the decision.

A sequence you can run today

Probe cold and warm on the machine you actually use. Compare against one remote tier. Write the two thresholds into the router as constants with comments.

Then leave it alone for a week. Re-run the probe, because driver updates and model swaps move the cold minimum. Warmth is a property of an environment, not of a model.

If you want a warm remote tier to park the cold window in, the free model access and free server option are worth one probe run before you commit to any route.