The Warm-Up Tax: When Local LLMs Lose to a Hosted Endpoint

Your local model is not slow. It is cold. Those are different problems.

A cold model loads weights into memory. The first request pays for that load. The second request borrows the warm memory. A hosted endpoint pays a network hop every time. It never pays the load. The interesting question is when local actually wins. This article gives you a probe, a decision table, and a routing rule.

The Warm-Up Problem

Open a laptop. Start the local runtime. Send a prompt. Watch the gap between submit and first token. Then send another prompt. The difference between those two timings is the warm-up tax.

Battery, memory pressure, and model size change the tax. A quantized model on a workstation pays less. A big model on a laptop pays more. Most teams never measure it. They assume local equals fast because the marketing material said so.

One Metric Decides the Fight

The warm-up tax only matters if sessions are short. Short sessions die before the model warms. Long sessions amortize the tax across many requests.

Take a simple shape. Warm-up time W. Warm latency L. Hosted latency H. A session with N requests costs W plus N times L locally. The same session costs N times H hosted. Local wins when the first expression is lower.

Teams skip the arithmetic. They argue about privacy and hardware instead. Privacy is a constraint, not a performance argument. Measure first. Argue later.

The math is not the point. The point is that the decision is measurable. If you cannot measure it, you are guessing. Guessing is how support tools get slow.

The Probe

The script below measures both sides. It runs a local command, then a hosted call, and prints the warm-up tax. Replace the local command with your runtime. Replace the hosted URL with your endpoint.

import argparse
import json
import subprocess
import time

import httpx


def local(parts):
    start = time.perf_counter()
    proc = subprocess.Popen(
        parts,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.DEVNULL,
        text=True,
    )
    first = None
    for line in proc.stdout:
        if first is None and line.strip():
            first = time.perf_counter()
            break
    proc.terminate()
    return {"first_token_s": round((first or time.perf_counter()) - start, 2)}


def hosted(url, token):
    start = time.perf_counter()
    r = httpx.post(
        url,
        headers={"Authorization": f"Bearer {token}"},
        json={"prompt": "Say hi.", "max_tokens": 16},
        timeout=60,
    )
    r.raise_for_status()
    return {"round_trip_s": round(time.perf_counter() - start, 2)}


parser = argparse.ArgumentParser()
parser.add_argument("--mode", choices=["local", "hosted"], required=True)
parser.add_argument("--cmd", nargs="+", default=["llama-cli", "-p", "Say hi."])
parser.add_argument("--url")
parser.add_argument("--token")
args = parser.parse_args()

if args.mode == "local":
    print(json.dumps(local(args.cmd)))
else:
    print(json.dumps(hosted(args.url, args.token)))

Run it twice for local. First run cold. Second run warm. The difference is the tax.

python probe.py --mode local --cmd llama-cli -p "Say hi."
python probe.py --mode local --cmd llama-cli -p "Say hi."
python probe.py --mode hosted --url "$URL" --token "$TOKEN"

The Decision Table

With three numbers, routing becomes mechanical. Warm-up W. Warm latency L. Hosted latency H. The table interprets them.

Condition	Meaning	Route
W is small	Laptop holds the model easily	Local
L is near H	No latency benefit locally	Hosted
H under 1s, W over 10s	Cold sessions pay a heavy tax	Hosted for short bursts
Long batches, W paid once	Tax amortizes	Local
No GPU, big model	Painful W	Hosted
Data must stay inside	Constraint beats math	Local

Read the row for your workload. Not for your ambition.

Where Free Models Change the Equation

The hosted side used to cost money. That changed the arithmetic for hobby projects. A paid API made short sessions feel wasteful, even when hosted clearly won the timing test.

MonkeyCode offers free models and a free server option. Free model access removes the bill from the hosted side of the comparison. The free server option can host the probe and the small routing logic without a separate cloud charge. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not claiming the free models match paid ones on every benchmark. I am claiming the hosted side no longer loses on cost. Cost is gone. Latency is the only fight left.

The routing logic itself is tiny.

def route(W, L, H, n):
    local_cost = W + n * L
    hosted_cost = n * H
    return "local" if local_cost < hosted_cost else "hosted"

Three lines. That is the whole router.

The Routing Rule

Apply the rule in four steps.

1. Measure W with the probe on the real machine.
2. Measure L and H on the same prompt.
3. Pick the session length that matches your tool.
4. Route short bursts hosted. Route long batches local.

A support widget gets three requests per session. Route it hosted. A nightly summarizer runs for hours. Keep it local.

Run the probe five times. Use the median. Cold-start values vary with system state. A single run will mislead you.

Limits

The probe measures one machine. Regenerate every number for the target environment. Free endpoints can get slower under load. Local runtimes change with quantization and GPU memory.

Do not use a hosted free endpoint for regulated data. Do not assume the free model behaves like a paid model. The routing rule optimizes latency and cost. It says nothing about output quality.

These numbers are not portable. Do not copy someone else's measurements into your design doc.

The Takeaway

Local is not automatically fast. Hosted is not automatically slow. The warm-up tax decides the real answer.

Measure it with the probe. Read the table. Route accordingly. Then your demo starts with an answer, not a spinner.

If your sessions are short and your model is big, hosted is not a compromise. It is the measured answer.