Break-Even Batch Size: When a Free Remote Model Server Actually Beats Local

A nightly job on a 16 GB laptop classifies 4,200 support tickets with a local model, and the run takes about two and a half hours at 2.1 seconds per item. The laptop periodically sleeps, the office wifi flaps, and the queue never quite drains by morning. The obvious suggestion is to move the job to a free remote server, and the obvious counter-argument is that the network is exactly the part that keeps failing.

Neither side is right in general. The decision is arithmetic: a remote path pays a fixed cold-start cost once, then runs faster per item, so there is a batch size below which local wins and above which remote wins. This article builds the measurement harness that produces those three numbers and the formula that turns them into a per-job-class rule.

Three numbers, not a philosophy

The whole local-versus-cloud debate collapses into three measured quantities: local (seconds per item on the local path, warm), remote (seconds per item on the remote path, warm), and cold (the one-time cost of the first remote call after an idle period). The break-even batch size is N* = cold / (local - remote), valid only when local > remote. Below N* the fixed cost dominates and the local path finishes first; above it the remote path amortizes the cold start and wins.

Two consequences follow immediately. First, a remote server that is barely faster than the local model has a break-even point in the thousands and is effectively useless for interactive work. Second, the same remote server is excellent for a 4,200-item nightly batch, because the cold start is paid once and amortized over hours of work.

Step 1 — Measure the local per-item cost, warm

The local measurement must discard the first call, because model load and page-cache warm-up will otherwise inflate the number by an order of magnitude. The script below issues 40 sequential calls with identical payload size and reports the median and p90.

# probe_local.py — per-item latency of the local path, warm
import json, statistics, time, urllib.request

ITEMS = [f"ticket-{i:04d}" for i in range(40)]

def run_local(item: str) -> float:
    body = json.dumps({"prompt": f"Label this ticket: {item}", "max_tokens": 16}).encode()
    req = urllib.request.Request(
        "http://127.0.0.1:11434/api/generate",  # your local runtime endpoint
        data=body, headers={"Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=120) as resp:
        resp.read()
    return time.perf_counter() - t0

samples = [run_local(i) for i in ITEMS[1:]]  # drop index 0: warm-up
print(json.dumps({
    "n": len(samples),
    "p50": round(statistics.median(samples), 3),
    "p90": round(statistics.quantiles(samples, n=10)[8], 3),
}))

Run this while the machine is on AC power with the same background load the real job sees, and record the p50 rather than the mean. The p90 matters for the tail, but the p50 is the number that belongs in the break-even formula.

Step 2 — Measure cold start separately from steady state

The remote measurement has to be split in two, because a cold start and a warm call are different physical events. The shell probe below times one call after an idle window, then five consecutive warm calls against the same endpoint.

# probe_remote.sh — cold start once, then five warm calls
URL="$MONKEYCODE_ENDPOINT"; KEY="$MONKEYCODE_KEY"
HEAD=(-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json')

# 1) cold: first call after the server has been idle for at least an hour
curl -sS -o /dev/null -w 'cold_total=%{time_total}\n' "${HEAD[@]}" \
  -d '{"prompt":"warm up","max_tokens":8}' "$URL/chat"

# 2) warm: identical payload size for every call
for i in 1 2 3 4 5; do
  curl -sS -o /dev/null -w "warm_$i=%{time_total}\n" "${HEAD[@]}" \
    -d '{"prompt":"Label this ticket: ticket-0001","max_tokens":16}' "$URL/chat"
done

Repeat the cold measurement on three different days and keep the median, because capacity scheduling on any free tier varies by hour. Never compare a cold remote number against a warm local number: that comparison is the most common way this decision gets made wrongly.

Step 3 — Compute the break-even point and read the table

With three numbers in hand, the calculation is a few lines and produces a table a team can argue about concretely. The example values below are placeholders from a single laptop run and must be replaced with local measurements.

# breakeven.py
cold, local, remote = 6.4, 2.10, 0.85   # seconds; replace with your probes

n_star = cold / (local - remote) if local > remote else float("inf")
print(f"break-even batch size: {n_star:.1f} items")

for n in (1, 5, 10, 20, 50, 200):
    t_local, t_remote = n * local, cold + n * remote
    winner = "local" if t_local < t_remote else "remote"
    print(f"n={n:>4}  local={t_local:7.1f}s  remote={t_remote:7.1f}s  {winner}")

With those inputs the break-even lands near five items, which means interactive single-item work stays local and any batch above a handful should be routed remotely. A slower remote path shifts the break-even upward quickly, so the formula doubles as a rejection test for endpoints that are not worth wiring up at all.

Step 4 — Spool remote-bound work so a dropped network loses nothing

Routing a batch to a remote server only makes sense if the spool survives disconnects, and an append-only JSONL file with a drain lock is usually enough. Each item is written before the call and removed only after a successful response, so a crash repeats work instead of dropping it.

SPOOL="$HOME/.agent/spool/remote.jsonl"; mkdir -p "$(dirname "$SPOOL")"

# enqueue before calling
printf '%s\n' "$ITEM_JSON" >> "$SPOOL"

# drain under a lock, one item at a time
(
  flock -n 9 || { echo 'another drain is running'; exit 0; }
  while IFS= read -r line; do
    post "$line" && sed -i "1{/^$(printf '%s' "$line" | sed 's/[\/&]/\\&/g')$/d}" "$SPOOL"
  done < "$SPOOL"
) 9>"$SPOOL.lock"

This is deliberately dumb, and that is the point: the spool is inspectable with wc -l and tail, which no in-memory queue can match during an incident.

The decision table

Job shape	Secret surface	Network	Recommended path
Single interactive request	Any	Any	Local: below break-even, fixed cost dominates
Nightly batch, thousands of items	Low after redaction	Stable	Remote: cold start amortizes over hours
Burst after a laptop wake	Low	Flaky	Remote with JSONL spool and retry
Prompts containing credentials or raw PII	High	Any	Local only, or redact before any handoff
Machine that cannot hold a local model	Low	Stable	Remote is the only option

Where a free server genuinely earns its place

A free remote server wins in two specific situations, and both are visible in the table rather than asserted. The first is the large batch, where a fixed cold start is amortized over thousands of items and per-item latency actually matters. The second is the machine that cannot host a local model at all, where the alternative is not a faster path but no path.

This is where MonkeyCode's free tier becomes relevant to the method rather than to a slogan. The project offers free model access and a free server option, plus a free token allowance reported as 10 million at the time of writing, which is enough to run the cold-start and warm probes above plus a realistic batch. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Because allowances and capacity change, the current figures should be checked on the project page before a team commits a production job to them.

The honest usage pattern is narrow: use the free server for the batch classes that clear the break-even test, keep secret-bearing work on the host, and re-run the cold probe monthly because free-tier capacity is not a constant.

Limitations and who should not use this

This method assumes the two paths produce comparable output quality, and latency alone is a terrible reason to ship worse labels. A quality gate must be pinned first, with a fixed evaluation set and a pass threshold, before any break-even number is allowed to influence routing. Earlier work in this series on contract tests for free models describes how to hold that gate stable across model changes.

Teams with hard data-residency requirements should not route any batch remotely, regardless of how favorable the arithmetic looks. Teams whose batches sit below the break-even point should not bother, because the added failure modes outweigh a saving measured in seconds. Finally, one laptop's cold-start measurement is not a capacity forecast, and a number gathered at 3 a.m. may not hold at 3 p.m.

A short checklist before switching a job class

1. Pin the quality gate and confirm the remote path passes it on a fixed evaluation set.
2. Measure local p50 over at least 40 warm items on the machine that will run the job.
3. Measure remote cold start on three separate days and keep the median.
4. Compute the break-even size and compare it with the real batch size, not a hypothetical one.
5. Add the JSONL spool and a drain lock before the first remote batch runs unattended.
6. Re-run the cold probe monthly and re-open the decision if the number moves.

A team that runs steps one and two on its own hardware will have a defensible placement rule within an afternoon, and the numbers, rather than the marketing, will decide where each job class lives.