Three Numbers Beat the Local-First Debate

Most teams choose local or cloud by taste. Three measurements settle it faster than any design debate.

The trap is treating "local-first" as an architecture. It is a budget class. Every call site has a latency ceiling, a secret boundary, and an offline requirement. Make those three explicit and the argument mostly evaporates.

Run three probes per call site. Print one verdict. Then move on.

Probe one: the latency ceiling

Local inference has no network hop. That is not the same as fast. Token generation dominates wall clock, and p95 is where users feel the pain.

Measure with your real prompt length. "Hello" is not your workload. Warm up once, then sample at least thirty runs. Percentiles need volume; five samples tell you nothing useful.

The harness below is a proposal. I have not run it against your stack, so treat it as a starting shape, not a result.

# triage.py — proposal, unexecuted here.
import json, time, urllib.request

PROMPT = "Ticket: my CSV export dies at 90%. Name the likely cause in one sentence."

def timed(url, payload, headers=None, timeout=60):
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", **(headers or {})},
    )
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        resp.read()
    return (time.perf_counter() - start) * 1000.0

The percentile helper stays boring on purpose. Small code is easier to trust.

def pct(samples, q):
    ordered = sorted(samples)
    idx = max(0, min(len(ordered) - 1, int(round(q * (len(ordered) - 1)))))
    return ordered[idx]

def latency_probe(fn, runs=30):
    fn()  # first call pays cold-start and model load
    samples = [fn() for _ in range(runs)]
    return {q: pct(samples, q) for q in (0.50, 0.95, 0.99)}

Ollama defaults to port 11434. llama.cpp's server defaults to 8080. Point the probe at whatever you actually run, then compare p95 against the ceiling you wrote down in the ticket.

A ceiling you never wrote down is not a requirement. It is a feeling.

Probe two: the secret boundary

Ask one question per call site. Can this exact payload leave the machine?

Tickets carry session tokens. Logs carry cookies. Prompts carry customer identifiers. Nobody intends to ship those to a third party, and yet they show up in request bodies constantly.

SECRET_HINTS = ("token", "password", "secret", "authorization", "cookie", "session")

def secret_hits(text):
    low = text.lower()
    return sorted({h for h in SECRET_HINTS if h in low})

This is a smoke test, not a data-loss-prevention system. It catches the obvious cases and misses encoded ones. Run it anyway, because the obvious cases are the ones that reach production.

A hit does not automatically block the server path. It demands redaction first. Strip the header, hash the identifier, keep the shape. Then re-measure, because your prompt just changed size.

Probe three: the offline trial

Cut the socket, not the Wi-Fi. The goal is a deterministic failure, not a flaky one.

import socket

def with_sockets_dead(fn):
    original = socket.socket
    def dead(*args, **kwargs):
        raise OSError("network disabled for offline trial")
    socket.socket = dead
    try:
        return fn()
    finally:
        socket.socket = original

Run your call site inside that guard. If it raises and has no fallback branch, it is cloud-dependent. That is allowed. It becomes a problem only when nobody wrote it down.

Offline capability is a product decision. It is also a cost. Local-only paths need model storage, updates, and a plan for slow machines.

The verdict function

Three booleans and two numbers produce one line of text. That line is the whole deliverable.

def verdict(p95_ms, budget_ms, hits, offline_ok):
    if not offline_ok:
        return "local-only: offline requirement"
    if hits and p95_ms <= budget_ms:
        return "local-first: secret stays home"
    if hits:
        return "redact, then server"
    if p95_ms <= budget_ms:
        return "local-first with server overflow"
    return "server primary"

Offline needed	Secrets in payload	Local p95 vs budget	Verdict
Yes	Either	Either	Local only
No	Yes	Within	Local first
No	Yes	Over	Redact, then server
No	No	Within	Local first, server overflow
No	No	Over	Server primary

The table is deliberately blunt. Real systems have exceptions, and every exception should name the number it breaks.

Where a free server earns its row

Three rows end at "server". That happens for long prompts, heavy models, and machines without a usable GPU.

It also happens during evaluation. You cannot judge a hosted model without calling one. Spinning up paid capacity to answer "is this good enough" is backwards.

That is the honest case for a free tier. MonkeyCode provides free model access and a free server option, as supplied by the operator. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it to exercise the server row of your table. Compare its p95 against the same prompt you used locally. Keep the measurement, throw away the enthusiasm.

Free terms move. Check the current quota and limits yourself before you build a plan on top of them. I am not quoting numbers here, because numbers on a free tier age badly.

Treat a free tier as a measuring stick. Do not treat it as a capacity plan.

Limitations worth naming

Local latency measured on your laptop does not transfer to a shared host. Cold start can dominate any single-shot call, so probe warm and cold separately. Thirty samples give a rough p95, not a service-level objective. The secret scan is keyword matching and will miss encoded payloads. Wall clock says nothing about answer quality, which needs its own eval set.

There is also a quiet failure mode. A local model that answers in 400 ms with the wrong answer is worse than a server call at 900 ms that is right. Measure quality before you optimize the number.

Who should skip this

Teams under strict data-residency rules should not send payloads to any third party, free or otherwise. Startups promising hard latency SLOs without owning capacity should not lean on a free server either. And teams already sitting on idle GPU hardware should just measure it and skip the debate entirely.

The workflow is for the middle. People with a real prompt, a real ceiling, and no strong prior.

Write the three numbers down. The vendor question gets much easier after that.