Write a Receipt for Every Remote Inference

Local-first is a default, not a doctrine. Remote should win one request at a time, on measured evidence.

Most teams pick a side once and never re-measure. That habit costs time in both directions. The last week of DEV posts on AI fatigue and in-browser AI features makes one point twice: inference is everywhere, and waiting for it is the part people resent.

Two constraints decide the route. The first is the wall-clock time your user will tolerate. The second is how many private bytes leave the machine. Cost sits third until somebody reads the invoice.

All three are measurable before you send a token. That is the entire trick. Measure once, encode a small decision function, and keep a receipt for every request that went remote.

Measure both paths on your own machine

Guessing tokens per second is how local-first setups quietly fail. Your laptop is not the laptop in the benchmark post. Put the numbers in the environment, and keep them out of the source tree.

# .envrc — numbers from your hardware, not from a blog post
export LOCAL_MODEL='the model you already run locally'
export LOCAL_TPS=$(python bench_local.py --model "$LOCAL_MODEL" --runs 5)
export REMOTE_TPS=$(python bench_remote.py --runs 5)
export NET_MS=$(python rtt.py --runs 20)
export REMOTE_BASE_URL='your remote endpoint, injected at runtime'

bench_local.py pushes the same twenty prompts through your local runtime and prints median tokens per second. bench_remote.py does the same against the remote endpoint. rtt.py measures round-trip time, which dominates short completions.

None of these scripts needs to be clever. They need to run on your machine, on your network, on a normal weekday. Then turn those numbers into a decision.

# route.py — illustrative example, not executed while writing this article.
import os, json, hashlib, time

LOCAL_TPS  = float(os.environ['LOCAL_TPS'])
REMOTE_TPS = float(os.environ['REMOTE_TPS'])
NET_MS     = float(os.environ['NET_MS'])
BUDGET_S   = 2.0

def local_seconds(tokens):
    return tokens / LOCAL_TPS

def remote_seconds(tokens):
    return NET_MS / 1000.0 + tokens / REMOTE_TPS

def choose_route(segments, budget_s):
    tokens = sum(s['tokens'] for s in segments)
    if any(s['sensitivity'] == 'secret' for s in segments):
        return 'local', 'secret segment present'
    if local_seconds(tokens) <= budget_s:
        return 'local', 'local path fits the budget'
    if remote_seconds(tokens) <= budget_s:
        return 'remote', 'local path misses the budget'
    return 'queue', 'no path meets the budget'

Notice what the function does not do. It does not ask whether you like cloud services. It does not care about brand. It answers one question per request: which path satisfies this budget for this payload?

Receipts, not logs

A log line records that something happened. A receipt records why a decision was made. Write one for every remote call, and never write the payload into it.

def receipt(segments, route, reason, latency_s, tokens_out):
    sent = json.dumps(segments).encode()
    return {
        'ts': time.time(),
        'route': route,
        'reason': reason,
        'prompt_sha256': hashlib.sha256(sent).hexdigest()[:16],
        'bytes_sent': len(sent),
        'tokens_out': tokens_out,
        'latency_s': round(latency_s, 3),
        'budget_s': BUDGET_S,
    }

Append these as JSONL. After one week you can answer the only questions that matter. Which prompts left the machine? Which ones missed their budget? Where did the local path keep winning anyway?

That hash is the important field. It lets you correlate a receipt with a prompt you still hold locally, without copying private text into a shared log. Rotate the mapping if your retention policy demands it.

The table I use

Payload	Budget	Default route
Any secret segment	any	local, always
Public, long context	over 10s	remote
Public, interactive	under 2s	local if it fits
Offline	any	local, queue the rest

The middle rows are where the interesting fights happen. A long public document may be far cheaper to summarize remotely, while an interactive edit will not tolerate the round trip. Users forgive latency on batch work and never forgive it in a chat box.

The secret row is not negotiable. Once a credential or customer record enters the payload, the budget stops mattering. Route it local, or fail closed and tell the user why.

When a free server wins

Remote wins in four honest cases. Cold start, where your machine must load weights first. Long context, where local memory runs out. Batch work, where p95 latency is irrelevant. And hardware you do not own, which is most hardware.

Loops change the calculus too. An agent that calls a model forty times per task feels every millisecond of round-trip time. That workload belongs local, or it belongs on a queue with a human-facing summary at the end.

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers free model access and a free server option. The operator states that 10 million free tokens are available, with the free server acting as the remote tier. Availability terms and quotas can change, so verify them on the project page before you plan around them.

In this workflow it is simply the remote branch of the table. Point REMOTE_BASE_URL at the free tier, run bench_remote.py against it, and let the receipts decide whether it earns a place. Test it the way you test anything else: your budgets, your payloads, your numbers.

Limitations and who should skip this

This approach fails in predictable places. Regulated data usually forbids remote inference no matter how good the latency is. Hard offline environments need a local queue and a conflict policy, not a routing function. Teams without an egress review process should not add a remote tier at all, because receipts are evidence, not permission.

Skip it if you only have one workload. A single fixed task rarely needs a router; hardcode the path and revisit when the second workload arrives. Free tiers are also the wrong foundation for a business. They are a fine place to measure one.

Local-first survives a bad network. A free server survives a weak laptop. Decide per request, write the reason down, and let the measurements change your mind.