DEV Community: Riley Wu

Measure Warmth Before You Measure Speed

Riley Wu — Tue, 27 Oct 2026 13:38:42 +0000

A warm remote call usually beats a cold local model. Average latency hides that difference. Measure warmth before you measure speed.

Most local-first comparisons sample steady state. That sample describes a machine that has already paid its startup cost. Your laptop is rarely in that state when you need it.

The lid closed. The kernel reclaimed memory. Another process took the GPU. The next call pays the full load again, and the developer feels it as a hang.

So the interesting number is not tokens per second. It is time to first token, split by warm and cold. Those two numbers can differ by an order of magnitude, and they route differently.

The probe that produces both numbers

The script below measures time to first token against any streaming chat-completions endpoint you control. Point it at a local runtime, then at a remote one. Keep the same prompt length for both.

# probe_warmth.py — time to first chunk, warm and cold
import json, os, statistics, time, urllib.request

ENDPOINT = os.environ["PROBE_ENDPOINT"]   # full chat-completions URL
MODEL    = os.environ["PROBE_MODEL"]
KEY      = os.environ.get("PROBE_KEY", "")

def first_chunk_ms(prompt: str) -> float:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=180) as resp:
        for line in resp:
            if line.startswith(b"data:"):
                return (time.perf_counter() - t0) * 1000
    raise RuntimeError("no stream chunk received")

def sample(label: str, n: int, prompt: str) -> dict:
    xs = [first_chunk_ms(prompt) for _ in range(n)]
    return {"label": label, "n": n,
            "min_ms": round(min(xs)),
            "p50_ms": round(statistics.median(xs)),
            "max_ms": round(max(xs))}

if __name__ == "__main__":
    import sys
    label, n = sys.argv[1], int(sys.argv[2])
    print(json.dumps(sample(label, n, "Name three colors. One word each.")))

Run the cold sample immediately after your local runtime starts. Run the warm sample after three throwaway calls. Append both lines to one log file.

# placeholder commands: substitute your own runtime start/stop
$LOCAL_STOP_CMD && $LOCAL_START_CMD
python probe_warmth.py cold 5 >> warmth.jsonl
python probe_warmth.py warm 20 >> warmth.jsonl

Five cold samples is enough to see the shape. One cold sample is a story, not a measurement.

What the two lines tell you

Look at p50_ms for warm, and min_ms for cold. That cold minimum is your best case after an eviction. Your worst case is the max_ms you will never fully control.

Now compare the cold minimum against the remote warm p50_ms. If cold local is larger, the local path is only defensible for reasons other than speed. Privacy is such a reason. Being on a train is another.

This is where a warm remote tier earns its place. During the cold window, the remote call answers first. The operator describes a free model allowance and a free server option for exactly that kind of low-stakes traffic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free tier as a buffer, not as the foundation. Free capacity is shared, so sample it at the hour you actually work. An idle Tuesday morning tells you nothing about a Monday deadline.

Keep-alive is a budget, not a setting

A warm model is a rental. You pay memory and power to hold it. Keeping it warm forever is how laptops start swapping during unrelated builds.

So convert warmth into a number you can argue about: the idle window. If the next call lands inside the window, keep local. If it lands outside, the model is probably gone anyway.

# pseudocode — not executed; wire your own measurements in
def route(cold_min_ms, remote_p50_ms, secret, offline,
          idle_s, window_s, budget_ms=1500):
    if offline:
        return "local"                    # nothing else exists
    if secret:
        return "local"                    # egress is the wrong cost
    if idle_s > window_s and cold_min_ms > budget_ms:
        return "remote"                   # cold local is the slow path
    if cold_min_ms < remote_p50_ms:
        return "local"
    return "remote"

Set window_s from your own log, not from a blog post. If you never idle longer than the window, the local model is effectively always warm. Then the whole debate shrinks.

Secrets and offline override everything

Some payloads must not leave the machine. For those, warmth is irrelevant and the local path is mandatory. That constraint belongs to a payload class, not to the whole application.

Offline is the same kind of constraint. It is a state you check, not a property you assume. A route that ignores it fails at the worst moment.

Everything else is negotiable. A color-picking prompt, a summary of public documentation, a lint suggestion: none of these justify a two-second stall behind a cold model.

The useful framing is a small table you can defend in review.

Condition	Route	Why
Secret payload	local	no egress allowed
Offline	local	no network exists
Cold, non-secret, bursty	remote	local pays load cost per burst
Warm, non-secret, steady	local	no per-token bill
Large context, non-secret	measure	transfer may cost more than load

Fill the last row with your own log. Bandwidth and payload size decide it, and neither is constant.

Who should not use this

If your traffic is uniform and always warm, this workflow adds nothing. If you cannot restart your local runtime, you cannot get a cold sample. Skip it.

If you handle regulated data, do not route anything to a remote tier to save two seconds. The classification work is worth more than the latency win.

And if your team cannot read the log, the log will not change the routing. Put the numbers in the pull request, next to the decision.

A sequence you can run today

Probe cold and warm on the machine you actually use. Compare against one remote tier. Write the two thresholds into the router as constants with comments.

Then leave it alone for a week. Re-run the probe, because driver updates and model swaps move the cold minimum. Warmth is a property of an environment, not of a model.

If you want a warm remote tier to park the cold window in, the free model access and free server option are worth one probe run before you commit to any route.

Let the Secret Decide Where the Prompt Runs

Riley Wu — Mon, 26 Oct 2026 12:14:59 +0000

Let the Secret Decide Where the Prompt Runs

A pasted config file should never reach a remote model. Not because remote is slow. Because the payload carries keys.

Latency arguments come later. Secret class comes first, and it is a policy call. Decide on what the text contains, then on how fast it must return.

The rule nobody enforces

You paste forty lines of .env into a chat box. The answer is good. The request is also gone.

Most teams handle this with a guideline. Do not paste secrets. Guidelines without a gate decay in about a week.

So build the gate. Three checks, one decision, fail closed. The gate runs before the client library, not inside your head.

The gate is three checks

Check one is pattern matching for known credential shapes. Check two is Shannon entropy over long tokens. Check three is the budget: offline, or local p50 above your limit.

#!/usr/bin/env python3
# egress_gate.py - decide local vs remote before a prompt leaves the machine.
import argparse, json, math, re, sys
from dataclasses import dataclass, asdict

PATTERNS = {
    "aws_key": re.compile(r"AKIA[0-9A-Z]{16}"),
    "github_token": re.compile(r"gh[pousr]_[A-Za-z0-9]{36,}"),
    "private_key": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    "slack_token": re.compile(r"xox[baprs]-[A-Za-z0-9-]{10,}"),
    "db_url_with_password": re.compile(
        r"(?i)\b(postgres|mysql|redis|mongodb)://[^\s:@/]+:[^\s:@/]+@"
    ),
}

def shannon(token: str) -> float:
    counts = {}
    for ch in token:
        counts[ch] = counts.get(ch, 0) + 1
    n = len(token)
    return -sum((c / n) * math.log2(c / n) for c in counts.values())

def entropy_hits(text: str, min_len: int = 20, threshold: float = 3.6):
    hits = []
    for tok in re.split(r'''[\s"'`,;(){}\[\]<>]+''', text):
        if len(tok) >= min_len and shannon(tok) >= threshold:
            hits.append(tok)
    return hits

@dataclass
class Decision:
    route: str      # "local" or "remote"
    reason: str
    findings: list

def gate(payload: str, offline: bool, local_p50_ms, budget_ms: float) -> Decision:
    findings = [name for name, pat in PATTERNS.items() if pat.search(payload)]
    hits = entropy_hits(payload)
    if hits:
        findings.append(f"entropy_hits={len(hits)}")
    if findings:
        return Decision("local", "secret_class", findings)
    if offline:
        return Decision("local", "offline")
    if local_p50_ms is None or local_p50_ms > budget_ms:
        return Decision("remote", "over_budget")
    return Decision("local", "fits_budget")

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("payload")
    ap.add_argument("--offline", action="store_true")
    ap.add_argument("--local-p50-ms", type=float, default=None)
    ap.add_argument("--budget-ms", type=float, default=400.0)
    a = ap.parse_args()
    text = open(a.payload, encoding="utf-8").read()
    d = gate(text, a.offline, a.local_p50_ms, a.budget_ms)
    print(json.dumps(asdict(d)))
    sys.exit(0 if d.route == "local" else 10)

The exit code is the policy. Zero means keep it on the machine. Ten means egress is permitted. Wire that number into a pre-commit hook or a client wrapper, and the guideline becomes code.

Entropy is the noisy check. Hashes, UUIDs, and base64 assets all look random. That noise is the price of catching keys you never wrote a regex for.

Measure the false positive rate first

Do not trust the scanner on faith. Run it across a real repository and count what it flags.

mkdir -p fixtures
printf 'AKIAABCDEFGHIJKLMNOP' > fixtures/aws.txt
python egress_gate.py fixtures/aws.txt; echo "exit=$?"

# scan every source file, one JSON line per file
find ~/src/myrepo -type f -name '*.js' -o -name '*.py' -o -name '*.tf' \
  | while read -r f; do
      python egress_gate.py "$f" | sed "s|^{|{\"file\":\"$f\",|"
    done > flagged.jsonl

python - <<'PY'
import json, collections
rows = [json.loads(l) for l in open("flagged.jsonl")]
print(collections.Counter(r["reason"] for r in rows).most_common())
PY

Sort the flagged files by entropy hit count and read the top fifty yourself. Some will be lockfiles and minified bundles. Those belong on a path allowlist, not in the gate's blind spot.

Set your own tolerance. I will not quote a false positive percentage I have not measured on your codebase. The number is repository-specific and moves with every dependency you add.

Latency uses your machine, not mine

A budget check needs a real baseline. Measure warm p50 over twenty runs, then measure cold start separately.

for i in $(seq 1 20); do
  /usr/bin/time -f '%e' python -c "
import your_client, pathlib
your_client.generate(pathlib.Path('fixtures/clean.txt').read_text())" 2>&1 | tail -1
done | sort -n | awk '{a[NR]=$1} END{print "p50_seconds", a[int(NR/2)+1]}'

Warm and cold differ by seconds, not milliseconds, on a laptop. Cold start belongs in the budget or it will surprise you on the first request of the morning.

Pass the measured p50 into --local-p50-ms. Now the gate compares your hardware to your deadline, not a vendor chart.

The decision table

Payload class	Local p50 vs budget	Link	Route
Credential or high entropy	any	any	local
Credential or high entropy	none available	any	local
Clean text	over budget	online	remote
Clean text	within budget	online	local
Clean text	any	offline	local

Read the table top to bottom. The first matching row wins. Secrets never fall through to a performance argument.

When a free server wins

The gate answers a policy question. It does not answer a hardware one. When the gate returns ten, you still need somewhere to send the request.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open source project that offers free model access and a free server option. The operator states the free tier includes ten million tokens. Treat that as a starting budget, not a permanent guarantee, and confirm current limits on the project page before you depend on them.

That fits one narrow case well. A solo developer, a clean payload, a tight deadline, and no GPU on the desk. The free server removes the hardware question for the first day of experiments.

It fits poorly when the payload is dirty. A free tier does not make an unsafe request safe. Run the gate before you spend a token.

Where this breaks

Encoded secrets slip through. A base64 blob of a private key has high entropy sometimes, and sometimes not. Decode before you scan if your payloads carry encoded files.

Entropy flags minified JavaScript and lockfiles. Expect allowlist maintenance, and keep the allowlist in review.

The gate is a tripwire, not a DLP system. It catches shapes you thought about. It will miss a password pasted as plain English.

Free tokens are finite by construction. A retry loop with no cap burns a budget in an afternoon. Log the decision JSON next to your spend.

Who should not use this

Regulated data should not take this route at all. If your payload needs an audit trail, a regex gate is not the control you are looking for.

Production traffic needs a policy engine, not a script. This gate is for interactive work on a developer machine, where the risk is a paste, not a pipeline.

Teams with a shared secret store already solved the problem upstream. Gate the store, and let this script retire.

Run it on your repo first

Start with the scan step, not the model call. Your repository is the honest test set, and the flagged list is your first deliverable.

Keep the exit code in version control. A gate that lives in a wiki is a suggestion. A gate that returns ten is a route.

If the gate allows egress and your laptop p50 misses the deadline, the free server option is the shortest path to a first measurement. Measure, then decide what stays local.

Sort the Payload Before the Server

Riley Wu — Tue, 20 Oct 2026 10:17:50 +0000

Sort the Payload Before the Server

Sensitivity decides the host. Latency only decides the runner-up.

Most local-first arguments start with a stopwatch. That is the wrong first instrument. One leaked API key costs more than a year of slow requests.

Invert the order. Classify every field in the payload first. Then measure latency, then choose a host.

Three classes and one rule

Public data can go anywhere. Derived data needs a contract. Secrets never leave the machine.

Define the classes as integers, so comparisons stay cheap and testable.

// classify.mjs
export const CLASS = { PUBLIC: 0, DERIVED: 1, SECRET: 2 };

const RULES = [
  [/^(api[_-]?key|access[_-]?token|secret|password|authorization|cookie)$/i, CLASS.SECRET],
  [/^(email|phone|ssn|address|card|iban|passport)$/i, CLASS.SECRET],
  [/^(user_?id|account_?id|tenant|session|ip)$/i, CLASS.DERIVED],
];

export function classifyField(name) {
  for (const [re, cls] of RULES) if (re.test(name)) return cls;
  return CLASS.PUBLIC;
}

export function maxClass(value) {
  if (value === null || typeof value !== 'object') return CLASS.PUBLIC;
  let worst = CLASS.PUBLIC;
  const entries = Array.isArray(value) ? value.entries() : Object.entries(value);
  for (const [key, child] of entries) {
    const own = Array.isArray(value) ? CLASS.PUBLIC : classifyField(key);
    const deep = maxClass(child);
    worst = Math.max(worst, own, deep);
  }
  return worst;
}

Name matching is a heuristic, not a proof. It catches the boring majority of leaks. Keep a manual override list for the rest.

The routing rule is one line of policy. Anything above the outbound ceiling runs locally. That ceiling belongs in config, not in a reviewer's memory.

Canary values make leaks visible

A passing test that never sees a secret proves nothing. Plant a unique string in every secret field instead. Then grep every outbound body for it.

// egress.test.mjs
import test from 'node:test';
import assert from 'node:assert/strict';
import { CLASS, maxClass } from './classify.mjs';

const CANARY = 'CANARY-6f2a9c-LEAK';
const outbound = [];

const transport = async (_url, init = {}) => {
  outbound.push(String(init.body ?? ''));
  return new Response('{"ok":true}', { headers: { 'content-type': 'application/json' } });
};

async function route(payload, ceiling = CLASS.DERIVED) {
  const body = JSON.stringify(payload);
  if (maxClass(payload) > ceiling) return { where: 'local', text: 'local-stub' };
  const res = await transport('https://cloud.invalid/v1/chat', { method: 'POST', body });
  return { where: 'cloud', text: await res.text() };
}

test('secret payload stays local', async () => {
  outbound.length = 0;
  const result = await route({ api_key: CANARY, prompt: 'summarize this ticket' });
  assert.equal(result.where, 'local');
  assert.equal(outbound.length, 0);
});

test('clean payload may leave', async () => {
  outbound.length = 0;
  const result = await route({ prompt: 'rewrite this changelog entry' });
  assert.equal(result.where, 'cloud');
  assert.equal(outbound.length, 1);
});

test('no canary ever crosses the wire', () => {
  assert.ok(outbound.every((body) => !body.includes(CANARY)));
});

Run it with plain Node.

node --test egress.test.mjs

Wire the same canary into one real payload in staging. Fill a test tenant with the sentinel string. Then read proxy logs, not application logs. Proxies see what the app forgot it sent.

When the third test fails, you found a real leak for one afternoon of work. That is the whole value of the exercise.

Measure second, not first

Once classification is done, timing becomes simple. Send ten public payloads to both hosts. Record p50 and p95, never the mean.

const times = [];
for (const p of payloads) {
  const t0 = performance.now();
  await route(p);
  times.push(performance.now() - t0);
}
times.sort((a, b) => a - b);
const q = (x) => times[Math.min(times.length - 1, Math.floor(x * times.length))];
console.log({ p50: q(0.5).toFixed(0), p95: q(0.95).toFixed(0) });

Ten samples show a shape, not a verdict. Cold starts and queue depth dominate tiny runs. Warm up once, keep the payloads fixed, and repeat the loop.

This ordering kills most local-first arguments early. Latency rarely changes the host for secret data. It only settles public data.

The decision table

Payload class	Offline needed	Runs on	Reason
PUBLIC	no	free or paid cloud	cheapest per token, no containment cost
PUBLIC	yes	small local model	availability beats quality here
DERIVED	no	cloud, redacted or under contract	policy review, not a firewall
DERIVED	yes	local	a contract cannot cover an outage
SECRET	any	local only	no endpoint qualifies

Read the table top to bottom. The rows never argue with each other.

Offline is a property of the session, not of the model. A local model that fetches a remote tokenizer is not offline. Test that path with the network unplugged.

Where a free server earns a slot

The cloud column needs a sandbox before it needs a contract. MonkeyCode offers free model access and a free server option. The operator currently advertises a 10 million token allotment; quotas and terms change, so read the live page before sizing any work.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it for the PUBLIC row and the redacted half of DERIVED. Point the transport stub at it and rerun the canary tests there. Free capacity is good for proving a route works. It is not a reason to relax the ceiling.

Never send a SECRET field to a free endpoint. No free tier changes that rule.

The free server wins when the payload is already public. It also wins when your team has no GPU and needs a second data point. It loses when the payload carries credentials, or when you need a throughput guarantee in writing.

Limitations and who should skip this

Field-name classification is shallow. Renamed keys slip through. Pair it with an egress allowlist at the proxy.

The canary test only covers paths you wrote yourself. Third-party SDKs can send data you never see. Audit their network calls on a separate track.

Skip this approach if your data cannot leave a VPC at all. A policy boundary beats a routing trick every time. Skip it too if you need certified isolation; check the operator's terms and your own compliance rules first.

Start with the classifier. It costs one file and one afternoon. Run the canary before you benchmark anything.

If you want to exercise the cloud column without a billing conversation, the free tier is a reasonable sandbox for public payloads.

Three Numbers Beat the Local-First Debate

Riley Wu — Mon, 19 Oct 2026 08:59:40 +0000

Most teams choose local or cloud by taste. Three measurements settle it faster than any design debate.

The trap is treating "local-first" as an architecture. It is a budget class. Every call site has a latency ceiling, a secret boundary, and an offline requirement. Make those three explicit and the argument mostly evaporates.

Run three probes per call site. Print one verdict. Then move on.

Probe one: the latency ceiling

Local inference has no network hop. That is not the same as fast. Token generation dominates wall clock, and p95 is where users feel the pain.

Measure with your real prompt length. "Hello" is not your workload. Warm up once, then sample at least thirty runs. Percentiles need volume; five samples tell you nothing useful.

The harness below is a proposal. I have not run it against your stack, so treat it as a starting shape, not a result.

# triage.py — proposal, unexecuted here.
import json, time, urllib.request

PROMPT = "Ticket: my CSV export dies at 90%. Name the likely cause in one sentence."

def timed(url, payload, headers=None, timeout=60):
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", **(headers or {})},
    )
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        resp.read()
    return (time.perf_counter() - start) * 1000.0

The percentile helper stays boring on purpose. Small code is easier to trust.

def pct(samples, q):
    ordered = sorted(samples)
    idx = max(0, min(len(ordered) - 1, int(round(q * (len(ordered) - 1)))))
    return ordered[idx]

def latency_probe(fn, runs=30):
    fn()  # first call pays cold-start and model load
    samples = [fn() for _ in range(runs)]
    return {q: pct(samples, q) for q in (0.50, 0.95, 0.99)}

Ollama defaults to port 11434. llama.cpp's server defaults to 8080. Point the probe at whatever you actually run, then compare p95 against the ceiling you wrote down in the ticket.

A ceiling you never wrote down is not a requirement. It is a feeling.

Probe two: the secret boundary

Ask one question per call site. Can this exact payload leave the machine?

Tickets carry session tokens. Logs carry cookies. Prompts carry customer identifiers. Nobody intends to ship those to a third party, and yet they show up in request bodies constantly.

SECRET_HINTS = ("token", "password", "secret", "authorization", "cookie", "session")

def secret_hits(text):
    low = text.lower()
    return sorted({h for h in SECRET_HINTS if h in low})

This is a smoke test, not a data-loss-prevention system. It catches the obvious cases and misses encoded ones. Run it anyway, because the obvious cases are the ones that reach production.

A hit does not automatically block the server path. It demands redaction first. Strip the header, hash the identifier, keep the shape. Then re-measure, because your prompt just changed size.

Probe three: the offline trial

Cut the socket, not the Wi-Fi. The goal is a deterministic failure, not a flaky one.

import socket

def with_sockets_dead(fn):
    original = socket.socket
    def dead(*args, **kwargs):
        raise OSError("network disabled for offline trial")
    socket.socket = dead
    try:
        return fn()
    finally:
        socket.socket = original

Run your call site inside that guard. If it raises and has no fallback branch, it is cloud-dependent. That is allowed. It becomes a problem only when nobody wrote it down.

Offline capability is a product decision. It is also a cost. Local-only paths need model storage, updates, and a plan for slow machines.

The verdict function

Three booleans and two numbers produce one line of text. That line is the whole deliverable.

def verdict(p95_ms, budget_ms, hits, offline_ok):
    if not offline_ok:
        return "local-only: offline requirement"
    if hits and p95_ms <= budget_ms:
        return "local-first: secret stays home"
    if hits:
        return "redact, then server"
    if p95_ms <= budget_ms:
        return "local-first with server overflow"
    return "server primary"

Offline needed	Secrets in payload	Local p95 vs budget	Verdict
Yes	Either	Either	Local only
No	Yes	Within	Local first
No	Yes	Over	Redact, then server
No	No	Within	Local first, server overflow
No	No	Over	Server primary

The table is deliberately blunt. Real systems have exceptions, and every exception should name the number it breaks.

Where a free server earns its row

Three rows end at "server". That happens for long prompts, heavy models, and machines without a usable GPU.

It also happens during evaluation. You cannot judge a hosted model without calling one. Spinning up paid capacity to answer "is this good enough" is backwards.

That is the honest case for a free tier. MonkeyCode provides free model access and a free server option, as supplied by the operator. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it to exercise the server row of your table. Compare its p95 against the same prompt you used locally. Keep the measurement, throw away the enthusiasm.

Free terms move. Check the current quota and limits yourself before you build a plan on top of them. I am not quoting numbers here, because numbers on a free tier age badly.

Treat a free tier as a measuring stick. Do not treat it as a capacity plan.

Limitations worth naming

Local latency measured on your laptop does not transfer to a shared host. Cold start can dominate any single-shot call, so probe warm and cold separately. Thirty samples give a rough p95, not a service-level objective. The secret scan is keyword matching and will miss encoded payloads. Wall clock says nothing about answer quality, which needs its own eval set.

There is also a quiet failure mode. A local model that answers in 400 ms with the wrong answer is worse than a server call at 900 ms that is right. Measure quality before you optimize the number.

Who should skip this

Teams under strict data-residency rules should not send payloads to any third party, free or otherwise. Startups promising hard latency SLOs without owning capacity should not lean on a free server either. And teams already sitting on idle GPU hardware should just measure it and skip the debate entirely.

The workflow is for the middle. People with a real prompt, a real ceiling, and no strong prior.

Write the three numbers down. The vendor question gets much easier after that.

Find the Concurrency Where Local Inference Stops Winning

Riley Wu — Sun, 18 Oct 2026 07:35:49 +0000

Local inference wins on latency, but only below a concurrency threshold. Above that line, one accelerator becomes a queue, and each new request pays for the ones ahead of it. So stop arguing about which side is "faster" and measure where the two curves cross.

That is the entire method: sweep concurrency, watch p95, find the swap point. Everything below is the harness, the reading rule, and the cases where the cheap remote endpoint genuinely wins.

The claim that gets repeated

Most local benchmarks use exactly one request. The model answers in a few hundred milliseconds and the team declares local faster. Then ten developers share the endpoint, p95 lands in the seconds, and the same team declares local broken.

Both readings are correct at different loads. A single accelerator serves requests roughly serially once it saturates. A hosted fleet spreads them across machines. The number that matters is not latency, it is the concurrency where the two lines exchange places.

The analogy is a supermarket. One open checkout lane is the fastest option when you are the only shopper in the store. It is the worst option at noon.

What to measure, and what to throw away

Four numbers per concurrency level, nothing else: p50 latency, p95 latency, error rate, and completion tokens per second. Pin the prompt, pin max_tokens, set temperature to zero. Warm the model once and discard that call.

Exclude cold start on purpose. Cold start is a separate problem with its own fix. Mixing it into the sweep hides the queueing effect you are trying to observe.

Also count latency where the request actually starts. If your harness queues tasks in a thread pool, the clock must begin inside the worker, not when the task is submitted. Otherwise you measure your own scheduler and learn nothing about the endpoint.

The harness

Labelled clearly: this is an example harness. I have not run it against your machines, and no number in this article comes from it. Your sweep produces your data, which is the point.

# sweep.py - concurrency sweep for any OpenAI-compatible endpoint
import csv, os, sys, time
from concurrent.futures import ThreadPoolExecutor
import requests

BASE   = os.environ["LLM_BASE"].rstrip("/")
MODEL  = os.environ["LLM_MODEL"]
KEY    = os.environ.get("LLM_KEY", "local")
LEVELS = [1, 2, 4, 8, 16]
REQS_PER_LEVEL = 24
PROMPT = "Return the single word ok."


def call(_):
    body = {
        "model": MODEL,
        "max_tokens": 32,
        "temperature": 0,
        "messages": [{"role": "user", "content": PROMPT}],
    }
    t0 = time.perf_counter()
    try:
        r = requests.post(
            f"{BASE}/chat/completions",
            json=body,
            headers={"Authorization": f"Bearer {KEY}"},
            timeout=120,
        )
        ok = r.status_code == 200
        toks = r.json().get("usage", {}).get("completion_tokens", 0) if ok else 0
        return (time.perf_counter() - t0) * 1000, ok, toks
    except Exception:
        return (time.perf_counter() - t0) * 1000, False, 0


def pct(xs, p):
    xs = sorted(xs)
    k = max(0, min(len(xs) - 1, int(round(p / 100 * (len(xs) - 1)))))
    return xs[k]


def main():
    call(0)  # warm-up, discarded
    w = csv.writer(sys.stdout)
    w.writerow(["concurrency", "p50_ms", "p95_ms", "err_rate", "tok_per_s"])
    for n in LEVELS:
        t0 = time.perf_counter()
        with ThreadPoolExecutor(max_workers=n) as pool:
            rows = list(pool.map(call, range(REQS_PER_LEVEL)))
        wall = time.perf_counter() - t0
        lat = [r[0] for r in rows]
        err = sum(1 for r in rows if not r[1]) / len(rows)
        tok = sum(r[2] for r in rows) / wall
        w.writerow([n, round(pct(lat, 50), 1), round(pct(lat, 95), 1),
                    round(err, 3), round(tok, 1)])


if __name__ == "__main__":
    main()

Run it twice, once against each column, with identical prompt and identical max_tokens.

export LLM_BASE=http://127.0.0.1:8080/v1 LLM_MODEL=local-model
python sweep.py > local.csv

export LLM_BASE=https://your-endpoint/v1 LLM_MODEL=remote-model LLM_KEY=...
python sweep.py > remote.csv

Keep REQS_PER_LEVEL at 24 or higher. A p95 computed from eight samples is noise, and you will chase that noise for a week.

Any OpenAI-compatible endpoint fits the second column. MonkeyCode's free model access is one candidate, per the operator, and it also offers a free server option for the case where your laptop should not be the host. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Reading the sweep

The rule is one sentence. The crossover is the smallest concurrency N where remote p95 is lower than local p95 and remote error rate stays at or below one percent. Below that N, the local endpoint keeps the request.

Write the result into the README as a plain statement: "Above six concurrent requests the remote endpoint has the better p95." That line ends most routing arguments before they start.

Watch the gap between p50 and p95 as well. A flat p50 with a climbing p95 means a straggler in the queue, not a bandwidth ceiling. Add a timeout and a fallback before tuning anything else, because tuning a straggler usually moves it rather than removing it.

The offline run

Local-first pays off when the wire is gone, so test that claim instead of assuming it. Two commands are enough.

unshare -n --map-root-user python sweep.py
# or, less polite:
sudo iptables -A OUTPUT -d <remote-ip> -j DROP && python sweep.py; sudo iptables -D OUTPUT -d <remote-ip> -j DROP

Verify two things under isolation. The client fails with a clear error rather than hanging on a socket. And the local path still answers, which requires weights on disk rather than fetched per request.

The offline row is just another column in the same table. If it does not exist, offline support is a claim in a slide, not a property of the system.

Where the free server wins

Three cases, all decidable from the CSV.

Bursts are the first. A CI job or a backfill fires thirty-two requests at once, your laptop queues them, and a hosted endpoint absorbs them. The sweep prints the exact N where that matters.

Model size is the second. If the weights do not fit in memory, local is not slow, it is impossible. That is a capability gap, not a latency trade-off, and no amount of quantization tuning changes the verdict.

Cold hardware is the third. Sleep, thermal throttling, and battery saver all move p95 in ways you cannot control from a config file.

Signal from the sweep	Keep it local	Move it to the free server
Remote p95 wins below N=4	yes	no
Burst of 16 or more	no	yes
Payload contains customer PII	yes	never
Laptop on battery during demos	no	yes

Allowances are the part people over-fit. The operator states free model access and a free server option; confirm the current allowance inside the product before you write it into an architecture document. Quotas move. Your crossing point does not move nearly as fast.

Who should skip this

Do not send traffic to any remote endpoint if the payload cannot leave the machine. That covers customer data, credentials, and most incident text. No latency number outranks that constraint, and the sweep cannot help you there.

Skip the free server if you need a contractual p95. Free tiers are best-effort by nature, and an SLA is a different product.

Skip local if nobody owns the upgrade path. A pinned model still needs patching, and a stale runtime is a security decision made by inaction.

Keep the sweep in the repo

Re-run it when the hardware changes, when the model changes, and once a quarter regardless. Commit both CSVs and the one-sentence crossing point. Reviewers then argue with evidence instead of intuition.

If you want a second column without a payment method, MonkeyCode's free model access and free server option gives you one this afternoon. Run the sweep first, keep whichever side wins at your real concurrency, and write down why you chose it.

Offline Is a State, Not a Flag

Riley Wu — Sat, 17 Oct 2026 06:34:43 +0000

Offline Is a State, Not a Flag

Local-first apps fail at the seam, not at the start.

Teams usually test offline as a boolean. Airplane mode on, then off. That test passes and the duplicate-charge bug ships anyway. The failure sits in the gap between a local step and a remote step inside one task.

The same shape shows up in bug reports often. A job runs three steps. Two run on the laptop, one runs on a server. The process dies between step two and step three. On restart, the app replays step three.

Now the bill has two line items instead of one.

Latency probes will not catch this. Routing scores will not catch this either. Those answer a placement question. The seam asks a different one: what already happened.

Durability before placement

Placement is a performance decision. Durability is a correctness decision. Ship the second one first.

One rule holds up well. Record intent before the side effect, record completion after it. Then replay only what the journal says is unfinished.

That gives three states per step, not two. New, sent, done. The sent state is the interesting one, and it is the state most apps forget to persist.

Offline is the same case viewed from the other side. When the network returns, the queue is full of sent rows. You either deduplicate them or you pay twice.

A journal you can kill

Here is a single-file runner. No dependencies, Python 3.10 or newer. Save it as seam.py.

# seam.py - resumable steps across the local/remote seam
import json, os, sys, hashlib

JOURNAL = "journal.jsonl"
REMOTE_STORE = "remote_idem.json"  # stands in for the server's dedupe table


def append(entry):
    with open(JOURNAL, "a", encoding="utf-8") as f:
        f.write(json.dumps(entry) + "\n")
        f.flush()
        os.fsync(f.fileno())      # durability happens here, not later


def load():
    if not os.path.exists(JOURNAL):
        return []
    with open(JOURNAL, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]


def status(key):
    rows = [r for r in load() if r["key"] == key]
    if not rows:
        return "new", None
    last = rows[-1]
    return last["status"], last.get("out")


def idem_key(task_id, step):
    raw = f"{task_id}:{step}".encode()
    return hashlib.sha256(raw).hexdigest()[:16]


def step_double(x):
    return x * 2


def step_remote_summary(text, idem):
    store = json.load(open(REMOTE_STORE)) if os.path.exists(REMOTE_STORE) else {}
    if idem in store:
        return store[idem]        # replay: no second side effect
    out = {"idem": idem, "summary": text[:40].upper()}
    store[idem] = out
    json.dump(store, open(REMOTE_STORE, "w"))
    return out


def run(task_id, kill_at=None):
    st, out = status("double")
    if st != "done":
        append({"key": "double", "status": "started"})
        if kill_at == 1:
            os._exit(137)         # crash inside the local step
        append({"key": "double", "status": "done", "out": step_double(21)})
        st, out = status("double")
    doubled = out

    st, out = status("summary")
    if st == "done":
        return out

    key = idem_key(task_id, "summary")
    if st != "sent":
        append({"key": "summary", "status": "sent", "idem": key})
        if kill_at == 2:
            os._exit(137)         # crash after intent, before completion

    result = step_remote_summary(f"value={doubled}", key)
    append({"key": "summary", "status": "done", "out": result, "idem": key})
    return result


if __name__ == "__main__":
    kill = int(sys.argv[1]) if len(sys.argv) > 1 else None
    print(run("task-1", kill_at=kill))

Two details carry the weight. The append call hits fsync before the network call. And every remote step carries a deterministic idempotency key.

The fsync is not decoration. Without it, the journal itself is the crash.

The crash test

Run this sequence. The first invocation dies on purpose.

rm -f journal.jsonl remote_idem.json
python seam.py 2; echo "exit=$?"     # 137, killed at the seam
python seam.py                       # resumes, reuses the same idem key
python seam.py                       # no-op, reads "done"
python -c "import json;print(len(json.load(open('remote_idem.json'))))"

That last command prints 1. The remote side saw exactly one side effect, even though the local process touched the step twice.

Now change one line. Drop the idem field from step_remote_summary and rerun the whole sequence. The count becomes 2, and the bug reproduces in under two minutes.

That is the entire test plan. Kill at every seam, count the effects, assert one.

The seam is also a data boundary

The moment a step leaves the laptop, the input becomes someone else's log line. That cost is real and separate from latency.

An allowlist gates the payload. Nothing else crosses.

ALLOW = {"prompt", "locale", "max_tokens"}

def gate(payload):
    return {k: v for k, v in payload.items() if k in ALLOW}

After the gate, the remote side sees a prompt and a locale. No bearer token, no customer id, no raw database row. Redaction is cheap. Rotation after a leak is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode fits this workflow as the spillover tier, not the primary one. It offers free model access and a free server option, which is enough to run the seam test on hardware you do not own. Check the project page for the current allowance, since free-tier terms move and a quoted number ages badly.

When the free server wins

Local wins when the constraint is the network or the payload. The server wins when the constraint is the machine.

Constraint	Keep local	Free server tier
Hard latency floor under ~50 ms	yes	no
Payload holds credentials or customer rows	yes	no, redact first
Session must survive hours offline	yes	no
Task exceeds laptop time or memory budget	no	yes
Queued burst of batch work	no	yes

Read that table by row, not by column. Most real apps take both paths, and take them per step.

A cheap diagnostic helps here. Count journal rows still in sent after five minutes. A rising number means the remote leg is unreliable, not that the routing rule is wrong.

Limitations

The journal grows without compaction. fsync per step costs milliseconds that hurt at high volume. The remote side has to genuinely support idempotency keys, and accepting a header while ignoring it does not count.

Free tiers are not SLAs. Treat the server tier as best-effort capacity that can vanish, and keep the local path able to finish degraded work.

Skip this pattern if data cannot cross a boundary at all. Skip it if you need the remote side to be a system of record. This is a spillover mechanism, not a ledger.

The takeaway

Test the seam, not the switch. Write intent before the effect, and carry a key that makes replay safe. Placement then gets cheap, because a wrong choice costs latency instead of money.

If you want to rehearse the spillover step without provisioning anything, MonkeyCode's free model access and free server option are enough to run the crash test above end to end.

Probe Both Ends Before You Route

Riley Wu — Fri, 16 Oct 2026 05:40:31 +0000

Probe Both Ends Before You Route

A routing rule is only as good as its worst measurement.

Local-first means the local runtime is the default. Default is not the same as correct.

Two runtimes never behave alike. A local model and a free remote server drift apart quietly. Versions change, limits change, and free tiers can change without a changelog.

Your parser will not complain about the drift. It will coerce, default, or truncate instead.

So measure both ends with one harness. Store the result as data. Let the router read the data.

That is the whole method, and the rest of this post is the artifact.

Five checks, one harness

The harness targets any OpenAI-compatible chat endpoint. It runs the same prompts against local and remote. It writes one JSON card per endpoint.

Here is the call wrapper. It is a sketch, not a benchmark suite.

# probe.py — design sketch. Run it against your own endpoints.
# Pin your own model ids; do not hardcode names you have not tested.
import json, time, statistics, datetime, urllib.request

FILLER = "The archive log records routine events. " * 400

def call(url, key, model, prompt, stop=None, temperature=0.0):
    body = {"model": model, "temperature": temperature,
            "messages": [{"role": "user", "content": prompt}]}
    if stop:
        body["stop"] = stop
    req = urllib.request.Request(
        url, data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=60) as r:
        out = json.loads(r.read())
    return out["choices"][0]["message"]["content"], (time.perf_counter() - t0) * 1000

Check one is strict schema compliance. Ask for three fields, parse the reply, count violations without repairing anything.

SCHEMA_PROMPT = ('Reply with JSON only: '
                 '{"id": int, "label": str, "ok": bool}. No prose.')

def check_json(url, key, model, n=3):
    good = 0
    for _ in range(n):
        text, _ = call(url, key, model, SCHEMA_PROMPT)
        try:
            obj = json.loads(text)
            good += int(set(obj) == {"id", "label", "ok"})
        except json.JSONDecodeError:
            pass
    return good / n

Check two is stop-sequence honor. Vendors differ on whether the stop string itself is echoed. Read your own output once and pin the rule.

def check_stop(url, key, model):
    text, _ = call(url, key, model, "Count: one two three four five.", stop=["three"])
    return "three" not in text and "four" not in text

Check three is instruction retention at depth. Bury a unique token at 25 percent and again at 75 percent of filler text.

TOKEN = "ZQ7-KESTREL"

def check_needle(url, key, model, position):
    cut = int(len(FILLER) * position)
    prompt = FILLER[:cut] + TOKEN + " " + FILLER[cut:] + "\nReply with only the token."
    text, _ = call(url, key, model, prompt)
    return TOKEN in text

Check four is repeat stability at temperature zero. Identical prompts should not produce different structures.

def check_stability(url, key, model, prompt, n=3):
    outs = {call(url, key, model, prompt)[0].strip() for _ in range(n)}
    return 1.0 if len(outs) == 1 else 0.0

Check five is a latency floor. Five sequential calls give a weak but honest shape of the distribution.

def check_latency(url, key, model, n=5):
    times = [call(url, key, model, "Reply with the word ready.")[1] for _ in range(n)]
    return {"median_ms": round(statistics.median(times)),
            "p90_ms": round(sorted(times)[-1])}

Every card carries a timestamp. A measurement without a date is a rumor.

def card(name, url, key, model):
    return {
        "endpoint": name,
        "checked_at": datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds"),
        "model": model,
        "json_strict": check_json(url, key, model),
        "stop_honor": check_stop(url, key, model),
        "needle_25": check_needle(url, key, model, 0.25),
        "needle_75": check_needle(url, key, model, 0.75),
        "repeat_stability": check_stability(url, key, model, SCHEMA_PROMPT),
        "latency": check_latency(url, key, model),
    }

python probe.py --name local  --url "$LOCAL_URL"  --key "$LOCAL_KEY"  --model "$LOCAL_MODEL"  > cards/local.json
python probe.py --name remote --url "$REMOTE_URL" --key "$REMOTE_KEY" --model "$REMOTE_MODEL" > cards/remote.json

A plausible card looks like this. The numbers below are placeholders, not measured results.

{
  "endpoint": "local",
  "checked_at": "2026-10-16T09:12:04+00:00",
  "json_strict": 1.0,
  "stop_honor": true,
  "needle_25": true,
  "needle_75": false,
  "repeat_stability": 1.0,
  "latency": {"median_ms": 640, "p90_ms": 1180}
}

Route by check result, not by machine size

The interesting cases are mixed. A small local model can win the schema check and lose the depth check. A large remote model can win depth and lose stability.

Here is a router that reads only cards. It fails closed when a card is missing or stale.

MAX_AGE = datetime.timedelta(days=3)

def fresh(card, now):
    age = now - datetime.datetime.fromisoformat(card["checked_at"])
    return age <= MAX_AGE

def route(task, cards, now):
    local, remote = cards.get("local"), cards.get("remote")
    if not local or not fresh(local, now):
        raise RuntimeError("no trustworthy local card")
    if task.wants_json:
        if local["json_strict"] == 1.0 and local["repeat_stability"] == 1.0:
            return "local"
        if remote and fresh(remote, now) and remote["json_strict"] == 1.0 \
           and remote["repeat_stability"] == 1.0 and not task.has_secrets:
            return "remote"
        raise RuntimeError("no endpoint meets the structured contract")
    if task.prompt_tokens > 4000 and not local["needle_75"]:
        if remote and fresh(remote, now) and remote["needle_75"] and not task.has_secrets:
            return "remote"
    return "local"

Three rules do most of the work. A stale card is treated like no card. Secret-bearing tasks never leave, whatever the card says. Local stays the default for everything the probes do not justify moving.

A capability card is a passport with an expiry date. An expired passport does not prove you cannot travel. It proves nobody checked recently.

Consider a common failure. A remote endpoint passes the schema check but fails stability. Two of three runs append an extra note field. The parser accepts both shapes. A downstream dedupe keyed on the whole object then counts one record twice.

Nothing throws. The card catches it before the data does.

Where a free endpoint fits

MonkeyCode is an open-source project offering free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Its operator also advertises a free token allowance on the order of ten million tokens.

Treat those limits as operator claims, not verified facts. I have not independently confirmed them, and you should re-probe rather than assume.

What matters for this workflow is narrower. A free endpoint gives the harness a second card to compare against, without a billing setup first. That is enough to answer the only question this method asks: which end satisfies the contract for this task class today.

Model names, quotas, and retention behavior vary. Pin them in your own card and never hardcode what you have not measured.

Limits you should respect

Three repeats is a smoke test, not a confidence interval. Five latency samples bound nothing. Raise both counts before you route money through them.

The probe prompts are not your traffic. A synthetic needle at 75 percent of filler does not prove behavior on your real documents.

The probes cost tokens. A four-thousand-token filler prompt is not free, even on a free tier.

Endpoints change without notice. A three-day expiry is a guess, not a schedule. Shorten it after any incident.

The harness measures behavior, not cause. It cannot tell you why a field vanished.

A capability card never changes your data policy. If bytes must not leave the machine, do not build the remote path at all.

Who should not use this

Skip this if you owe anyone a formal SLA, or if you cannot re-run the probe when an endpoint changes. Skip it if your data cannot leave the machine, since the remote branch is then pure risk.

Skip it if you need statistically solid latency numbers. A handful of calls will mislead you more than no calls.

Close the loop

Probe both ends with the same harness. Date every measurement. Let the router obey the date, and let it fail closed.

If you want a second endpoint to probe and you would rather not put a card on file first, a free tier is the cheapest way to get one.

Write a Receipt for Every Remote Inference

Riley Wu — Thu, 15 Oct 2026 11:36:29 +0000

Local-first is a default, not a doctrine. Remote should win one request at a time, on measured evidence.

Most teams pick a side once and never re-measure. That habit costs time in both directions. The last week of DEV posts on AI fatigue and in-browser AI features makes one point twice: inference is everywhere, and waiting for it is the part people resent.

Two constraints decide the route. The first is the wall-clock time your user will tolerate. The second is how many private bytes leave the machine. Cost sits third until somebody reads the invoice.

All three are measurable before you send a token. That is the entire trick. Measure once, encode a small decision function, and keep a receipt for every request that went remote.

Measure both paths on your own machine

Guessing tokens per second is how local-first setups quietly fail. Your laptop is not the laptop in the benchmark post. Put the numbers in the environment, and keep them out of the source tree.

# .envrc — numbers from your hardware, not from a blog post
export LOCAL_MODEL='the model you already run locally'
export LOCAL_TPS=$(python bench_local.py --model "$LOCAL_MODEL" --runs 5)
export REMOTE_TPS=$(python bench_remote.py --runs 5)
export NET_MS=$(python rtt.py --runs 20)
export REMOTE_BASE_URL='your remote endpoint, injected at runtime'

bench_local.py pushes the same twenty prompts through your local runtime and prints median tokens per second. bench_remote.py does the same against the remote endpoint. rtt.py measures round-trip time, which dominates short completions.

None of these scripts needs to be clever. They need to run on your machine, on your network, on a normal weekday. Then turn those numbers into a decision.

# route.py — illustrative example, not executed while writing this article.
import os, json, hashlib, time

LOCAL_TPS  = float(os.environ['LOCAL_TPS'])
REMOTE_TPS = float(os.environ['REMOTE_TPS'])
NET_MS     = float(os.environ['NET_MS'])
BUDGET_S   = 2.0

def local_seconds(tokens):
    return tokens / LOCAL_TPS

def remote_seconds(tokens):
    return NET_MS / 1000.0 + tokens / REMOTE_TPS

def choose_route(segments, budget_s):
    tokens = sum(s['tokens'] for s in segments)
    if any(s['sensitivity'] == 'secret' for s in segments):
        return 'local', 'secret segment present'
    if local_seconds(tokens) <= budget_s:
        return 'local', 'local path fits the budget'
    if remote_seconds(tokens) <= budget_s:
        return 'remote', 'local path misses the budget'
    return 'queue', 'no path meets the budget'

Notice what the function does not do. It does not ask whether you like cloud services. It does not care about brand. It answers one question per request: which path satisfies this budget for this payload?

Receipts, not logs

A log line records that something happened. A receipt records why a decision was made. Write one for every remote call, and never write the payload into it.

def receipt(segments, route, reason, latency_s, tokens_out):
    sent = json.dumps(segments).encode()
    return {
        'ts': time.time(),
        'route': route,
        'reason': reason,
        'prompt_sha256': hashlib.sha256(sent).hexdigest()[:16],
        'bytes_sent': len(sent),
        'tokens_out': tokens_out,
        'latency_s': round(latency_s, 3),
        'budget_s': BUDGET_S,
    }

Append these as JSONL. After one week you can answer the only questions that matter. Which prompts left the machine? Which ones missed their budget? Where did the local path keep winning anyway?

That hash is the important field. It lets you correlate a receipt with a prompt you still hold locally, without copying private text into a shared log. Rotate the mapping if your retention policy demands it.

The table I use

Payload	Budget	Default route
Any secret segment	any	local, always
Public, long context	over 10s	remote
Public, interactive	under 2s	local if it fits
Offline	any	local, queue the rest

The middle rows are where the interesting fights happen. A long public document may be far cheaper to summarize remotely, while an interactive edit will not tolerate the round trip. Users forgive latency on batch work and never forgive it in a chat box.

The secret row is not negotiable. Once a credential or customer record enters the payload, the budget stops mattering. Route it local, or fail closed and tell the user why.

When a free server wins

Remote wins in four honest cases. Cold start, where your machine must load weights first. Long context, where local memory runs out. Batch work, where p95 latency is irrelevant. And hardware you do not own, which is most hardware.

Loops change the calculus too. An agent that calls a model forty times per task feels every millisecond of round-trip time. That workload belongs local, or it belongs on a queue with a human-facing summary at the end.

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers free model access and a free server option. The operator states that 10 million free tokens are available, with the free server acting as the remote tier. Availability terms and quotas can change, so verify them on the project page before you plan around them.

In this workflow it is simply the remote branch of the table. Point REMOTE_BASE_URL at the free tier, run bench_remote.py against it, and let the receipts decide whether it earns a place. Test it the way you test anything else: your budgets, your payloads, your numbers.

Limitations and who should skip this

This approach fails in predictable places. Regulated data usually forbids remote inference no matter how good the latency is. Hard offline environments need a local queue and a conflict policy, not a routing function. Teams without an egress review process should not add a remote tier at all, because receipts are evidence, not permission.

Skip it if you only have one workload. A single fixed task rarely needs a router; hardcode the path and revisit when the second workload arrives. Free tiers are also the wrong foundation for a business. They are a fine place to measure one.

Local-first survives a bad network. A free server survives a weak laptop. Decide per request, write the reason down, and let the measurements change your mind.

Route by Payload, Not by Habit

Riley Wu — Tue, 13 Oct 2026 03:22:00 +0000

Keep every working tree on the local disk. Send one redacted hunk only after local checks pass. A remote pass stays optional, never the default path.

Pasting a whole repository is the common failure. That habit spends latency and leaks secret lines. A filing cabinet should not travel for one page.

Local inference starts without a network round trip. Remote inference waits on DNS, TLS, and queue time. For a tiny rename, that wait rarely pays.

Measure the path before you trust a feeling. A short timed connect shows whether the route answers. Reachability is not model quality, and it is not a quota.

Secrets ride inside diffs more often than people think. Tokens, private keys, and customer rows do not belong on a free server. Scan the hunk on disk before any upload begins.

Offline work is a normal state, not an error page. Write the request to a local queue and keep editing. Promote that queue only after the link returns.

A free server wins on a narrow class of jobs. The hunk is small, clean, and hard for local models. You still keep tests and secrets on the laptop.

MonkeyCode can serve as the remote hop here. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes open-source code plus free model access and a free server.

This draft claims no quota, hardware, or permanent tier. Check current project docs before you plan around them. A blog post cannot freeze terms that later change.

Treat the free server like a reading room across town. You carry one page, never the whole cabinet. You leave when the room closes, and the cabinet stays.

Latency has three visible parts, even without a benchmark lab. Name lookup, handshake, and first token are separate waits. A local process skips all three, then spends only compute.

That split matters on flaky links more than on a campus LAN. A laptop on a train loses the path between keystrokes. The edit should not freeze while a socket times out.

Secret risk is not the same problem as slow risk. A fast server can still be the wrong destination. Speed does not wash a key that left the disk.

Offline risk is the third axis, and it is blunt. If the route is down, remote quality is irrelevant. The honest result is a queued file beside the diff.

The decision below uses those three axes and nothing fancier. Payload bytes stand in for the cabinet size. A secret hit or a dead route blocks the trip.

Thresholds in the script are example knobs, not measured optima. Eight thousand bytes is a starting fence, not a law. Change it after you inspect your own diffs.

Classify one hunk

The artifact below is a proposal, not a measured benchmark. It classifies one diff as local, queued, or remote. Run it on a sample file before you trust it.

#!/usr/bin/env python3
"""Proposal only. Routes one diff. Not a benchmark."""
import os
import re
import socket
import sys
from pathlib import Path

# Example fence, not a measured optimum.
MAX_REMOTE_BYTES = 8000
CONNECT_TIMEOUT_S = 1.5

SECRET_RULES = (
    re.compile(r"AKIA[0-9A-Z]{16}"),
    re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    re.compile(r"(?i)\b(api[_-]?key|secret|token|password)\b\s*[:=]\s*\S+"),
)

def load_diff(path: Path) -> str:
    return path.read_text(encoding="utf-8", errors="replace")

def secret_hits(text: str) -> int:
    return sum(len(rule.findall(text)) for rule in SECRET_RULES)

def route_open(host: str, port: int) -> bool:
    try:
        with socket.create_connection((host, port), CONNECT_TIMEOUT_S):
            return True
    except OSError:
        return False

def decide(text: str, host: str, port: int) -> str:
    if secret_hits(text) > 0:
        return "local"
    if len(text.encode("utf-8")) > MAX_REMOTE_BYTES:
        return "local"
    if not text.strip():
        return "local"
    if not host:
        return "local"
    if not route_open(host, port):
        return "queued"
    return "remote"

def main() -> int:
    if len(sys.argv) != 2:
        print("usage: route_hunk.py DIFF_FILE", file=sys.stderr)
        return 2
    text = load_diff(Path(sys.argv[1]))
    host = os.environ.get("REMOTE_HOST", "")
    port = int(os.environ.get("REMOTE_PORT", "443"))
    choice = decide(text, host, port)
    print(choice)
    if choice == "queued":
        queue = Path("queue")
        queue.mkdir(exist_ok=True)
        target = queue / Path(sys.argv[1]).name
        target.write_text(text, encoding="utf-8")
        print(f"queued {target}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Capture a hunk with git, then classify it. Keep the command local until the script prints remote. Do not pipe the diff to a network tool yourself.

git diff --unified=3 -- src/app.py > /tmp/hunk.diff
python3 route_hunk.py /tmp/hunk.diff

An unset host means the script stays local. That empty host is intentional, not a bug. Set a host only when you mean to test the path.

REMOTE_HOST=example.invalid REMOTE_PORT=443 python3 route_hunk.py /tmp/hunk.diff

Use a host you operate or the docs name. This article does not invent a product endpoint. The name example.invalid should land in the queue.

A clean sample looks like a few changed lines. No keys, no dumps, and no giant fixture. That shape is the page you might carry across town.

diff --git a/src/app.py b/src/app.py
--- a/src/app.py
+++ b/src/app.py
@@ -10,3 +10,3 @@
-def timeout_ms():
-    return 200
+def timeout_ms():
+    return 350

That hunk is small and has no secret shape. If the route answers, the script may print remote. If the route fails, it prints queued and writes a copy.

A dirty sample should never print the remote word. Plant a fake key in a scratch diff and confirm the local result. Delete the scratch file when the check ends.

printf 'api_key = "not-a-real-secret"\n' > /tmp/dirty.diff
python3 route_hunk.py /tmp/dirty.diff

The expected printed word is local, not remote. The regex is a coarse net, not a full secret scanner. It misses some formats and flags some harmless lines.

False positives belong on the local side of the gate. A missed remote call is cheaper than a leaked row. Widen the rules only after you review real diffs.

False offline signals happen on slow DNS too. A one point five second timeout is an example, not a standard. Raise it if your network is slow but stable.

Do not treat a remote print as permission to upload the tree. Upload the same hunk the script read, or upload nothing. The queue file is still private local state.

When the choice is remote, ask for a bounded review. Name the function, the bug, and the test you already hold. Refuse a request that needs the whole history.

A useful prompt stays smaller than the hunk fence. You can paste the diff into a free-server session by hand. This script does not call an API, and it does not store a token.

That gap is deliberate, because upload would hide the last check. The printout is a gate, and you are the latch. Automatic shipping would train a careless copy habit.

Local remains better for rename sweeps and format noise. Those diffs are wide, dull, and full of names. A remote model adds delay and little new structure.

Remote remains better for a short logic knot. You want a second reading of a clean algorithm. Free model access matters only in that narrow slot.

If the free server is down, the queue still preserves the ask. Replay is a later command, not a background daemon. A daemon that retries blindly can ship a stale hunk.

python3 route_hunk.py queue/hunk.diff

Compare the queued bytes with the current git diff before replay. If the bytes differ, discard the queue file. Stale context produces confident notes that miss the edit.

There is no latency table in this piece, on purpose. No run was executed for publication, so no millisecond claim is honest. Time your own path if a number will guide a team rule.

The same restraint covers cost and capacity claims. Free model access can still meet a fair-use cap you have not read. A blog sentence cannot reserve capacity for your sprint.

Where the route stops

Limitations sit in the regex, the timeout, and the byte fence. Binary diffs may decode as noise and look small. Review binary changes by hand, because the scanner is text-biased.

A successful connect proves a socket, not a healthy model. Auth, region, and capacity can still fail after the port opens. Treat remote as eligible, then confirm with a manual prompt.

The script does not understand licenses or customer contracts. A clean regex is not a legal review. If policy forbids external models, the only result is local.

Do not use this approach for production incident logs. Those files mix tokens, hostnames, and personal data. Keep that material on disk even when the hunk looks short.

Do not use it when the local tree itself is untrusted. A malicious diff can try to confuse a reviewer. Remote reasoning will not purify a poisoned patch.

Do not use it as a silent team default. Each developer should see the printed choice first. Hidden routing trains people to forget where text went.

Skip the free server when you cannot read the current terms. Skip it when the docs disagree with a remembered quota. Memory is a poor source of capacity planning.

If your work is air-gapped, stop at the local branch. Leave the host unset and treat remote as dead code. The queue remains optional even in that setting.

The workflow is a loop you can say in one breath. You diff, scan, classify, and then edit. Cross the wire only for a page that earned the trip.

If you already work this way, the free server is a spare reading room. Read the live MonkeyCode docs, then try one clean hunk. Keep the cabinet at home either way you choose.

Secret Bytes Pick the Free Server

Riley Wu — Sun, 11 Oct 2026 22:24:40 +0000

A free server is a valid route only after redaction. Latency savings never repay a secret on the wire. Offline work must finish on the machine alone.

Remote model quality moves faster than most gates. That movement does not relax the exit rule. A sharper answer is still wrong with a leaked key.

Treat the prompt as a packed suitcase at a dock. You do not mail the suitcase to save folding time. You remove the passport, then you time the walk.

This note is a method, not a recorded lab report. The numbers below are fixtures for a dry run. They are not measured speeds from a live service.

Three costs before any client

Local inference keeps bytes inside one process boundary. A free server moves those bytes across a foreign network. The trade buys reach and sells forgotten fields.

Three costs decide the route before any client runs. Secret bytes come first, then delay, then offline survival. Drop any one cost and the choice becomes a guess.

Secret bytes are characters that must never leave. Tokens, passwords, private paths, and customer rows all count. A placeholder is clean, but the raw value is not.

Round-trip time is the gap a person can feel. Measure it with a tiny probe from the real network. A status page does not know your last mile.

Offline survival is the rule that closes the design. If the link drops, local code must still emit a result. A free server cannot be the only brain available.

A gate you can diff

The artifact is a small Python gate you can diff. It never calls a model and never stores a secret. The sample targets Python 3.10 or newer for hints.

The script reads JSON from standard input and counts fields. It compares a local delay fixture with an optional probe. It prints a route plus a redacted copy of the body.

Label the script as a proposal, not a finished study. This draft has not executed it against a live fleet. Treat every threshold as a knob, not a law.

#!/usr/bin/env python3
# hop_gate.py - proposal, not a recorded benchmark.

from __future__ import annotations

import json
import os
import sys
import time
import urllib.request

SECRET_KEYS = {
    "api_key",
    "token",
    "password",
    "secret",
    "authorization",
    "cookie",
    "private_key",
    "ssn",
    "email",
}

MAX_SECRET_BYTES = 0
MIN_RTT_GAIN_MS = 80
PROBE_TIMEOUT_S = 1.5


def secret_bytes(payload: dict) -> int:
    total = 0
    for key, value in payload.items():
        name = str(key).lower()
        text = "" if value is None else str(value)
        if name in SECRET_KEYS and text not in ("", "[REDACTED]"):
            total += len(text.encode("utf-8"))
    return total


def redact(payload: dict) -> dict:
    clean = {}
    for key, value in payload.items():
        if str(key).lower() in SECRET_KEYS:
            clean[key] = "[REDACTED]"
        else:
            clean[key] = value
    return clean


def probe_rtt_ms(url: str) -> float | None:
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(url, timeout=PROBE_TIMEOUT_S) as resp:
            resp.read(64)
    except OSError:
        return None
    return (time.perf_counter() - start) * 1000.0


def decide(payload: dict, local_ms: float, remote_ms: float | None) -> str:
    if secret_bytes(payload) > MAX_SECRET_BYTES:
        return "local"
    if remote_ms is None:
        return "local"
    gain = local_ms - remote_ms
    if gain < MIN_RTT_GAIN_MS:
        return "local"
    return "free_server"


def main() -> int:
    raw = json.loads(sys.stdin.read() or "{}")
    local_ms = float(os.environ.get("LOCAL_MS", "400"))
    probe = os.environ.get("PROBE_URL", "")
    remote_ms = probe_rtt_ms(probe) if probe else None
    route = decide(raw, local_ms, remote_ms)
    report = {
        "route": route,
        "secret_bytes": secret_bytes(raw),
        "redacted": redact(raw),
        "local_ms_fixture": local_ms,
        "remote_probe_ms": remote_ms,
    }
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Feed the gate a fixture, not a live customer record. An empty token is not a leak under this classifier. A filled token is a hard stay on the machine.

Point the probe only at a host you already accept. The sample uses example.com as a stand-in address. It is not an endpoint for the product discussed later.

Read the JSON report before any client sends a body. The redacted object is the only payload allowed out. The original object stays on disk beside the diff.

Where the remote lane fits

MonkeyCode is the optional remote lane inside this gate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator notes cite free model access and a free server option.

This draft states no cap, no hardware, and no duration. Speed figures were not supplied, so none are printed here. Plan capacity from the docs, not from this page.

Read the current project docs before you trust the lane. Then try one redacted hop on a fixture you control. If the docs disagree with this note, trust the docs.

Use that lane only when the gate prints free_server. Keep the local branch able to finish the same task class. If the lane vanishes, the loop should still emit a patch.

An offer can change without notice in this kind of lane. Plan for withdrawal the same way you plan for packet loss. Your local branch is the continuity plan, not a slogan.

Ferry days and bridge days

Picture a ferry beside a short footbridge on an island. The ferry is the free server on a good weather day. The footbridge is local inference, slower but contained.

You board only after the crate has been searched. A found passport means you walk, even if the ferry waits. A late ferry or a closed dock means you walk too.

The order stays stable across tasks and across weeks. Redaction comes before timing, and timing before the send. The send is optional, while the local result is not.

A free server wins inside a narrow and boring band. The payload is clean, and the probe beats your knob. The task is synthesis, not a lookup of private rows.

You must be willing to discard a weak remote answer. A pretty diff is not proof that the route was safe. Safety was decided before the request left the disk.

A free server loses across a much wider set of cases. Any remaining secret byte forces an immediate local stay. A failed probe, a tie, or a dead link does too.

Legal export bans also end the remote option at once. So does a local model that already solves the edit. Reach is not a prize when the island is sufficient.

A rehearsal row can show the arithmetic without a benchmark claim. Set the local fixture at 420 ms and the probe at 90. The gap is 330 ms, so a clean payload may hop.

Swap the probe result to 400 ms and the route stays local. The knob made that call, not the name of the lane. Change the knob only after traces from your own editor.

Four fixtures without source text

Run four fixtures and record the route, not the source. Store counts, routes, and timings in the test log. Leave private file text out of that log entirely.

Fixture A carries a filled token and a short task. Expect local, and expect a secret count above zero. Any other route means you must fix the classifier.

Fixture B has no secret fields and a slow local fixture. Expect free_server only when the probe beats the knob. A slower probe that returns local is still a pass.

Fixture C blocks the probe host or drops the network. Expect local, and expect the probe time to be null. The script must return before the timeout knob expires.

Fixture D replays a saved redacted file with no network. The local editor must still produce a usable diff. Otherwise you have a remote dependency in a local costume.

The function checks below do not touch the network. They lock the route rules before any probe is trusted. Run them after you save the script as hop_gate.py.

mkdir -p fixtures
printf '%s\n' '{"task":"rename symbol","token":"fixture-token-value","path":"src/app.py"}' > fixtures/a_token.json
printf '%s\n' '{"task":"rename symbol","token":"","path":"src/app.py"}' > fixtures/b_clean.json
printf '%s\n' '{"task":"rename symbol","path":"src/app.py"}' > fixtures/c_offline.json
python3 -c 'import hop_gate; assert hop_gate.decide({"token":"x"}, 420, 90)=="local"; assert hop_gate.decide({"task":"rename"}, 420, 90)=="free_server"; assert hop_gate.decide({"task":"rename"}, 420, 400)=="local"; assert hop_gate.decide({"task":"rename"}, 420, None)=="local"'
export LOCAL_MS=420
export PROBE_URL=https://example.com
python3 hop_gate.py < fixtures/a_token.json | tee /tmp/a.json
unset PROBE_URL
python3 hop_gate.py < fixtures/c_offline.json | tee /tmp/c.json
python3 -c 'import json; a=json.load(open("/tmp/a.json")); c=json.load(open("/tmp/c.json")); assert a["route"]=="local"; assert c["route"]=="local"; assert c["remote_probe_ms"] is None'

Keep fixtures in the repo and keep real prompts out. The test proves the gate, not the wit of a model. A green run can still hide a bad client binding.

Teams often measure tokens and skip secret byte counts. A short prompt can still hold a production key. Length is a weak proxy, and field class is stronger.

Another miss is timing only the office happy path. Train wifi and hotel links fail in different ways. Run the probe from the network you actually use.

A third miss prints redaction and then sends the original. The report is not the request your client will make. Bind the client to the redacted object in review.

The remote lane should see the redacted task only. Public error text and a chosen file slice can join it. Shell history and environment files stay off that lane.

Limits and who should walk away

This gate is a heuristic built on key names alone. Secrets buried in prose will pass that shallow net. A pasted connection string is the classic miss.

Add a high-entropy scan if your tasks include blobs. The proposal script does not ship that second scan. Do not pretend the key list is a complete scanner.

The delay knobs are placeholders until your samples replace them. The default local delay exists so the script can run. Do not publish that default as a service benchmark.

This draft records no personal timing study on a fleet. Replace the fixture with traces from your own editor. Until then, treat printed milliseconds as rehearsal data.

Free model access can be limited, changed, or withdrawn. This article does not know the current numeric cap. Do not design a launch around an unnamed quota.

Need a signed capacity number before you ship traffic. Then buy capacity and write that number into a contract. A blog note is the wrong place to store that promise.

Skip this approach for health data and for payment data. Skip it for customer secrets and for air-gapped networks. Redaction by key name is not a compliance program.

The free server is egress, even when the price is zero. Calling the lane free does not make the bytes local. An air-gapped rule forbids that hop without exception.

Skip the pattern if you cannot keep a local fallback. A single remote path is a bet, not a workflow. The gate matters only when both branches can finish.

Skip it when on-device edits are already fast and clean. The hop then adds failure modes without a real gain. Local stays the whole design in that quiet case.

Close on the decision record

Write the route decision beside the diff on disk. A later reader should see why the hop was allowed. Counts, route, and probe time are the fields to keep.

Raw prompts do not belong in that decision record. When the route says local, finish the edit on disk. When it says free_server, send the redacted object only.

When the probe time is null, the island does the work. Remote quality will keep moving over the coming months. Your suitcase rule should stay stubborn and unchanged.

Four Scores Before You Leave Local

Riley Wu — Fri, 09 Oct 2026 15:36:37 +0000

Local help should remain the default path for daily coding. A free remote server wins only on narrow, non-secret work. Four scores decide the hop before any token leaves the desk.

Developers often treat the network as a free lunch. The lunch is free only when the plate is empty of secrets. Offline days and slow links still charge you in waiting.

This note is a decision method, not a product tour. The scores come from checks you can run on one laptop. No vendor benchmark is assumed, and none is invented here.

The hop is a tax

A completion request looks small until it carries a secret. File paths, tokens, and customer names ride inside ordinary prompts. Once they cross the wire, local undo cannot pull them back.

Latency has the same shape as a tax on each edit loop. A local model pays that tax in compute on your machine. A remote model pays it in round trips plus queue time.

Offline work breaks the remote path without a warning banner. Trains, flights, and flaky cafe links are normal developer weather. A workflow that requires the cloud will stall in that weather.

Cost is the fourth vote, and free is not the same as fit. A free server can be the right tool for a redacted fixture. It is the wrong tool when the fixture still holds a secret.

Four scores, one route

Score each factor from zero to two before you send anything. A zero on any factor means the turn stays on the machine. Only a full pass on all four factors opens the server.

A timer alone cannot see a secret or an offline flag. Cost also stays invisible if you only watch the clock. The card exists so those missing votes get a number.

Latency score starts with a budget you choose for the task. Interactive edits often need a reply inside one second. Batch refactors can wait longer without breaking the editor flow.

Measure the empty hop before you measure the model. A health check with no repository text shows network time alone. If that empty hop already breaks the budget, do not send code.

Secret score is binary in practice, even if the scale has steps. Any live credential, customer row, or private diff scores zero. Redacted fixtures and public docs can score two after a scan.

Offline score asks whether the task must finish without a network. If the answer is yes, the remote option scores zero today. Queue the request on disk and retry when the link returns.

Cost score asks whether a free server changes the decision at all. If local compute already fits the budget, free remote adds little. Use the free path when local hardware cannot hold the context.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator supplied two availability claims for this draft. MonkeyCode offers free model access and a free server option.

Model names, token quotas, hardware, and duration are not stated here. Those details were not verified against a primary source for this draft. Treat free access as a trial lane, not as a permanent contract.

A proposal you can run

The artifact below is a proposal, not a recorded run. It encodes the four scores in one small function. You can paste it into a scratch file and run the tests locally.

Fail closed works like a turnstile with four arms. One open arm does not let the bag through. A secret hit or a missing hop time keeps the arm down.

# Proposal only. Not executed for this draft.
# Score 0 keeps the turn local. Score 2 allows that factor.
# Any zero fails the remote hop closed.

from dataclasses import dataclass

@dataclass
class TaskCard:
    name: str
    latency_ms_budget: int
    empty_hop_ms: int | None
    secret_hits: int
    must_finish_offline: bool
    local_can_hold_context: bool
    free_server_available: bool

def score_latency(card: TaskCard) -> int:
    if card.empty_hop_ms is None:
        return 0
    if card.empty_hop_ms > card.latency_ms_budget:
        return 0
    if card.empty_hop_ms > card.latency_ms_budget // 2:
        return 1
    return 2

def score_secrets(card: TaskCard) -> int:
    if card.secret_hits > 0:
        return 0
    return 2

def score_offline(card: TaskCard) -> int:
    if card.must_finish_offline:
        return 0
    return 2

def score_cost(card: TaskCard) -> int:
    if not card.free_server_available and not card.local_can_hold_context:
        return 0
    if card.local_can_hold_context:
        return 1
    return 2

def decide(card: TaskCard) -> str:
    scores = (
        score_latency(card),
        score_secrets(card),
        score_offline(card),
        score_cost(card),
    )
    if any(value == 0 for value in scores):
        return "local"
    if all(value == 2 for value in scores):
        return "free_server"
    return "hold"

def sample_cards() -> list[TaskCard]:
    return [
        TaskCard("private-diff", 800, 120, 1, False, False, True),
        TaskCard("flight-edit", 800, 90, 0, True, False, True),
        TaskCard("public-fixture", 5000, 140, 0, False, False, True),
    ]

def test_cards() -> None:
    cards = sample_cards()
    assert decide(cards[0]) == "local"
    assert decide(cards[1]) == "local"
    assert decide(cards[2]) == "free_server"

if __name__ == "__main__":
    test_cards()
    for card in sample_cards():
        print(card.name, decide(card))

Fill empty_hop_ms from a health probe that carries no prompt. Use the total time field, converted to milliseconds. A failed probe should leave the hop time empty.

# Unexecuted probe. Health route only. No repo files.
export HEALTH_URL="https://example.invalid/health"
curl -o /dev/null -s -w 'dns:%{time_namelookup} connect:%{time_connect} total:%{time_total}\n' "$HEALTH_URL"

Read dns, connect, and total as three separate fields. Dns time points at resolver trouble, not at the model. Connect time points at the path, and total includes the server ack.

Do not paste a prompt into that probe to make it realistic. Realism here would smuggle the secret you meant to protect. Realism belongs in a local fixture after redaction, not on the wire.

Fill secret_hits from a local scan, not from a remote classifier. Count matches for keys, private blocks, and your own token prefixes. A count above zero must force the decision back to local.

# Local secret scan proposal. Tune patterns to your formats.
rg -n --hidden -g '!.git' -e 'AKIA[0-9A-Z]{16}' -e '-----BEGIN ' .

Redaction is a local edit, done before any score can reach two. Replace names, ids, and urls with stable placeholders. Then scan again, because a placeholder can still hide a live key.

Read the three sample cards

Consider a public fixture that holds no customer data. The laptop cannot fit the context, and the link is up. The empty hop sits inside a five second batch budget.

That card can score two on every factor. The decision name is free_server, not a promise of quality. You still review the patch before it lands.

Now flip one field and watch the arm drop. Add a single secret hit to the same card. The function returns local, even when the server is free and fast.

A hold result is a pause, not a silent send. Latency scored one, or local compute already fits. Keep the turn local until a human raises the budget on purpose.

Write held cards to a local queue instead of a chat box. A JSON line per task is enough for a later replay. Replay only after the four scores pass again.

A queue line can store the task name and the four scores. Leave the prompt body out of that line. A later run should rebuild the card from the repo, not from the log.

Short edit loops should prefer the local path even when the server is free. A two hundred millisecond hop can still feel late inside a tight edit. Longer jobs can spend that same hop because nobody is waiting.

Context size is a local constraint, not a moral one. If the window will not fit, local fails for a mechanical reason. That is the cleanest case for a free server, once secrets are gone.

Record the hop time next to the decision in a plain text log. Do not copy a number from a blog into that log. Your link is the only latency figure that belongs in the card.

A free lane can be slow, full, or simply gone that day. The score does not retry forever in the background. On a timeout, return the card to the local queue.

Add the three asserts to your usual test command. A secret card must stay on the local path. An offline card must stay local, and a clean batch card may leave.

Raise a latency budget in the card, not in a hidden config. The number should sit next to the task name in review. Reviewers can then see why the hop was allowed.

When the free server wins

A free server wins when four facts line up on the same card. The hop fits the budget, and the scan is clean. The task can wait for a link, but local hardware cannot hold the window.

Use the gate when a local assistant is already in place. An optional remote lane is the only cloud piece required. Solo developers and small teams can share one secret list.

Who should skip this

Skip this gate if you cannot name what counts as a secret. A vague policy will mark everything safe and leak the rest. Teams without a scanner should stay local until that gap closes.

Skip the free server if your work must be reproducible next quarter. Unverified quotas can change, and this draft does not lock a number. Air-gapped repos should ignore the remote branch entirely.

Limits

The script does not call a model and does not open a socket. It only maps fields you already measured on the machine. Wrong inputs produce a confident wrong route, so measure first.

These checks do not prove a vendor is safe or fast. They prove your task cleared your own bars on this day. Run them again when the network, the repo, or the budget changes.

If a local loop is already your default, score one redacted fixture. Compare that card with the free server option only. Publish the four numbers beside the patch, not a slogan.

Three Clocks Pick Local or Server

Riley Wu — Thu, 08 Oct 2026 10:58:00 +0000

A model call should stay local until three clocks agree. Latency, secrets, and offline need are those clocks. A free server wins only inside that narrow overlap.

The laptop is a workshop with the lights already on. The network is a ferry with a posted schedule. You board when the cargo is clean and the crossing fits the deadline.

Habit is a bad dispatcher for this choice. A fast demo last week does not price today's hop. A free seat does not make a secret safe to move.

The latency clock

Latency is the first clock on the desk. Interactive completion dies when the pause outlasts the thought. Batch work can spend more time and still stay useful.

Write the budget before you choose a host. A completion loop might allow four hundred milliseconds. A nightly rename can allow several seconds of delay.

Measure the hop with a probe you control. Keep the median of five runs from the same network. Discard a blog number that names a different city and hour.

# Proposed probe. Point it at a host you operate or trust.
for i in 1 2 3 4 5; do
  curl -o /dev/null -s -w '%{time_total}\n' https://example.invalid/health
done

Sort those five totals and take the middle value. That middle value is your median round trip. Compare it with the budget you wrote down first.

If the median already consumes the budget, the server cannot win. Local execution or a plain deferral is the honest result. Do not average away a failure with a lucky first sample.

Cold start is a separate reading on the same clock. The first call often pays a load cost later calls skip. Warm the process once, then record the five interactive samples.

Use the warm median for a loop that stays open. Use the cold reading when each task starts a fresh process. Mixing those two numbers produces a policy that lies.

The secret clock

Secrets are the second clock on the same desk. A prompt can carry keys, rows, and unreleased diffs. Those fields do not become harmless because a server invoice is zero.

Scan the payload before any client opens a socket. A keyword hit is a hard stay on the laptop. A clean scan only allows the other clocks to continue.

SENSITIVE = ("api_key", "authorization", "password", "private_key", "ssn")

def secret_boundary(text: str) -> bool:
    lowered = text.lower()
    return any(token in lowered for token in SENSITIVE)

This scan is a gate, not a proof of safety. It misses novel field names and encoded blobs. A person still owns the last look at regulated text.

Treat source trees as secret until a review says otherwise. A private monorepo can leak through a stack trace alone. Strip traces before you even run the keyword gate.

The offline clock

Offline work is the third clock on the desk. Planes, incident bridges, and sealed build rooms do not retry politely.

The job must finish while the network is down. A server route has already failed that job.

Write the offline flag before you open a client. A missing flag is not a recorded false. It is an unknown, and unknowns should defer the call.

task:
  name: rename-internal-helper
  latency_budget_ms: 400
  offline_required: true
  contains_secrets: false
  payload_bytes: 12000

Payload size belongs beside the offline flag you stored. A small rename may fit a local model easily. A multi-file context can blow past laptop memory and still be secret-free.

Where a free server wins

A free server wins a small set of jobs. The payload has no secrets, and the offline flag is false. Measured round trip fits the budget, and the laptop cannot finish.

Cost is a vote, not the chair of the meeting. A zero invoice removes one objection from the table. It does not erase a secret or a missed deadline.

MonkeyCode enters here as one host that can take the server side. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states that free model access and a free server option exist.

The operator also presents the project as open source. This draft claims no model names, quotas, hardware, or offer duration. Pin nothing until the docs you open still match the task.

Check the project docs on the day you run the probe. Availability moves, and a copied figure goes stale quickly. The clocks still outrank any offer you find there.

Rerun the route

Decide local_can_finish with a time box, not with a feeling. Give the local runner the same budget you gave the network. A timeout is evidence, and a success is evidence too.

# Proposed local time box. Exit 124 means the box was missed.
timeout 2s python3 -m local_runner --dry-run < task.json
echo "exit:$?"

Exit zero can set local_can_finish to true for that task shape. Exit 124 can set it to false for that same shape. Neither exit is a quality score for the model text.

The function below is a proposed policy module. It was not executed against a live host for this draft. Replace the timing field with your probe before you trust a label.

from dataclasses import dataclass

@dataclass
class Task:
    latency_budget_ms: int
    offline_required: bool
    contains_secrets: bool
    local_can_finish: bool
    measured_rtt_ms: int | None

def route(task: Task) -> str:
    if task.offline_required or task.contains_secrets:
        return "local"
    if task.measured_rtt_ms is None:
        return "defer"
    if task.measured_rtt_ms > task.latency_budget_ms:
        return "local"
    if not task.local_can_finish:
        return "free_server"
    return "either_local_preferred"

Read the labels as policy, not as a vendor score. local means the bytes stay on the machine that created them. free_server means the clocks passed and the laptop cannot finish.

defer means you refused to invent a round trip. either_local_preferred means the server is fast enough, yet the laptop can still finish. Prefer local in that tie, because movement still has a cost.

Save the function as route_task.py before you run the driver. The driver imports that file from the working directory. A missing import is a setup error, not a policy result.

Pass fields in dataclass order, or switch to keywords. A swapped boolean silently changes the route you meant.

python3 - <<'PY'
from route_task import Task, route

open_job = Task(800, False, False, False, 180)
secret_job = Task(800, False, True, False, 180)
print(route(open_job))
print(route(secret_job))
PY

Expect free_server on the first line of output. Expect local on the second line of output. If either line drifts, fix the policy before you fix the network.

RTT vs budget	Offline	Secrets	Laptop finishes	Route
inside	no	no	no	free server
inside	no	no	yes	local preferred
missing	no	no	no	defer
any	yes	any	any	local
any	any	yes	any	local

The table is a policy, not a measurement of any vendor. Fill the round-trip cell from a probe you ran today. Do not paste a foreign median and call it current evidence.

Order the checks so the cheap refusal happens first. Offline and secrets return local before any socket opens. The probe runs only after those two gates pass.

That order saves time, and it also saves mistakes. A latency script that uploads a sample prompt has already lost. Probe a health path, never a real task body.

What this will not do

Limitations sit in the gaps the labels cannot name. A keyword list misses secrets buried in minified bundles. A median hides a slow tail that ruins a live demo.

A free server can be limited, blocked, or withdrawn without notice. A local model can be too weak for the refactor you need. None of those facts appear inside the returned label.

Region and identity add two more gaps to the method. A health check from home is not a check from a build agent. A shared free server may log metadata even when you send no secrets.

Skip this route when regulated data lacks a written review. Skip it when the only prompt copy is live production traffic. Skip it as a hiding place for where source code went.

The method sorts ordinary developer tasks, and it does nothing more. It is not a control for a certified security boundary. Do not stretch it into a compliance certificate.

Keep a local log of the decision, not of the prompt text. Store the route, the budget, the median, and the clock time. Review that file when a task felt slow or unsafe.

You want a boring pattern over a week of real edits. Most edits stay local, and the server shows up rarely. It shows up for secret-free jobs the laptop cannot finish.

If the server column is still an open question, read the current MonkeyCode docs. Confirm free model access and the free server option yourself. Send a byte only after the three clocks still agree.