Secrets Stay Local, Compute Can Travel

Route by data class, not by vendor. Secrets and repo material stay on disk. Bulk text transformation can take the hop. That one rule settles most local-first arguments before they start.

Last week the DEV feed argued about whether AI writes better code than people. Useful heat, wrong question for me. My daily decision is narrower: this prompt, this laptop, this link — which side runs it? That question has a stopwatch answer, not an opinion.

Measure the hop before you trust it

A local model call pays almost no network cost. It pays in CPU contention instead. Your editor, test runner, and containers share the same cores. A remote call has the opposite profile: fixed overhead first, elastic capacity after. Small payloads lose that trade every time.

So measure both sides with the same harness. Keep it dependency-free and boring.

// route.js — measure first, route second. Node 18+, no dependencies.
const LOCAL  = process.env.LOCAL_URL  ?? "http://127.0.0.1:11434/v1/chat/completions";
const REMOTE = process.env.REMOTE_URL ?? ""; // fill from docs you have verified yourself
const MODEL  = process.env.MODEL ?? "local-default";

const SECRET_PATTERNS = [
  /-----BEGIN [A-Z ]*PRIVATE KEY-----/,
  /\b(sk|ghp|glpat)-[A-Za-z0-9_-]{16,}\b/,
  /\bAKIA[0-9A-Z]{16}\b/,
];

function classify(prompt) {
  // Fail closed: anything that looks like a credential stays local.
  return SECRET_PATTERNS.some((re) => re.test(prompt)) ? "secret" : "plain";
}

async function timed(url, prompt, timeoutMs) {
  const ctl = new AbortController();
  const timer = setTimeout(() => ctl.abort(), timeoutMs);
  const start = process.hrtime.bigint();
  try {
    const res = await fetch(url, {
      method: "POST",
      signal: ctl.signal,
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ model: MODEL, messages: [{ role: "user", content: prompt }] }),
    });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    await res.text();
    return Number(process.hrtime.bigint() - start) / 1e6;
  } finally {
    clearTimeout(timer);
  }
}

function route({ kind, bytes, localMs, remoteMs }) {
  if (kind === "secret") return { side: "local", why: "data class wins" };
  if (!REMOTE) return { side: "local", why: "no remote configured" };
  if (bytes < 4096) return { side: "local", why: "hop cost dominates" };
  return remoteMs * 1.4 < localMs
    ? { side: "remote", why: "remote wins by more than 40%" }
    : { side: "local", why: "margin too thin to risk" };
}

Run it against a fixed prompt set with a repeat count, then read the medians.

LOCAL_URL=http://127.0.0.1:11434/v1/chat/completions \
REMOTE_URL=https://your-endpoint.example/v1/chat/completions \
node route.js --prompts bench/*.md --repeat 5

The 1.4 multiplier is not decoration. A remote win under forty percent disappears when your connection hiccups. Local never retries because the train entered a tunnel.

The policy function is the whole product

Three inputs decide the route: data class, payload size, measured time. Everything else is commentary. Local-first is a default setting, never a religion.

Condition	Route	Reason
Credential pattern matched	local	fail closed, no exceptions
Payload under ~4 KB	local	hop overhead exceeds token cost
Offline or captive wifi	local	availability, not speed
50 KB refactor, remote measured 3x faster	remote	elastic capacity earns the hop
Remote ahead by less than 40%	local	noise and retries eat the margin
Regulated data, no signed agreement	local	policy, not latency

Cache those measurements on disk instead of memory. A cold editor start should not re-benchmark anything. Write route.json with the last medians plus a timestamp. Re-measure when the link changes, or on a fixed weekly schedule.

Where a free server changes the calculus

The harness needs a REMOTE_URL worth testing. Standing up your own GPU box is one answer, but it is slow to provision and idle most of the day. An idle box is a cost with no measurement attached.

MonkeyCode is an open-source project that documents free model access and a free server option, so the remote branch of that policy function has somewhere to point. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am deliberately not restating a model list, hardware spec, quota, or retention policy here, because those terms change. Read the project's own docs and copy the current limits into your runbook before you route real work.

That restriction matters for one specific reason. The free server is the cheapest way to fill REMOTE_URL and obtain a real remoteMs sample. Do not let it become the default route on day one. A free tier is a measuring instrument first and a production dependency second.

What this approach does not solve

Numbers from my laptop will not transfer to your network. Measure again, and measure with the prompt sizes you actually send. Long-context jobs change shape, because uploading 200 KB of source can cost more than the inference itself. Test that case explicitly, since the break-even rarely sits where intuition places it.

Skip this design if you handle regulated data without a processor agreement. Skip it if you need a per-call audit trail and cannot log locally. Skip it if you work offline by habit, because the remote branch never fires. And skip it if you cannot run the stopwatch — an unrouted guess is just a guess with extra steps.

The rule stays small on purpose. Secrets stay local, compute can travel, and the stopwatch decides. If you want to test the hop, point REMOTE_URL at the project's free server once and run the harness on your own prompts.