When Local CPU Loses to RTT

Local inference loses when the laptop CPU saturates. A measured remote hop can be faster then. This article shows a break-even harness, not a slogan.

Teams treat local-first as a moral rule. Morals alone do not shrink tail latency. A cheap remote path sometimes wins on warm links.

Think of the machine as a harbor. Local work is a dinghy you already own. A free remote hop is a scheduled ferry.

You pick the boat after you time the crossing. Guesswork here creates flaps and wasted tokens. A clock on the envelope beats a slogan on the wiki.

Agent loops multiply model calls across a workday. Each call still pays CPU, network, or both. Cheap generation does not make the hop free.

Recent architecture posts often skip this accounting. They assume the network is an infinite hallway. Hallways still have doors, locks, and weather.

Tight latency budgets punish chatty tool loops. Secret material punishes any hop that leaves disk. True offline work punishes every remote model call.

The same feature can fail all three tests. A public summarizer may still miss the budget. A private refactor must never board the ferry.

MonkeyCode provides free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat that remote path as overflow, not a default brain.

The method is simple and reproducible on one laptop. Time a local stub on your actual hardware. Time a remote complete on your real link.

Keep the faster path for that payload class. Do not copy sample timings into production configs. Hardware and networks drift across a single afternoon.

The harness below is a method, not a vendor benchmark. Numbers inside comments are labeled examples only. Replace the origin with your own measured endpoint.

#!/usr/bin/env node
/**
 * budget.mjs
 * Example harness. Unexecuted until OVERFLOW_URL is a real origin.
 * Local hashing is a CPU stand-in, not a language model.
 */
import { createHash } from "node:crypto";
import { appendFile, mkdir, readFile } from "node:fs/promises";
import { argv, env } from "node:process";

function parseArgs(args) {
  const out = { origin: env.OVERFLOW_URL || "", sample: "", trials: 5 };
  for (let i = 2; i < args.length; i += 1) {
    if (args[i] === "--origin") out.origin = args[++i];
    if (args[i] === "--sample") out.sample = args[++i];
    if (args[i] === "--trials") out.trials = Number(args[++i]);
    if (args[i] === "--secret") out.secret = true;
    if (args[i] === "--offline") out.offline = true;
  }
  return out;
}

function classify(text, flags) {
  const secret =
    Boolean(flags.secret) ||
    /api[_-]?key|BEGIN (RSA |OPENSSH )?PRIVATE/i.test(text);
  const bytes = Buffer.byteLength(text, "utf8");
  const offline = Boolean(flags.offline);
  const classId = secret ? "secret" : bytes > 4000 ? "public-heavy" : "public-light";
  return { secret, bytes, offline, classId };
}

function localWork(text) {
  const start = performance.now();
  let digest = text;
  for (let i = 0; i < 2500; i += 1) {
    digest = createHash("sha256").update(digest).digest("hex");
  }
  const tokens = text.split(/\s+/).filter(Boolean).length;
  return { ms: performance.now() - start, tokens, digest: digest.slice(0, 12) };
}

async function remoteWork(origin, text, envelope) {
  if (!origin) throw new Error("missing origin");
  const start = performance.now();
  const res = await fetch(origin, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      classId: envelope.classId,
      bytes: envelope.bytes,
      text,
    }),
  });
  const body = await res.text();
  return { ms: performance.now() - start, status: res.status, bytesOut: body.length };
}

function median(values) {
  const sorted = [...values].sort((a, b) => a - b);
  const mid = Math.floor(sorted.length / 2);
  if (sorted.length === 0) return Number.POSITIVE_INFINITY;
  return sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;
}

class Breaker {
  constructor({ limit = 3, coolMs = 60_000 } = {}) {
    this.limit = limit;
    this.coolMs = coolMs;
    this.breaches = 0;
    this.preferRemote = false;
    this.openedAt = 0;
  }

  observe({ localMs, remoteMs }) {
    if (remoteMs + 8 < localMs) this.breaches += 1;
    else this.breaches = Math.max(0, this.breaches - 1);

    if (this.breaches >= this.limit) {
      this.preferRemote = true;
      this.openedAt = Date.now();
    }
    if (this.preferRemote && Date.now() - this.openedAt > this.coolMs) {
      this.preferRemote = false;
      this.breaches = 0;
    }
    return this.preferRemote;
  }
}

async function enqueue(job) {
  await mkdir(".harbor", { recursive: true });
  await appendFile(".harbor/queue.jsonl", `${JSON.stringify(job)}\n`, "utf8");
}

async function main() {
  const opts = parseArgs(argv);
  const text = opts.sample
    ? await readFile(opts.sample, "utf8")
    : "public summary of a build log";
  const envelope = classify(text, opts);

  if (envelope.secret) {
    const local = localWork(text);
    console.log(JSON.stringify({ path: "local-forced", envelope, local }));
    return;
  }

  if (envelope.offline) {
    await enqueue({ at: Date.now(), envelope, bytes: envelope.bytes });
    console.log(JSON.stringify({ path: "queued", envelope }));
    return;
  }

  const localSamples = [];
  const remoteSamples = [];
  for (let i = 0; i < opts.trials; i += 1) {
    localSamples.push(localWork(text).ms);
    if (!opts.origin) continue;
    try {
      remoteSamples.push((await remoteWork(opts.origin, text, envelope)).ms);
    } catch {
      remoteSamples.push(Number.POSITIVE_INFINITY);
    }
  }

  const localMs = median(localSamples);
  const remoteMs = median(remoteSamples);
  const preferRemote = new Breaker().observe({ localMs, remoteMs });
  console.log(JSON.stringify({ envelope, localMs, remoteMs, preferRemote }, null, 2));
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

A local stub should resemble real CPU work. Token splitting is a decent public proxy. Hashing a buffer is another honest stand-in.

Avoid empty functions that return in one millisecond. They flatter local-first without teaching anything. The ferry always loses a fake race like that.

Remote timing must include DNS and TLS cost. A localhost mock lies about the crossing. Point the client at a real origin when you can.

Secret payloads must never ride the remote ferry. Mark them in the envelope before timing starts. The breaker should refuse the hop, not retry it.

Offline mode is a closed harbor mouth. Queue only public jobs on local disk. Flush them when the link returns under budget.

Hysteresis stops the fleet from flapping boats. One slow sample should not flip the default. Require several breaches before you leave local.

// hysteresis.mjs — example only, same clock, slower hands
export function shouldFlip(samples, { marginMs = 8, need = 3 } = {}) {
  let remoteWins = 0;
  for (const row of samples) {
    if (row.remoteMs + marginMs < row.localMs) remoteWins += 1;
    else remoteWins = Math.max(0, remoteWins - 1);
  }
  return remoteWins >= need;
}

Start with a public sample under four kilobytes. Run five trials on an idle core. Then rerun on the same origin after a pause.

chmod +x budget.mjs
node budget.mjs --origin "$OVERFLOW_URL" --sample ./public.txt --trials 5
node budget.mjs --origin "$OVERFLOW_URL" --sample ./public.txt --trials 5
node budget.mjs --sample ./private.env --secret
node budget.mjs --sample ./public.txt --offline

Record both samples with the class name attached. Compare medians, not a single lucky trip. Lucky trips make a poor routing law.

Walk a hypothetical afternoon, labeled as fiction. A two thousand character prompt remains public text. Local CPU then spends one hundred sixty milliseconds.

The remote ferry returns in one hundred ten milliseconds. Those figures are story numbers, not measurements. Your laptop will disagree by a wide margin.

The breaker would prefer remote for that class. The same prompt on train Wi-Fi might take nine hundred milliseconds. Local wins again without a code change.

A secret environment chunk stays in the dinghy. The remote timer never runs for that class. The offline queue also skips it during flush.

Payload size changes the race in quiet ways. Tiny completions often die in TLS handshake cost. Large embeddings often die on laptop cores instead.

That is why a single global default fails. Class the envelope, then time that class. Store the winner with a measured expiry.

{
  "classId": "public-heavy",
  "localMs": 160,
  "remoteMs": 110,
  "preferRemote": true,
  "expiresAt": "labeled-example-not-a-measurement",
  "notes": "Story numbers. Replace after five idle trials."
}

Run the harness from a cold shell first. Then run it again on a warm DNS cache. The gap between those runs is the lie.

Cheap model calls raise volume, not wisdom. Volume makes the wrong boat expensive at scale. A wrong default taxes every later agent hop.

A free overflow origin can absorb public bursts. It cannot absorb a secret, a policy, or a dead link. Those remain local problems with strictly local answers.

Flush is not a second brain either. It is a delayed ferry for cargo already cleared. Uncleared cargo waits on disk without a ticket.

// flush.mjs — example queue drain, still not a model client
import { readFile, writeFile } from "node:fs/promises";

export async function flushPublic(origin, now = Date.now()) {
  const raw = await readFile(".harbor/queue.jsonl", "utf8").catch(() => "");
  const rows = raw.split("\n").filter(Boolean).map((line) => JSON.parse(line));
  const kept = [];
  for (const job of rows) {
    if (job.envelope.secret) {
      kept.push(job);
      continue;
    }
    if (now - job.at > 86_400_000) continue;
    await fetch(origin, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify(job),
    });
  }
  await writeFile(".harbor/queue.jsonl", kept.map((j) => JSON.stringify(j)).join("\n"), "utf8");
}

Limitations matter more than the happy path. This harness does not rank model quality. It does not prove any vendor uptime claim.

It also does not replace a secret scanner. Classification is only as honest as the caller. A missed flag will ship text you meant to keep.

The method fails when every prompt contains secrets. Air-gapped shops should not open the harbor. Teams without clocks cannot run a break-even test.

Quality-sensitive work needs evals beside this timer. Fast and wrong is still wrong for reviews. The ferry is a latency tool, not a judge.

A busy shared laptop also distorts the race. Background builds will steal the local core. The dinghy looks slower than it is.

So pin the test to an idle machine. Close the extra browsers before you measure. Otherwise you time contention instead of completion.

Expiry must stay short on mobile links. A cafe winner can be a tunnel loser. Re-time after the network class changes.

A free remote origin wins on public bulky jobs. It wins when several developers share one warm path. It loses when the link is the product risk.

Local still wins for secrets, silence, and tight loops. Measure first, and then pick the boat. Keep the dinghy for cargo that cannot sink.

Public jobs may need an overflow origin. Time MonkeyCode's free server against this harness.