The Pairing Hour That Split the Review Bot: Browser Prefilter, Server Verdict, One Token Budget

The review bot had been posting a one-paragraph risk summary on every pull request for eleven days. Nobody had asked, until that Thursday, which requests actually reached a model. A senior reviewer booked a pairing hour with one goal: find the boundary between work a laptop can do and work that needs a server.

The hour produced three dead ends and one decision. The dead ends are documented below because they cost most of the time. The decision is a small script and a table, both of which survive the removal of any vendor from the story.

Three questions written on the whiteboard

The senior did not open an editor for the first ten minutes. Three items went on the board, in this order:

1. Which bytes in a diff need a model at all?
2. Where does the model run: the reviewer's browser, a shared server, or CI?
3. What number makes the pair stop shipping?

Question three was the one that changed the architecture. Without it, every other answer turns into a preference.

Dead end 1: move the whole bot into the browser

The first proposal was the obvious one. Browsers can run inference locally, so the summary could be generated on the reviewer's machine with no server, no cold start, and no egress. It looked like free capacity.

It failed on state, not on speed. The bot needs to know which pull requests are already summarized, and a browser tab cannot hold that queue across machines. Re-implementing the queue in IndexedDB meant a second source of truth next to the webhook log. The pair also found that the local-file classifier had been duplicated into the browser bundle, so two copies of the same rules file could drift.

What survived from this dead end: the prefilter stays client-side, the model call does not. Rules that decide "this diff is a lockfile, skip it" are cheap, deterministic, and testable without a network.

Dead end 2: send every diff to the shared endpoint

The second attempt removed the classifier entirely and sent raw diffs to a hosted endpoint. The first generated lockfile in the sample set was 900 lines. The model spent its context describing dependency bumps that a regex already knew about.

Trimming is the fix, and it has to happen before the request, not after the bill:

// trim.mjs — deterministic, no network, unit-testable
const SKIP = /(?:^|\/)(?:package-lock\.json|pnpm-lock\.yaml|yarn\.lock|dist|vendor|__snapshots__)(?:\/|$)/;
const MAX_LINES_PER_FILE = 120;

export function trimDiff(raw) {
  return raw
    .split(/^diff --git /m)
    .filter((chunk) => chunk.trim().length > 0)
    .filter((chunk) => !SKIP.test(chunk.split("\n")[0]))
    .map((chunk) => chunk.split("\n").slice(0, MAX_LINES_PER_FILE).join("\n"))
    .join("\n");
}

Do not take a blog post's word for the savings. Measure on the repository you actually own:

git log --merges -n 30 --pretty=%H | while read sha; do
  git show --format= --no-color "$sha" | node -e '
    const { trimDiff } = await import("./trim.mjs");
    let raw = "";
    process.stdin.on("data", (c) => (raw += c));
    process.stdin.on("end", () => {
      const t = trimDiff(raw);
      console.log(raw.length, t.length);
    });
  '
done | awk '{ raw += $1; kept += $2 } END { printf "kept %.1f%% of diff bytes\n", 100 * kept / raw }'

If that percentage is high, the prefilter is not doing its job and the split below will not help.

Dead end 3: cache verdicts by prompt hash

The third idea was the most tempting and the most dangerous. Verdicts were cached on a hash of the prompt text so repeated diffs would not be re-sent. Two weeks later, a merged PR carried a summary written against a prompt template that had since changed.

The cache key was the bug. A verdict is a function of the diff, the prompt template, and the model that answered. Hashing only the diff means an identifier change upstream silently reuses stale text.

// The key that survived review: all three inputs, none optional.
import { createHash } from "node:crypto";

export function verdictKey({ modelId, templateHash, diffHash }) {
  return createHash("sha256")
    .update([modelId, templateHash, diffHash].join("\u0000"))
    .digest("hex");
}

Caching the prefilter decision is safe. Caching the verdict is only safe with all three inputs present, and the model identifier has to come from configuration rather than a default.

The decision kept: a split, plus a budget checked before enabling

The pair ended with a routing table and a script that refuses to enable the bot when the numbers do not fit.

Work	Runs where	Why	Fails when
File classification	Caller's machine	Zero model cost, pure function	Rules file drifts from repo layout
Diff trimming	Same process as classification	Deterministic and unit-testable	Hunk slicing cuts mid-context
Risk summary	Shared server + hosted model	Needs a queue and a stable webhook target	Tier latency or quota changes
Verdict cache	Server, keyed on three inputs	Avoids repeat spend	Model id not pinned in config
Merge decision	CI, never the model	A human stays accountable	Never delegate this step

The script below runs before the first real review. It sends two small requests and one representative diff, records cold and warm latency, projects tokens for a day of reviews, and exits non-zero if any measured number breaks the declared budget.

{
  "endpoint": "http://127.0.0.1:8787/v1/chat/completions",
  "model": "<id copied from the provider page, never defaulted>",
  "dailyReviews": 40,
  "maxDailyTokens": 250000,
  "maxColdStartMs": 4000
}

#!/usr/bin/env node
// split-budget.mjs — Node 20+, no dependencies.
import { readFile, writeFile } from "node:fs/promises";

const budget = JSON.parse(await readFile("budget.json", "utf8"));
const diff = await readFile("sample-diff.txt", "utf8");
const PROMPT = "Summarize the merge risk in this diff in one sentence.";

async function probe(label, text) {
  const start = performance.now();
  const res = await fetch(budget.endpoint, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      model: budget.model,
      messages: [
        { role: "system", content: PROMPT },
        { role: "user", content: text },
      ],
    }),
  });
  const ms = Math.round(performance.now() - start);
  if (!res.ok) return { label, ms, ok: false, status: res.status, tokens: 0 };
  const body = await res.json();
  const reported = body?.usage?.total_tokens;
  return {
    label,
    ms,
    ok: true,
    status: res.status,
    tokens: reported ?? Math.ceil((PROMPT.length + text.length) / 4),
    tokensSource: reported ? "reported" : "estimate",
  };
}

const probes = [
  await probe("cold-tiny", "x"),
  await probe("warm-tiny", "x"),
  await probe("warm-real", diff),
];

const cold = probes[0];
const real = probes[2];
const projected = real.tokens * budget.dailyReviews;
const failures = [];

if (probes.some((p) => !p.ok)) failures.push("probe_failed");
if (cold.ms > budget.maxColdStartMs) failures.push("cold_start_over_budget");
if (projected > budget.maxDailyTokens) failures.push("tokens_over_budget");

const ledger = {
  at: new Date().toISOString(),
  model: budget.model,
  probes,
  projectedTokensPerDay: projected,
  verdict: failures.length ? failures : ["pass"],
};
await writeFile("split-ledger.json", JSON.stringify(ledger, null, 2) + "\n");
console.log(JSON.stringify(ledger, null, 2));
process.exit(failures.length ? 1 : 0);

node split-budget.mjs; echo "exit=$?"

Illustrative output shape, not a measurement of anyone's tier:

{
  "model": "<pinned-id>",
  "probes": [
    { "label": "cold-tiny", "ms": 2110, "ok": true, "tokens": 41 },
    { "label": "warm-tiny", "ms": 380, "ok": true, "tokens": 41 },
    { "label": "warm-real", "ms": 1240, "ok": true, "tokens": 1850 }
  ],
  "projectedTokensPerDay": 74000,
  "verdict": ["pass"]
}

exit=0 is permission to enable the webhook. exit=1 names the reason and leaves the bot off. The gate is small enough that the senior read every line in the last six minutes of the hour — which was the point.

Where the free tier entered the session

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pair needed two things the laptop could not provide: a hosted model to answer the warm-real probe, and a machine with a public address to receive webhooks during the test. The operator's free tier supplied both — free model access and a free server option — and the project is open source, so the routing code can be read before it is trusted with a diff. The operator states that the tier includes 10 million tokens; treat that as a claim with a date on it, not a constant. The script above therefore reads limits from budget.json and never hardcodes a quota, so a change in tier terms shows up as tokens_over_budget rather than as a silent bill.

Limitations and who should skip this

• Low-volume repositories. Under roughly twenty pull requests a week, the gate and the routing table cost more attention than they save. Keep the prefilter, drop the split.
• Compliance-constrained teams. Sending diffs to a shared hosted tier may be disallowed outright. The trimming and classification stages still work alone.
• Unknown latency guarantees. A shared free endpoint is not a merge-path dependency. If cold-start behavior cannot be measured, keep the bot advisory and never blocking.
• Token estimates. Char-based fallbacks are labeled estimate in the ledger for a reason; they drift on code-heavy input. Prefer provider-reported usage when present.
• The gate measures, it does not guarantee. A pass means the numbers fit today's configuration, not that they will fit next quarter.

A checklist for the next pairing hour

1. Count how many diffs in the last thirty merges needed a model at all.
2. Pin the model identifier in config and fail loudly when it is missing.
3. Key any verdict cache on model id, template hash, and diff hash.
4. Run the budget script against the real endpoint before the webhook exists.
5. Recheck the ledger whenever the tier terms change.

If the point is to try the same split against a free endpoint, start with budget.json and two probe requests, then read the ledger before wiring a webhook — the project page carries the current free-tier terms, and today's numbers are the only ones worth planning around.