DEV Community: Quinn Sun

A Pairing Hour on Retries: The Failure Taxonomy We Kept and the Three Loops We Deleted

Quinn Sun — Mon, 26 Oct 2026 11:23:48 +0000

A review bot retried a single file 41 times overnight. The model was fine. The loop was the bug. A senior engineer joined for one hour, and the hour turned into an argument about what deserves a retry rather than how long the backoff should be.

What follows is the questions asked, the three dead ends, and the one decision that survived. The 41-attempt story is the motivating scenario; every number in the tables below comes from a local fixture you can run in a few seconds, not from production telemetry.

The question that reframed the hour

The senior did not open with backoff math. The first question was narrower: which of these failures are actually the same failure?

A retry loop is a claim about failure equivalence. Every for loop with a catch inside says, implicitly, that these errors deserve the same response. Most of them lump a 429, a schema mismatch, and a model refusal into one bucket labelled "transient" — and then spend budget proving otherwise.

The taxonomy we kept

Class	Signal	Retry?	Attempt cap	Handling
`transport`	`ECONNRESET`, `ENOTFOUND`, 5xx	yes	4	exponential backoff + jitter
`throttle`	429 with `Retry-After`	yes	6	honour the header, no jitter guessing
`schema`	200 with invalid body shape	yes, once	2	stricter prompt on the second attempt
`refusal`	model declines the request	no	1	re-scope the input
`budget`	quota or limit exhausted	no	1	degrade and alert

Two rules fell out of the table. Rate limiting is a scheduling signal, not an error, so Retry-After is obeyed rather than jittered around (see RFC 9110 on 429). And anything unclassified fails closed: no policy entry means no retry.

Dead end 1: one wrapper to retry them all

The first suggestion was the obvious one, and it is the one most codebases ship.

// The loop we deleted
async function withRetry(fn, attempts = 5) {
  for (let i = 1; i <= attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === attempts) throw err;
      await sleep(2 ** i * 100);
    }
  }
}

This wrapper retried the whole job, diff parsing included. A malformed diff coming from our own parser looked exactly like a flaky network call, so a deterministic bug consumed the entire attempt budget.

Forty-one attempts produced zero new information. That is the signature of a loop that should have been a crash: the error repeated, but the state never advanced.

Dead end 2: retry at the HTTP client

Moving the policy down into the SDK felt cleaner. Except an HTTP client only sees status codes.

A 200 whose body fails your schema is not a transport failure and never will be. The client returns it happily, the job fails later, and the retry policy has already been bypassed.

// A 200 is not a success until the shape is right
const parsed = CommentSchema.safeParse(body);
if (!parsed.success) {
  throw Object.assign(new Error('bad shape'), {
    name: 'SchemaError',
    issues: parsed.error.issues,
  });
}

The lesson is not "wrap your SDK". It is that the failure class has to be decided by the layer that can actually recognise it.

Dead end 3: dedupe on a payload hash

Hashing the request body looks like idempotency and is not. Two different prompt versions produce different hashes, so the same file gets reviewed twice. The same file re-uploaded with identical bytes produces the same hash, so a genuinely new run gets silently dropped.

Worse, a hash gives you nowhere to write what happened. There is no attempt history, no last known outcome, and no way to answer "did attempt 3 succeed" without grepping logs.

The senior's summary of this branch: a hash identifies the request, not the work.

The decision: retry the job, not the call

The thing that survived the hour was a single idempotency tuple plus an append-only ledger. Identity is the tuple; history is the ledger; the retry policy reads the ledger before it reads any error.

Key and ledger

// ledger.js
import { appendFileSync, readFileSync, existsSync } from 'node:fs';

export function idempotencyKey(job) {
  return [
    job.repo,
    job.commit,
    job.path,
    job.promptVersion,
    job.modelAlias,
  ].join('|');
}

export function attemptsFor(ledgerPath, key) {
  if (!existsSync(ledgerPath)) return [];
  return readFileSync(ledgerPath, 'utf8')
    .split('\n')
    .filter(Boolean)
    .map((line) => JSON.parse(line))
    .filter((row) => row.key === key);
}

export function record(ledgerPath, row) {
  appendFileSync(ledgerPath, JSON.stringify(row) + '\n');
}

Five fields, no more. promptVersion and modelAlias belong in the key because changing either one changes the work, and reviewers who forget that end up deduping across a prompt rewrite.

Classifier and policy

// retry.js
export const POLICY = {
  transport: { maxAttempts: 4, retryable: true },
  throttle:  { maxAttempts: 6, retryable: true },
  schema:    { maxAttempts: 2, retryable: true },
  refusal:   { maxAttempts: 1, retryable: false },
  budget:    { maxAttempts: 1, retryable: false },
};

export function classifyAttempt(err) {
  if (err?.code === 'ECONNRESET' || err?.code === 'ENOTFOUND') return 'transport';
  if (err?.status >= 500) return 'transport';
  if (err?.status === 429) return 'throttle';
  if (err?.name === 'SchemaError') return 'schema';
  if (err?.name === 'RefusalError') return 'refusal';
  if (err?.name === 'BudgetError') return 'budget';
  return 'unknown';
}

export function shouldRetry(cls, attemptsSoFar) {
  const policy = POLICY[cls];
  if (!policy || !policy.retryable) return false;
  return attemptsSoFar < policy.maxAttempts;
}

unknown has no policy entry, so it returns false. Failing closed here is deliberate: an unrecognised error is a missing branch in your own code, not a network hiccup.

The runner

// runJob.js
import { attemptsFor, idempotencyKey, record } from './ledger.js';
import { classifyAttempt, shouldRetry } from './retry.js';

export async function runJob({ job, call, ledgerPath, sleep }) {
  const key = idempotencyKey(job);
  const prior = attemptsFor(ledgerPath, key);
  const done = prior.find((row) => row.outcome === 'ok');
  if (done) return done.result;

  let attempts = prior.length;

  while (true) {
    attempts += 1;
    try {
      const result = await call(job);
      record(ledgerPath, { key, attempt: attempts, outcome: 'ok', result });
      return result;
    } catch (err) {
      const cls = classifyAttempt(err);
      record(ledgerPath, {
        key,
        attempt: attempts,
        cls,
        outcome: 'error',
        at: new Date().toISOString(),
      });
      if (!shouldRetry(cls, attempts)) throw err;
      await sleep(cls === 'throttle' ? (err.retryAfterMs ?? 1000) : 2 ** attempts * 100);
    }
  }
}

Restarting after a crash now resumes from the ledger instead of from attempt zero. That single property is why the process can be recycled at any time without burning budget.

Tests you can run now

// retry.test.js
import test from 'node:test';
import assert from 'node:assert/strict';
import { classifyAttempt, shouldRetry } from './retry.js';

test('transport retries to the cap, refusal never retries', () => {
  const reset = Object.assign(new Error('reset'), { code: 'ECONNRESET' });
  assert.equal(classifyAttempt(reset), 'transport');
  assert.equal(shouldRetry('transport', 3), true);
  assert.equal(shouldRetry('transport', 4), false);
  assert.equal(shouldRetry('refusal', 0), false);
});

test('unknown errors fail closed', () => {
  assert.equal(classifyAttempt(new Error('???')), 'unknown');
  assert.equal(shouldRetry('unknown', 0), false);
});

node --test retry.test.js

The Node test runner is enough; no framework required.

Fault-injection matrix

Injected fault	Expected attempts	Final state
3× `ECONNRESET`, then 200	4	`ok`
6× 429 with `Retry-After`	6	thrown `throttle`
200 with broken schema, twice	2	thrown `schema`
model refusal	1	thrown `refusal`
unrecognised error	1	thrown `unknown`

Run the matrix against a stub call that reads a scripted fault list. If any row produces more attempts than listed, the policy has a hole.

Where the free tier actually helped

The harness above needs iteration, not scale: rerun the matrix, bump promptVersion, watch which rows change. That is exactly the workload where MonkeyCode's free model access and free server option were used in this session — the fault matrix ran against the hosted model while the job runner sat on the free server, with the ledger written to disk rather than held in memory.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operative assumption is the honest one: on any free-tier host, treat the process as recyclable and keep state outside it. That assumption is what made the ledger decision load-bearing instead of decorative.

Limitations and who should skip this

The ledger is a local file. Two runners on the same key will race; move to a real queue with a uniqueness constraint before you scale out.
Attempt caps are guesses until you have real failure distributions. Four and six are starting points, not constants.
Jobs that must complete inside a tight latency SLO will find one attempt too many here. This design trades latency for budget discipline.
Code that cannot leave your network, or that needs an audit trail beyond your own storage, should not use hosted inference at all.

The closing checklist

Classify every failure your job can produce, including your own parser bugs.
Put identity in one tuple and write it down before the first call.
Fail closed on unknown.
Run the fault matrix before you trust the retry cap.

If you want to reproduce the fixture, start with node --test retry.test.js and the matrix above; the free tier is enough to run it end to end.

The Pairing Hour That Replaced a Webhook Queue With 60 Lines of Admission Control

Quinn Sun — Tue, 20 Oct 2026 09:27:24 +0000

A pull-request review bot was ready to ship. The plan on the whiteboard was the usual one: webhook in, job into a queue, worker out, model call, comment back on the PR. It had been drawn in ten minutes and it looked finished.

The pairing hour that followed did not add features. It removed a queue, added a deadline, and produced a small module that the team could reason about at 3 a.m. What follows is the questions the senior asked, the two designs that died, and the one decision that survived contact with a free-tier endpoint.

The constraint nobody wrote down

The bot was going to run against a free model endpoint on a free server option. Both were operator-supplied availability claims, not guarantees: no SLA, no reserved capacity, no promise that a slow minute stays a slow minute.

That single sentence changes the shape of the system. A queue is a promise that every accepted job eventually runs. On a free tier, that promise is the thing you cannot keep.

Question 1: what happens when forty PRs land in ninety seconds?

The senior did not ask about model quality. The first question was about arrival rate versus service rate.

A monorepo morning looks like this: a dependency bump opens forty PRs, each firing opened, synchronize, and a couple of labeled events. The handler accepts all of them because accepting is cheap. Then the worker discovers that one review takes eight to forty seconds.

Queue depth grows faster than it drains.
Comments arrive on code that has already been amended twice.
The slowest PRs get reviews last, which is exactly backwards.

Dead end A: a priority queue. Sort by PR age so the oldest gets served first. It solves ordering and solves nothing else — depth still grows without bound, and the bot still comments on stale diffs. Priority is a scheduling answer to a capacity question.

Dead end B: per-PR concurrency of four. Let each PR have up to four parallel model calls so a big diff can be chunked. On a free endpoint this multiplies the failure surface by four and makes the timeout story unreadable. When a review half-succeeds, nobody can say which chunk was dropped.

Question 2: what does the endpoint do when it is slow rather than down?

Silence is the hard failure mode. A 500 is easy: log it, retry once, move on. A request that hangs for ninety seconds and then returns a truncated body is the one that corrupts the review.

So the design needs a deadline that is enforced on the caller's side, not hoped for on the server's side. That means an AbortController with a timer, and a policy for what to do when the timer wins.

Question 3: can you reconstruct a failure from the log alone?

This one produced the artifact. The senior wanted three fields on every decision: the idempotency key, the outcome, and the reason. Not a full trace — three fields, written at the moment the decision is made.

The artifact: an admission gate, not a queue

The kept design is roughly sixty lines. It takes a job only if a token bucket allows it, refuses duplicates for the same commit, and aborts at the deadline. The excerpt below is written for this article and has not been benchmarked against any particular endpoint; treat the constants as knobs, not measurements.

// gate.js — admission control in front of a free-tier model call
const log = [];
const record = (entry) => log.push({ t: Date.now(), ...entry });

export function snapshot() {
  return { log: [...log], inFlight: inFlight.size, tokens: bucket.tokens };
}

const bucket = { tokens: 8, capacity: 8, refillPerSec: 0.25, last: Date.now() };

function take() {
  const now = Date.now();
  bucket.tokens = Math.min(
    bucket.capacity,
    bucket.tokens + ((now - bucket.last) / 1000) * bucket.refillPerSec
  );
  bucket.last = now;
  if (bucket.tokens < 1) return false;
  bucket.tokens -= 1;
  return true;
}

const inFlight = new Map(); // key -> AbortController

export function submit({ key, run, deadlineMs = 45_000, onDrop = () => {} }) {
  if (inFlight.has(key)) {
    record({ key, outcome: "deduped", reason: "same-key-in-flight" });
    return { status: "deduped", key };
  }

  if (!take()) {
    record({ key, outcome: "dropped", reason: "bucket-empty" });
    onDrop(key);
    return { status: "dropped", key };
  }

  const ac = new AbortController();
  inFlight.set(key, ac);
  const timer = setTimeout(() => ac.abort(new Error("deadline")), deadlineMs);
  const done = (outcome, detail) => record({ key, outcome, detail });

  run(ac.signal)
    .then((result) => done("ok", { bytes: String(result ?? "").length }))
    .catch((err) =>
      done(err?.name === "AbortError" ? "timeout" : "error", { message: err?.message })
    )
    .finally(() => {
      clearTimeout(timer);
      inFlight.delete(key);
    });

  return { status: "accepted", key };
}

The key is repo#pr#headSha. That choice is the whole trick: a force-push creates a new key, so the new commit is admitted while the old run is already gone. A redelivered webhook creates the same key, so GitHub's at-least-once delivery costs nothing.

Reproducing the burst without a network

The point of the probe is not to measure someone else's endpoint. It is to prove the gate's own behaviour — dedupe, drop, timeout — under a burst you can replay.

// probe.mjs — burst test for the gate, no network required
import { submit, snapshot } from "./gate.js";

// Placeholder latency profile. Replace with p50/p95 you measured yourself.
const latency = () => 4000 + Math.random() * 26000;
const sleep = (ms, signal) =>
  new Promise((res, rej) => {
    const id = setTimeout(res, ms);
    signal?.addEventListener("abort", () => {
      clearTimeout(id);
      rej(Object.assign(new Error("aborted"), { name: "AbortError" }));
    });
  });

const totals = { accepted: 0, deduped: 0, dropped: 0 };

for (let i = 0; i < 90; i++) {
  // 40 PRs, some force-pushed while a review is still running
  const pr = i % 40;
  const sha = `s${Math.floor(i / 40)}`;
  const key = `repo#${pr}#${sha}`;
  const r = submit({
    key,
    deadlineMs: 45_000,
    run: (signal) => sleep(latency(), signal),
  });
  totals[r.status]++;
  await sleep(33); // ~90 events over 3s
}

await sleep(50_000);
const s = snapshot();
const byOutcome = s.log.reduce((a, e) => ((a[e.outcome] = (a[e.outcome] ?? 0) + 1), a), {});
console.log({ totals, byOutcome, stillInFlight: s.inFlight });

Run it with node probe.mjs. What you are looking for is boring: no entry in byOutcome that you cannot explain, and stillInFlight back to zero. If a burst leaves work in flight past the deadline plus a margin, the timer is not the only thing holding the job.

The decision table we kept

Observed on the probe	Decision
Duplicate keys dominate the burst	Keep dedupe; it is cheaper than retry logic
Drops cluster at burst start	Raise `capacity`, keep `refillPerSec` low
Drops spread evenly across the burst	The service rate is the problem; reduce chunk size
Timeouts above ~5%	Lower `deadlineMs` and comment "partial review" instead
Timeouts with tiny payloads	Suspect the endpoint, not the diff
`stillInFlight` never returns to zero	You have a leak, not a capacity issue

A drop is not free. Each one is a PR without a review, so onDrop posts a single comment with the exact command to re-run. That converts an invisible failure into a human decision, which is the only kind of failure a small team can act on that week.

Where the free tier actually enters the design

The reason this bot never grew a queue is that a free model endpoint plus a free server option makes capacity an external, unknown variable. MonkeyCode is the project this bot was being wired to — its operator describes free model access and a free server option, which is exactly the setting that punishes unbounded queues. I am not repeating quotas, hardware, or limits here: those numbers move, and the project's current documentation is the only place they should be read before you size anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two consequences shaped the code. First, the deadline had to be ours. Second, the token bucket had to be small enough that a bad minute degrades into drops rather than a five-hour backlog. Neither is a product feature; both are what a free tier forces you to write.

Limitations and who should not use this

No guaranteed latency. The gate bounds queue growth, not response time. If a comment must land within thirty seconds, this design will disappoint you.
Drops are real loss. Any team that needs every commit reviewed needs a durable queue and a paid capacity plan, not a token bucket.
Single process only. The inFlight map is in memory. Two replicas double both the bucket and the dedupe window.
Not for compliance work. If reviews are an audit artifact, you need storage you control and retention rules; a log array in RAM is not that.
Not for monorepos with thousand-file diffs. Chunking multiplies calls, and this gate is deliberately hostile to parallelism.

What the senior kept

The queue was four hundred lines: retries, backoff, dead-letter storage, a dashboard nobody opened. The gate is sixty, plus a probe you can run in a minute and a log you can read without tooling.

That was the pairing hour's real output — not a better review bot, but a system that fails in one visible way at a time. If you want to try the same gate against a free endpoint, start with the project's docs, then run the probe before you run the bot.

The Pairing Hour That Split the Review Bot: Browser Prefilter, Server Verdict, One Token Budget

Quinn Sun — Mon, 19 Oct 2026 08:11:06 +0000

The review bot had been posting a one-paragraph risk summary on every pull request for eleven days. Nobody had asked, until that Thursday, which requests actually reached a model. A senior reviewer booked a pairing hour with one goal: find the boundary between work a laptop can do and work that needs a server.

The hour produced three dead ends and one decision. The dead ends are documented below because they cost most of the time. The decision is a small script and a table, both of which survive the removal of any vendor from the story.

Three questions written on the whiteboard

The senior did not open an editor for the first ten minutes. Three items went on the board, in this order:

Which bytes in a diff need a model at all?
Where does the model run: the reviewer's browser, a shared server, or CI?
What number makes the pair stop shipping?

Question three was the one that changed the architecture. Without it, every other answer turns into a preference.

Dead end 1: move the whole bot into the browser

The first proposal was the obvious one. Browsers can run inference locally, so the summary could be generated on the reviewer's machine with no server, no cold start, and no egress. It looked like free capacity.

It failed on state, not on speed. The bot needs to know which pull requests are already summarized, and a browser tab cannot hold that queue across machines. Re-implementing the queue in IndexedDB meant a second source of truth next to the webhook log. The pair also found that the local-file classifier had been duplicated into the browser bundle, so two copies of the same rules file could drift.

What survived from this dead end: the prefilter stays client-side, the model call does not. Rules that decide "this diff is a lockfile, skip it" are cheap, deterministic, and testable without a network.

Dead end 2: send every diff to the shared endpoint

The second attempt removed the classifier entirely and sent raw diffs to a hosted endpoint. The first generated lockfile in the sample set was 900 lines. The model spent its context describing dependency bumps that a regex already knew about.

Trimming is the fix, and it has to happen before the request, not after the bill:

// trim.mjs — deterministic, no network, unit-testable
const SKIP = /(?:^|\/)(?:package-lock\.json|pnpm-lock\.yaml|yarn\.lock|dist|vendor|__snapshots__)(?:\/|$)/;
const MAX_LINES_PER_FILE = 120;

export function trimDiff(raw) {
  return raw
    .split(/^diff --git /m)
    .filter((chunk) => chunk.trim().length > 0)
    .filter((chunk) => !SKIP.test(chunk.split("\n")[0]))
    .map((chunk) => chunk.split("\n").slice(0, MAX_LINES_PER_FILE).join("\n"))
    .join("\n");
}

Do not take a blog post's word for the savings. Measure on the repository you actually own:

git log --merges -n 30 --pretty=%H | while read sha; do
  git show --format= --no-color "$sha" | node -e '
    const { trimDiff } = await import("./trim.mjs");
    let raw = "";
    process.stdin.on("data", (c) => (raw += c));
    process.stdin.on("end", () => {
      const t = trimDiff(raw);
      console.log(raw.length, t.length);
    });
  '
done | awk '{ raw += $1; kept += $2 } END { printf "kept %.1f%% of diff bytes\n", 100 * kept / raw }'

If that percentage is high, the prefilter is not doing its job and the split below will not help.

Dead end 3: cache verdicts by prompt hash

The third idea was the most tempting and the most dangerous. Verdicts were cached on a hash of the prompt text so repeated diffs would not be re-sent. Two weeks later, a merged PR carried a summary written against a prompt template that had since changed.

The cache key was the bug. A verdict is a function of the diff, the prompt template, and the model that answered. Hashing only the diff means an identifier change upstream silently reuses stale text.

// The key that survived review: all three inputs, none optional.
import { createHash } from "node:crypto";

export function verdictKey({ modelId, templateHash, diffHash }) {
  return createHash("sha256")
    .update([modelId, templateHash, diffHash].join("\u0000"))
    .digest("hex");
}

Caching the prefilter decision is safe. Caching the verdict is only safe with all three inputs present, and the model identifier has to come from configuration rather than a default.

The decision kept: a split, plus a budget checked before enabling

The pair ended with a routing table and a script that refuses to enable the bot when the numbers do not fit.

Work	Runs where	Why	Fails when
File classification	Caller's machine	Zero model cost, pure function	Rules file drifts from repo layout
Diff trimming	Same process as classification	Deterministic and unit-testable	Hunk slicing cuts mid-context
Risk summary	Shared server + hosted model	Needs a queue and a stable webhook target	Tier latency or quota changes
Verdict cache	Server, keyed on three inputs	Avoids repeat spend	Model id not pinned in config
Merge decision	CI, never the model	A human stays accountable	Never delegate this step

The script below runs before the first real review. It sends two small requests and one representative diff, records cold and warm latency, projects tokens for a day of reviews, and exits non-zero if any measured number breaks the declared budget.

{
  "endpoint": "http://127.0.0.1:8787/v1/chat/completions",
  "model": "<id copied from the provider page, never defaulted>",
  "dailyReviews": 40,
  "maxDailyTokens": 250000,
  "maxColdStartMs": 4000
}

#!/usr/bin/env node
// split-budget.mjs — Node 20+, no dependencies.
import { readFile, writeFile } from "node:fs/promises";

const budget = JSON.parse(await readFile("budget.json", "utf8"));
const diff = await readFile("sample-diff.txt", "utf8");
const PROMPT = "Summarize the merge risk in this diff in one sentence.";

async function probe(label, text) {
  const start = performance.now();
  const res = await fetch(budget.endpoint, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      model: budget.model,
      messages: [
        { role: "system", content: PROMPT },
        { role: "user", content: text },
      ],
    }),
  });
  const ms = Math.round(performance.now() - start);
  if (!res.ok) return { label, ms, ok: false, status: res.status, tokens: 0 };
  const body = await res.json();
  const reported = body?.usage?.total_tokens;
  return {
    label,
    ms,
    ok: true,
    status: res.status,
    tokens: reported ?? Math.ceil((PROMPT.length + text.length) / 4),
    tokensSource: reported ? "reported" : "estimate",
  };
}

const probes = [
  await probe("cold-tiny", "x"),
  await probe("warm-tiny", "x"),
  await probe("warm-real", diff),
];

const cold = probes[0];
const real = probes[2];
const projected = real.tokens * budget.dailyReviews;
const failures = [];

if (probes.some((p) => !p.ok)) failures.push("probe_failed");
if (cold.ms > budget.maxColdStartMs) failures.push("cold_start_over_budget");
if (projected > budget.maxDailyTokens) failures.push("tokens_over_budget");

const ledger = {
  at: new Date().toISOString(),
  model: budget.model,
  probes,
  projectedTokensPerDay: projected,
  verdict: failures.length ? failures : ["pass"],
};
await writeFile("split-ledger.json", JSON.stringify(ledger, null, 2) + "\n");
console.log(JSON.stringify(ledger, null, 2));
process.exit(failures.length ? 1 : 0);

node split-budget.mjs; echo "exit=$?"

Illustrative output shape, not a measurement of anyone's tier:

{
  "model": "<pinned-id>",
  "probes": [
    { "label": "cold-tiny", "ms": 2110, "ok": true, "tokens": 41 },
    { "label": "warm-tiny", "ms": 380, "ok": true, "tokens": 41 },
    { "label": "warm-real", "ms": 1240, "ok": true, "tokens": 1850 }
  ],
  "projectedTokensPerDay": 74000,
  "verdict": ["pass"]
}

exit=0 is permission to enable the webhook. exit=1 names the reason and leaves the bot off. The gate is small enough that the senior read every line in the last six minutes of the hour — which was the point.

Where the free tier entered the session

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pair needed two things the laptop could not provide: a hosted model to answer the warm-real probe, and a machine with a public address to receive webhooks during the test. The operator's free tier supplied both — free model access and a free server option — and the project is open source, so the routing code can be read before it is trusted with a diff. The operator states that the tier includes 10 million tokens; treat that as a claim with a date on it, not a constant. The script above therefore reads limits from budget.json and never hardcodes a quota, so a change in tier terms shows up as tokens_over_budget rather than as a silent bill.

Limitations and who should skip this

Low-volume repositories. Under roughly twenty pull requests a week, the gate and the routing table cost more attention than they save. Keep the prefilter, drop the split.
Compliance-constrained teams. Sending diffs to a shared hosted tier may be disallowed outright. The trimming and classification stages still work alone.
Unknown latency guarantees. A shared free endpoint is not a merge-path dependency. If cold-start behavior cannot be measured, keep the bot advisory and never blocking.
Token estimates. Char-based fallbacks are labeled estimate in the ledger for a reason; they drift on code-heavy input. Prefer provider-reported usage when present.
The gate measures, it does not guarantee. A pass means the numbers fit today's configuration, not that they will fit next quarter.

A checklist for the next pairing hour

Count how many diffs in the last thirty merges needed a model at all.
Pin the model identifier in config and fail loudly when it is missing.
Key any verdict cache on model id, template hash, and diff hash.
Run the budget script against the real endpoint before the webhook exists.
Recheck the ledger whenever the tier terms change.

If the point is to try the same split against a free endpoint, start with budget.json and two probe requests, then read the ledger before wiring a webhook — the project page carries the current free-tier terms, and today's numbers are the only ones worth planning around.

A Pairing Hour on Token Budgets: Three Dead Ends Before We Kept the Usage Ledger

Quinn Sun — Sun, 18 Oct 2026 06:55:20 +0000

Free tiers invite a specific kind of optimism: the first hundred calls cost nothing, so nobody measures them. Then a test suite gets pointed at the endpoint, the allowance disappears, and the team cannot say which calls ate it. The pairing hour described below is about closing that gap before it opens.

The session is a composite of the review a senior engineer typically runs on a "we tried the free tier" pull request. It is not a transcript, and the figures are placeholders. The artifacts are labelled: runnable in shape, unexecuted as printed.

The question that started the hour

The pull request added a model call inside a CI job. It worked. The description said the free tier covered it.

The senior asked one question and then stopped talking: what number is that claim based on?

Not "which model", not "how fast". Just the number. The author had a dashboard total, a rough per-request guess, and a green CI run. None of those three is a measurement of the same thing.

Dead end 1: the dashboard aggregate

The first instinct was to screenshot the provider console and divide by the number of test runs. That failed for a boring reason: the console total included the fixture-writing conversations from three days earlier, mixed with suite runs and a few manual probes.

An aggregate answers "how much did this account spend". It does not answer "how much does one pull request cost". Those are different denominators, and CI needs the second one.

The console total could not be split by job.
The job could not be replayed against the same counter.
Nothing in the repository recorded which calls happened at all.

Dead end. The pair moved on.

Dead end 2: estimating with `len(text) / 4`

The second attempt was arithmetic. Character count divided by four, summed over the prompt, multiplied by a fudge factor for the completion.

It survived exactly one experiment. The suite sends JSON fixtures, indented, with long keys repeated across cases. Tokenizers do not treat whitespace and braces the way a character heuristic assumes, and the estimate drifted by a margin wide enough to flip the decision in both directions on different days.

An estimate that is sometimes high and sometimes low is not a conservative estimate. It is noise with a decimal point.

Dead end 3: wall-clock time as a cost proxy

Third attempt: if a run is slow, it must be expensive. Tempting, and wrong for the same reason cold-start timing is misleading in any free-tier environment — variance dominates. A queue hiccup on one run outweighed the real token delta across ten others.

Latency is worth measuring. It just measures a different thing than usage.

The artifact we kept: a usage ledger and a fail-closed gate

Two small files, no framework. The first wraps every call and appends one JSON line per request.

# budget_probe.py — illustrative, not executed as printed.
# Appends one ledger row per call to an OpenAI-compatible endpoint.
import json, os, time, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
LEDGER = os.environ.get("BUDGET_LEDGER", "usage.jsonl")

def call(prompt: str, tag: str) -> dict:
    body = json.dumps({
        "model": os.environ["LLM_MODEL"],
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=120) as resp:
        payload = json.load(resp)
    usage = payload.get("usage") or {}
    row = {
        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "tag": tag,
        "model": payload.get("model"),
        "prompt_tokens": usage.get("prompt_tokens"),
        "completion_tokens": usage.get("completion_tokens"),
        "seconds": round(time.monotonic() - started, 3),
    }
    with open(LEDGER, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")
    return row

The second file reads the ledger and refuses to guess. This is the part that mattered in review: unmeasured calls fail the gate instead of being silently averaged into a comfortable number.

# budget_gate.py — illustrative, not executed as printed.
import json, sys
from datetime import datetime

ALLOWANCE = int(sys.argv[1])   # operator-declared allowance, in tokens
DAYS = int(sys.argv[2]) if len(sys.argv) > 2 else 30

rows, missing = [], 0
for line in open("usage.jsonl", encoding="utf-8"):
    r = json.loads(line)
    if r["prompt_tokens"] is None or r["completion_tokens"] is None:
        missing += 1
        continue
    rows.append(r)

if not rows:
    sys.exit("no measured rows: refusing to project")

used = sum(r["prompt_tokens"] + r["completion_tokens"] for r in rows)
stamps = sorted(datetime.fromisoformat(r["ts"].replace("Z", "+00:00")) for r in rows)
span = max((stamps[-1] - stamps[0]).total_seconds() / 86400, 1 / 24)
per_day = used / span
projected = per_day * DAYS

print(f"rows={len(rows)} unmeasured={missing} used={used} per_day={per_day:.0f}")
print(f"projected={projected:.0f} allowance={ALLOWANCE} runway_days={ALLOWANCE / per_day:.1f}")

if missing:
    sys.exit("fail closed: unmeasured calls make the projection a guess")
if projected > ALLOWANCE * 0.8:
    sys.exit("headroom under 20%: do not send the full suite through this tier")

Run it as python budget_gate.py <allowance> 30. The threshold is deliberately blunt: eighty percent, not ninety-five, because the suite grows and the allowance does not.

The canary that keeps the ledger honest

Accounting drifts when the provider changes how it counts, or when a proxy strips the usage block. A fixed canary prompt — same text every week, stored in the repository — gives a stable token count to compare against. A shift of more than a few tokens is a signal to re-read the terms, not a signal to edit the threshold.

The decision table the pair wrote down

Signal from the ledger	Action
Usage is measured, headroom above 20%	Keep the free tier for CI
Usage is measured, headroom below 20%	Reduce suite to a labelled subset; keep the full suite manual
`unmeasured > 0`	Stop. Fix instrumentation before scaling calls
Canary token count drifts week over week	Re-verify allowance terms and tokenizer behaviour
Per-day burn exceeds what any free tier plausibly covers	Move the heavy path to a paid or self-hosted endpoint

Where a free tier and a free server actually fit

The operator describes free model access and a free server option for this workflow. That combination is genuinely useful for the first week of a ledger: the numbers you are collecting are exactly the numbers that tell you whether the free arrangement is enough.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two cautions before pointing a suite at it. Allowances and availability change, so read the current terms rather than a post from any month, including this one. And a free server is still a server: measure its cold-start behaviour separately, because latency variance will contaminate any cost reasoning you try to do from timing alone.

Limitations and who should skip this

The ledger assumes the endpoint returns a usage block. If it does not, the gate fails closed by design and the approach stalls until you measure server-side.
Token counts are not currency. If your provider bills in units you cannot map to tokens, this table is the wrong abstraction.
Teams under a hard data-residency rule should not route fixtures through a hosted free tier at all, regardless of allowance.
Anyone who needs a guaranteed latency floor should skip the free-server path and pay for isolation.

The boring conclusion held up in review: the ledger was worth more than the model. If the free access disappears tomorrow, the gate still tells you what the suite costs and where to cut it. That is the part you keep.

If you want to try the shape of this, start with one job, one canary prompt, and one week of rows before you trust any allowance number — including the one in this article.

Time the Cold Start Before You Trust a Free Server With Your Review Bot

Quinn Sun — Sat, 17 Oct 2026 05:58:13 +0000

The question nobody in the room could answer

A pairing hour in mid-October, two engineers, one prototype. The bot takes a unified diff, asks a model for structured findings, and posts them on a pull request. It ran on a free server instance and called a free model endpoint.

Nobody could say what the first request after ten minutes of idle actually cost in wall-clock time.

Three things were unknown, and all three were being papered over:

the latency of the first request after an idle period;
the timeout the caller had assumed (somewhere between 10 seconds and never);
whether the person who found the number could reproduce it a week later.

Five questions the senior wrote on the whiteboard

What is the first-request latency after ten minutes of idle, measured on this deployment, today?
Which component owns the timeout budget?
Can someone who was not in this session reproduce the measurement in one command?
What happens to a request that arrives during a cold start?
If the number is bad, what changes — the code, the deploy, or the promise made to the reviewer?

Question five is the one that mattered later. It forces the decision to be written down before the measurement, so the measurement cannot be reinterpreted to fit a preference.

Two dead ends before the first line of code

Dead end one: read the logs. The request log on the free tier listed status codes and nothing else. No timestamps, no upstream duration. There was nothing to compute from.

Dead end two: read the documentation. Sleep behaviour for that plan was not documented. Three blog posts produced two contradictory assumptions, which is worse than zero assumptions because each side now had a citation.

The rule the senior set after that: an external observation is only trustworthy if the observed process also reports its own timestamps, so two clocks can be compared instead of one being guessed.

Where the free model access and the free server fit

MonkeyCode provides free model access and a free server option, and that combination is what made this prototype cheap enough to exist at all. Without it, the pairing hour would have been a budget conversation instead of a measurement.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest notes before any code. The current limits of the free access and the free server — quotas, sleep policy, hardware, model list — are not inputs to the method below and are not asserted here; read them from the project's own documentation on the day you run the probe, because they can change and this article would not know. And every number in the sample output further down is a placeholder that shows column shape, not a benchmark.

Artifact one: a service that describes its own state

The first kept decision was small and unglamorous: the service must report its own clock. Every response carries a server-side duration, and /health reports how long ago the model was last called. That single field separates "the network is slow" from "nothing has run in eleven minutes."

// server.mjs — Node 20+, zero dependencies
import http from 'node:http';
import crypto from 'node:crypto';
import { performance } from 'node:perf_hooks';

const PORT = Number(process.env.PORT ?? 8080);
const BASE_URL = process.env.MODEL_BASE_URL; // OpenAI-compatible endpoint
const MODEL_ID = process.env.MODEL_ID;
const API_KEY = process.env.MODEL_API_KEY;
const BUDGET_MS = Number(process.env.REVIEW_BUDGET_MS ?? 45000);

let lastModelCallAt = null;

function json(res, code, body) {
  const payload = JSON.stringify(body);
  res.writeHead(code, {
    'content-type': 'application/json',
    'content-length': Buffer.byteLength(payload),
  });
  res.end(payload);
}

async function review(diff, requestId) {
  const t0 = performance.now();
  try {
    const upstream = await fetch(`${BASE_URL}/chat/completions`, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        authorization: `Bearer ${API_KEY}`,
      },
      body: JSON.stringify({
        model: MODEL_ID,
        temperature: 0,
        response_format: { type: 'json_object' },
        messages: [
          { role: 'system', content: 'Return JSON only: {"findings":[{"file":string,"line":number,"claim":string}]}. No prose.' },
          { role: 'user', content: diff },
        ],
      }),
      signal: AbortSignal.timeout(BUDGET_MS),
    });
    const serverMs = Math.round(performance.now() - t0);
    lastModelCallAt = Date.now();
    if (!upstream.ok) return { requestId, ok: false, stage: 'upstream', upstreamStatus: upstream.status, serverMs };
    const body = await upstream.json();
    return { requestId, ok: true, serverMs, raw: body.choices?.[0]?.message?.content ?? null };
  } catch (err) {
    const serverMs = Math.round(performance.now() - t0);
    return { requestId, ok: false, stage: err.name === 'TimeoutError' ? 'budget' : 'transport', serverMs };
  }
}

const server = http.createServer(async (req, res) => {
  const requestId = crypto.randomUUID();

  if (req.url === '/health') {
    return json(res, 200, {
      status: BASE_URL && MODEL_ID ? 'configured' : 'missing_env',
      budgetMs: BUDGET_MS,
      lastModelCallAgoMs: lastModelCallAt ? Date.now() - lastModelCallAt : null,
    });
  }

  if (req.url === '/review' && req.method === 'POST') {
    const chunks = [];
    for await (const chunk of req) chunks.push(chunk);
    const diff = Buffer.concat(chunks).toString('utf8');
    if (diff.length > 40000) return json(res, 413, { requestId, error: 'diff_too_large' });
    return json(res, 200, await review(diff, requestId));
  }

  json(res, 404, { requestId, error: 'not_found' });
});

server.listen(PORT, () => console.log(`listening on ${PORT}`));

Two details are deliberate. The timeout is a named budget (REVIEW_BUDGET_MS) rather than a scattered literal, so question three has one answer. And a timeout returns a structured failure with the stage that failed, instead of an exception that the caller has to classify.

Artifact two: idle, then measure, then print CSV

Guessing was the whole problem, so the probe does the boring thing. It sleeps, fires one request, fires a second one immediately, and records both. Attempt 1 is the cold request; attempt 2 is the warm one. The difference between them is the entire finding.

// probe-idle.mjs — node probe-idle.mjs <baseUrl> <idleMinutes> <cycles>
const base = process.argv[2] ?? 'http://127.0.0.1:8080';
const idleMs = Number(process.argv[3] ?? 10) * 60000;
const cycles = Number(process.argv[4] ?? 3);
const diff = '--- a/x.js\n+++ b/x.js\n@@ -1,2 +1,2 @@\n-const a = 1\n+const a = 1;\n';

console.log('cycle,endpoint,attempt,status,ms');

for (let cycle = 1; cycle <= cycles; cycle++) {
  await new Promise((r) => setTimeout(r, idleMs));
  for (const endpoint of ['health', 'review']) {
    for (let attempt = 1; attempt <= 2; attempt++) {
      const t0 = performance.now();
      const res = endpoint === 'health'
        ? await fetch(`${base}/health`)
        : await fetch(`${base}/review`, { method: 'POST', body: diff });
      const ms = Math.round(performance.now() - t0);
      await res.arrayBuffer();
      console.log([cycle, endpoint, attempt, res.status, ms].join(','));
    }
  }
}

Run it and keep the CSV in the repo, because a committed CSV is a reproducible claim and a screenshot is not:

node probe-idle.mjs https://review-bot.example 10 3 > cold-start.csv

awk -F, 'NR>1 && $2=="review" && $3==1 {print $5}' cold-start.csv \
  | sort -n \
  | awk '{v[NR]=$1} END {printf "n=%d p50=%dms max=%dms\n", NR, v[int(NR/2)], v[NR]}'

Sample shape of the output — placeholders, not measurements:

cycle	endpoint	attempt	status	ms
1	health	1	200	NN
1	health	2	200	NN
1	review	1	200	NNNN
1	review	2	200	NNN

If attempt 1 and attempt 2 differ by an order of magnitude, the deployment sleeps and the reviewer will eventually see a timeout that has nothing to do with the model's answer quality.

The decision table, written before the run

Pattern	Fits when	Cost	How to verify
On-demand only	cold-start latency is inside the caller's budget	cheapest, but one slow first reviewer	probe attempt 1 under the caller's real timeout
Keep warm with a timer	cold start exceeds the budget and the free tier permits periodic requests	a timer that must not be mistaken for real traffic	read the tier's terms first, then confirm `lastModelCallAgoMs` stays bounded
Queue and batch reviews	reviewers tolerate minutes, not seconds	a queue you now operate	end-to-end timestamp on the queued job, not the HTTP call
Retry inside the gateway	never, for cold starts	hides the failure and doubles cost	it should fail review; a second request measures, it does not fix

The senior crossed out the fourth row personally. A retry that masks a cold start turns a measurable problem into an intermittent one.

What was kept, and what was dropped

Kept: /health with lastModelCallAgoMs, the named timeout budget, the structured failure stage, and probe-idle.mjs with its CSV.

Dropped: the keep-alive cron job, at least until someone reads the terms of the free tier rather than assuming periodic pings are welcome. A five-minute systemd timer is trivial to write and trivial to get wrong:

# /etc/systemd/system/review-health.timer
[Unit]
Description=Ping the review bot health endpoint

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

If that timer ever ships, it ships with a note explaining why it exists and what it costs, because the next reader will otherwise delete it as noise.

Limitations, and who should skip this

This method measures one deployment on one day. Sleep policy, quotas, and free-tier behaviour can change without a changelog, so a CSV from last month is evidence about last month.

The probe also says nothing about answer quality — a fast wrong review is still wrong, and latency numbers should never be used to rank models. And a single cold observation is not an SLO; if you need a p95 promise, you need paid capacity or your own host, not a free tier and a hopeful timer.

Skip this approach if the diffs contain anything that cannot leave your network, if reviewers already expect sub-second feedback, or if nobody on the team will own the CSV. A measurement nobody re-runs decays into folklore within a quarter.

The probe is the part worth keeping. If you want a free model endpoint and a free server to point it at while you learn your own numbers, MonkeyCode is a reasonable place to start — run probe-idle.mjs before you promise anyone a latency figure.

A Pairing Hour on a Timezone-Flaky Test: Three Dead Ends and the One Seam We Kept

Quinn Sun — Fri, 16 Oct 2026 04:55:04 +0000

The failing test only broke between 23:00 and 23:05 UTC. Locally it passed every time, on every laptop, for two weeks. Then a senior engineer sat down for a one-hour pairing session and spent the first ten minutes asking questions instead of opening an editor.

What follows is that hour, kept close to how it happened: the questions, the three fixes we abandoned, and the single change that survived review. The artifact is a probe script and a clock seam small enough to paste into a real repository.

The first ten minutes were questions, not code

The senior did not read the assertion first. They asked, in this order:

What does the test actually assert — a wall-clock time, a duration, or an ordering?
Which clock produced the value under test: the system clock, a library clock, or a clock inside the framework?
Whose offset matters — the machine's TZ, the database session's time zone, or the scheduler's own?
Who else in the codebase already reads time, and can they be changed without touching behavior?

The answers were uncomfortable. The test asserted a formatted string. The code read new Date() directly in three modules. Nobody could say which TZ CI used, because the workflow set it in one job and inherited it in another.

That last answer explained the 23:00 window without explaining the failure. A vague explanation is not a diagnosis.

Reproduce before you read the stack trace

We wrote a shell probe that runs the same spec under several wall-clock offsets, repeatedly, and prints a matrix instead of a wall of logs. It is deliberately dependency-light: one script, one runner variable, no framework hooks.

#!/usr/bin/env bash
# probe_tz.sh — run one spec under several wall-clock offsets.
set -euo pipefail

SPEC="${1:-tests/scheduler.spec.ts}"
RUNNER="${RUNNER:-npx vitest run}"
RUNS="${RUNS:-3}"
ZONES=(UTC Asia/Tokyo America/Sao_Paulo Pacific/Chatham)

fail=0
for tz in "${ZONES[@]}"; do
  for i in $(seq 1 "$RUNS"); do
    printf '%-20s run %d: ' "$tz" "$i"
    if TZ="$tz" $RUNNER "$SPEC" >/tmp/probe-$tz-$i.log 2>&1; then
      echo PASS
    else
      echo FAIL
      fail=1
    fi
  done
done

exit "$fail"

Two details matter more than the code. First, Pacific/Chatham is in the list on purpose: a 45-minute offset plus DST catches arithmetic that assumes whole-hour offsets. Second, the script keeps per-run logs under /tmp, so a failure is inspectable without re-running the matrix.

If RUNS is left at 3, a flaky result that appears once out of twelve looks like noise. Set RUNS=5 or higher before trusting a green matrix — a single pass proves nothing about a race.

Dead end 1: add retries to the test

The first suggestion was the cheapest one: wrap the assertion in a retry helper. It turned red into green in four minutes.

It also deleted the signal. The flake was deterministic given an offset, and retries converted a reproducible boundary bug into a probabilistic one that would eventually surface in production at the same hour, with real schedules attached. The senior's note was blunt: retries are for infrastructure you do not control, not for your own arithmetic.

We reverted it in the same session. It never reached review.

Dead end 2: a global fake timer

The second attempt replaced the whole timer system in the test process, froze time at a fixed instant, and advanced it manually. The spec passed. So did three unrelated specs that had been quietly relying on real timers, because the fake timer also stopped a retry backoff and made two slow tests look fast.

Worse, it hid the original bug. Freezing the clock removes the offset, and the offset was the bug.

A related trap: assigning process.env.TZ inside a test after the process has started. In current Node.js runtimes the time zone is resolved early, so a mid-test assignment is unreliable and can pass on one machine while failing on another. A probe matrix set from the shell avoids that entirely.

Dead end 3: a production freeze flag

The third proposal was the most tempting for a deadline: read an environment variable in the scheduler and let tests pin the current time.

We rejected it for two reasons that are cheap to state and expensive to discover late. A production code path now branches on test configuration, which means the code under test is not the code that runs. And anyone who sets that variable in the wrong environment silently stops scheduled work — a flag that looks like a convenience and behaves like an outage switch.

The decision we kept: one clock seam

The change that survived is small, explicit, and boring. Time enters the scheduler through a parameter with a default, and nothing else in the module calls new Date() at the top level.

// src/clock.ts
export interface Clock {
  now(): Date;
}

export const systemClock: Clock = { now: () => new Date() };

// src/scheduler.ts
import { type Clock, systemClock } from './clock.js';

export function nextRun(clock: Clock = systemClock): Date {
  const now = clock.now();
  const next = new Date(now.getTime());
  next.setUTCHours(2, 0, 0, 0);
  if (next.getTime() <= now.getTime()) {
    next.setUTCDate(next.getUTCDate() + 1);
  }
  return next;
}

// tests/scheduler.spec.ts
import { describe, expect, it } from 'vitest';
import { nextRun } from '../src/scheduler.js';

const at = (iso: string) => ({ now: () => new Date(iso) });

describe('nextRun', () => {
  it('returns the next 02:00 UTC when now is before it', () => {
    expect(nextRun(at('2026-03-08T01:30:00Z')).toISOString())
      .toBe('2026-03-08T02:00:00.000Z');
  });

  it('rolls to tomorrow when now is exactly 02:00 UTC', () => {
    expect(nextRun(at('2026-03-08T02:00:00Z')).toISOString())
      .toBe('2026-03-09T02:00:00.000Z');
  });
});

The two assertions above are boundary cases, not coverage. The matrix script is what proves offset independence, and it runs against the same suite in CI as a single job with RUNNER pinned to the project's test command.

What the seam costs

One interface file, one default parameter, and a reviewer who can see where time enters the module.
A small refactor if other modules read time directly. Do that separately, in its own commit, so the behavioral change stays reviewable.
No new dependency, no test-only branch in production code, and no configuration flag that can be mis-set.

Where a free model and a disposable server fit the hour

The senior ran the pairing hour; the tooling was mine. I used MonkeyCode's free model access to draft candidate hypotheses for the flake — five plausible causes, written down before anyone touched the code — and I used the free server option as a throwaway Linux box to run the TZ matrix somewhere other than my laptop, because a local machine with a cached build is not a clean environment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the model was good for: enumerating causes quickly, including offset-aware ones I had not considered, and drafting the first version of the probe script. What it was not good for: deciding which hypothesis was true. Two of its five suggestions were dead ends 1 and 3 above. The model generated options; the shell matrix and the senior's review decided between them.

Honest limits of that setup, as I experienced it:

The free server option is a shared, ephemeral environment. Treat it as a disposable probe box, not a place for endurance runs or timing-sensitive benchmarks.
Do not paste credentials, customer data, or private repository contents into any model prompt. The probe matrix needs an environment variable and a test command, nothing more.
Free-tier terms, allowances, and available models change. Check the project's own current documentation rather than a screenshot or a blog post, including this one.

A decision table for the next flake

Symptom	Adopt	Park	Revert
Fails only in a time window	Probe matrix + clock seam	—	Retry wrappers
Fails only on one machine	Pin the offset in CI	Containerize the runner	Global fake timers
Passes with fake timers, fails without	Inject the clock at one seam	Broaden the matrix	Freeze flag in production code
Fails under load, not under time	Reproduce with a concurrency probe	Add a soak job	Anything clock-related

Limits, and who should skip this

This approach fits deterministic boundary bugs that a wall-clock offset can expose. It does not fit every flake. If the failure comes from real concurrency, network timeouts, or a shared fixture, a TZ matrix will stay green and teach nothing.

Skip it if your tests run against regulated data that cannot leave a managed environment, if you need dedicated hardware for reproducible timing, or if the team cannot commit to keeping one CI job for the matrix. A probe script that nobody runs is worse than no script, because it suggests coverage that does not exist.

What went into the pairing notes

One line, written by the senior before the hour ended: the bug was in the arithmetic, the fix was in the seam, and the retries were the expensive part.

If your next pairing hour loses its first thirty minutes to environment setup rather than to the bug, that is the workflow worth testing before you spend anything on tooling.

Red Before Green: A Probe Gate for Free-Tier Model Output

Quinn Sun — Thu, 15 Oct 2026 11:12:30 +0000

Tuesday pairing hour, forty minutes left on the clock. The model proposed a four-line fix for a rounding bug in a price calculation, and it also wrote the test that covered the fix.

The diff looked reasonable. The test went green on the first run. That is usually the moment a session ends early, with a patch that nobody has actually challenged.

The senior on the call did not read the diff. The senior asked one question: does that test fail without your fix?

Nobody knew. The test had been written after the patch, against the patched tree. It had never been red.

Three dead ends that ate the hour

Before the useful question landed, the session tried three things that felt like progress and produced no decision.

Reading the diff for correctness. A four-line change is short enough to read and long enough to misread. Reviewing it produced opinions, not evidence, and opinions do not survive a code review two days later.
Re-prompting with more context. A second attempt with the surrounding module pasted in produced a larger diff and the same unanswered question. More context did not make the probe load-bearing.
Re-running the suite and watching it pass. Green on the patched tree is exactly what a test that asserts nothing will do. Passing is the weakest possible signal here.

None of these answered the only question that mattered: if the patch were reverted, would anything in the repository notice?

The question that ends an argument in one run

Does the probe fail on the tree without the patch? That is decidable. It takes one checkout, one command, and no judgement about code style or author intent.

A probe that has never been red is a description, not a test

This is the rule the session kept. Not "write tests first" as a ritual, but a narrower and checkable property: a probe must be red on the unfixed tree and green on the fixed tree, with the same inputs on both sides.

If the probe is green on both trees, it is documenting existing behaviour. If it is red on both trees, the patch is incomplete or the probe is wrong. Only one of those four combinations is a result.

The artifact: a probe gate in about sixty lines

The gate below runs a probe against two checkouts, hashes a frozen input set in each, repeats every run to catch flake, and writes a JSON report. It has no dependencies outside the standard library.

#!/usr/bin/env python3
"""probe_gate.py - a probe must be red before a patch and green after it.

Exit codes: 0 gate passed, 1 gate failed, 2 usage/environment error.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import shlex
import subprocess
import sys
from pathlib import Path


def digest(paths: list[Path]) -> str:
    h = hashlib.sha256()
    for p in sorted(paths):
        h.update(p.as_posix().encode())
        h.update(p.read_bytes())
    return h.hexdigest()[:12]


def run(cmd: list[str], cwd: Path) -> tuple[int, str]:
    proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=900)
    return proc.returncode, (proc.stdout + proc.stderr)[-2000:]


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--before", required=True, type=Path, help="checkout without the patch")
    ap.add_argument("--after", required=True, type=Path, help="checkout with the patch")
    ap.add_argument("--probe", required=True, help="e.g. 'pytest -q tests/test_rounding.py'")
    ap.add_argument("--frozen", nargs="*", default=[],
                    help="files whose bytes must match in both trees")
    ap.add_argument("--repeat", type=int, default=2, help="runs per tree, to catch flake")
    ap.add_argument("--report", type=Path, default=Path("probe-report.json"))
    args = ap.parse_args()

    cmd = shlex.split(args.probe)
    frozen = [Path(f) for f in args.frozen]

    before_digest = digest([args.before / f for f in frozen]) if frozen else "-"
    after_digest = digest([args.after / f for f in frozen]) if frozen else "-"

    report: dict[str, object] = {
        "probe": args.probe,
        "frozen_digest_before": before_digest,
        "frozen_digest_after": after_digest,
        "runs": [],
    }

    verdict = "pass"
    if before_digest != after_digest:
        verdict = "fail: frozen inputs differ between trees"

    before_codes, after_codes = [], []
    for _ in range(args.repeat):
        code, tail = run(cmd, args.before)
        before_codes.append(code)
        report["runs"].append({"tree": "before", "exit": code, "tail": tail})
    for _ in range(args.repeat):
        code, tail = run(cmd, args.after)
        after_codes.append(code)
        report["runs"].append({"tree": "after", "exit": code, "tail": tail})

    if len(set(before_codes)) != 1 or len(set(after_codes)) != 1:
        verdict = "fail: flaky probe (exit codes differ across repeats)"
    elif before_codes[0] == 0:
        verdict = "fail: probe already passes without the patch"
    elif after_codes[0] != 0:
        verdict = "fail: probe still red after the patch"

    report["verdict"] = verdict
    args.report.write_text(json.dumps(report, indent=2))
    print(verdict)
    print(f"report: {args.report}")
    return 0 if verdict == "pass" else 1


if __name__ == "__main__":
    sys.exit(main())

Running it

The two trees are cheap. A git worktree gives you the unpatched checkout without stashing anything.

git worktree add ../tree-before HEAD

python probe_gate.py \
  --before ../tree-before \
  --after . \
  --frozen fixtures/cart.json src/pricing.py \
  --probe "python -m pytest tests/test_rounding.py -q" \
  --repeat 2 \
  --report probe-report.json

--frozen is the part people skip. If the fixture the probe reads is different in the two trees, the gate fails immediately, because a probe that runs against two different worlds is not comparing a patch to its absence.

A log shape worth copying

The table below shows the columns the gate produces. The values are illustrative, not measurements from a specific deployment.

Probe	Tree	Exit	Reading
rounding	before	1	red, as required
rounding	before	1	stable across repeat
rounding	after	0	green
rounding	after	0	stable across repeat
tax-total	before	0	rejected: never red

Four of five rows are one probe doing its job. The fifth row is the common failure: a test that the model wrote to describe existing behaviour, presented in the same diff as an assertion about the fix.

Routing questions in a pairing hour

Not every question in a session deserves the same responder. The routing table below is the practical version of what the senior did on the call.

Question	Cheapest honest source	Reason
What does this code do now?	Run it	Execution beats explanation
What inputs might break it?	Model, freely	Hypothesis generation is cheap and wrong answers cost little
Is this probe load-bearing?	The unpatched tree	One run, binary answer
Is this acceptable to ship?	A human owner	No model output carries accountability
Is the failure environmental?	The frozen digest + a clean container	Distinguishes code from machine

The second row is where delegate-heavy workflows get their value. The third row is where they get their safety.

Where free model access and a free server change the economics

Probing is repetitive, and repetition is exactly what a free tier is good for. In this workflow the model's job is narrow: propose candidate probes for a failure that a human has already described. Each candidate is then run twice, on two trees, by a machine nobody has to pay for. MonkeyCode's operator states that the project offers free model access and a free server option, which is what makes that proposal-then-discard loop affordable rather than an expensive per-call ritual. I have not verified current quotas, limits, retention, or hardware, and a large free token allowance circulating in promotion is a marketing number until the project's own current terms confirm it. Check the docs before you plan a month around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caution that belongs next to the disclosure: a free server is a place to run experiments, not a place to keep the only copy of an artifact. Treat the container as disposable and push the JSON report back to your own repository or CI storage.

Limitations, and who should skip this

The gate cannot tell you whether the probe tests the right thing. A red-before, green-after probe that asserts a log line is technically valid and practically useless. Only review catches that.
Frozen inputs are a human decision. The gate verifies that the listed files match; it cannot know which files are causally relevant.
Flake detection costs time. --repeat 2 doubles every run. On slow suites, drop to one repeat and accept the risk.
It does not fit nondeterministic failures. Race conditions, timing bugs, and visual regressions need different instruments.
Skip it entirely if you cannot define acceptance criteria. A team that cannot write down what "fixed" means will only produce rows in a report.
Skip it for dependency bumps and generated code. There is no meaningful unpatched tree to compare against.

What the hour kept

The session kept a rule, not a tool: no model-authored test enters the review until the gate has shown it red on the tree without the patch. It costs about two minutes per probe and it removes the single most common way a pairing session produces confident, empty evidence.

The senior's last comment before the call ended was that the gate is boring, which is the point. Boring verification is what makes fast, generous delegation safe. If you want to try the red-before-green loop on a machine you do not have to provision yourself, the free server option is where this workflow was designed to run.

The Upload Set a Senior Signed Before Any Free-Server Run

Quinn Sun — Tue, 13 Oct 2026 00:12:51 +0000

This workflow starts from a pairing scenario, not from a measured production incident. In that scenario the block had forty minutes left. A webhook verifier had grown a new signature check, and the laptop suite was already busy with an unrelated package install. A free server seat sat unused. Free model access was already open in the editor.

The faster path looked obvious. Upload the fixture tree, ask the model to rerun the suite, and treat a green log as the end of the hour. The senior did not ban the tools. The senior banned that shortcut.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are treated here as operator-supplied availability claims. This draft does not name models, quotas, hardware, regions, or retention periods, because no primary source for those details was attached.

Commands below are unexecuted examples. Paths are illustrative. No speedup figure is claimed, and no customer incident is described.

What the hour had to protect

The diff itself was small. verify_signature had to reject a stale timestamp and a bad digest. The risk was not line count. The risk was letting a hosted run define done while fixtures still held a local capture, a clock assumption, and hostnames the team was not allowed to copy off the laptop.

Published pieces on this account already covered retry budgets, replay exit codes, and path denylists. This workflow keeps a different object. It keeps a host-split manifest: which files a model may read, which tests a free server may execute, and which checks stay on the laptop even when the remote seat is idle.

That split is the technical subject. Hosted capacity sits around it. Capacity does not replace the split.

Questions written down before any remote command

The senior required answers in the pairing notes, not in the chat window.

Which paths contain tokens, customer payloads, signing keys, or internal hostnames?
Which tests depend on clock, locale, DNS, or outbound network, so a different host can pass for the wrong reason?
Which failures must be reproduced by a command both people can rerun after the session closes?
What is the stop rule if the free server returns empty, skips the new test, or resets mid-run?
Who initials the upload set before anything leaves the laptop?

The answers took longer than the patch. A model paragraph cannot initial an upload set. The junior owns the draft. The senior owns the signature. Either person can stop the remote run.

Dead ends that did not survive

A skip looked like success. The first remote attempt ran the package target on the free server. The log ended at exit code 0. The new test never started. The server image had no stub for the local signer, and the runner treated the missing extra as a skip. Green meant not collected. It did not mean the verifier held.

Filename optimism classified a capture as safe. Asked which fixtures could be uploaded, the model marked payload.sample.json as fine. A comment inside that file said the bytes came from a production capture and only the outer name had been sanitized. The classification was discarded. Safety is a content decision, not a suffix decision.

A bypass header appeared from nowhere. With the diff and a fixture pasted into the free model session, the edit proposed X-Verify-Bypass: 1 for local debugging. No such header existed in the repository. Accepting it would have taught the suite to pass by disabling the check. The senior reverted the buffer and kept the original assertion.

Each dead end failed the same local test. Could both people rerun the claim after the chat closed, without trusting the transcript? If the answer was no, the claim did not enter the notes.

The decision that stayed

Three host classes. One manifest. Fail closed.

local-only stays on the laptop. Keys, captures, internal hosts, and clock-bound tests live here. They never enter an upload set.
server-eligible may run on the free server. Pure functions and synthetic fixtures belong here, with no outbound network and no copied captures.
model-visible may appear in the editor session. This class is for the diff, public interfaces, and synthetic examples. It is a subset of server-eligible, not a third dumping ground.

A file can be useful and still be local-only. Usefulness is not a transfer permit. The free server is a capacity valve for the server-eligible subset when the laptop is busy. Free model access is a drafting aid for names and comments. Neither class inherits trust from the other.

The hour's rule was written in one sentence and left in the notes. Nothing remote starts until the checker accepts the upload list and both names sit beside that list.

Sequence the pair followed

Freeze the diff. No further product edits until the host split exists.
Assign classes by hand. The model may suggest file names. Humans assign the class.
Write the candidate upload list as plain text, one path per line.
Run the checker. A non-zero exit stops the remote plan.
Run local-only tests on the laptop and record that exit code.
Only if the checker is clean, run one explicit server-eligible target on the free server. Record that exit code on a separate line.
If the codes disagree on a synthetic test, treat it as a host-class bug. Do not edit the assertion until collection, skip, and dependency causes are known.
Store the manifest and both codes. Do not store the model summary.

The laptop code wins for local-only tests. The free server never adjudicates those. A disagreement on a synthetic test is information about the host, not permission to weaken the check.

Unexecuted checker

The Python below is a proposal for the notes. It has not been timed. It does not call a network API. It reads a manifest and a candidate upload list, then exits non-zero when a local-only path is included, when an upload path lacks a server-eligible mark, or when model-visible is not a subset of server-eligible.

#!/usr/bin/env python3
"""Fail closed if an upload list violates a host-split manifest.

Unexecuted example. Paths and class names are illustrative.
"""
from __future__ import annotations

import sys
from pathlib import Path

REQUIRED = ("local-only", "server-eligible", "model-visible")


def parse_manifest(text: str) -> dict[str, set[str]]:
    classes: dict[str, set[str]] = {name: set() for name in REQUIRED}
    current = None
    for raw in text.splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        if line.endswith(":") and line[:-1] in classes:
            current = line[:-1]
            continue
        if current is None:
            raise SystemExit("path before class header: " + line)
        classes[current].add(line)
    return classes


def main(argv: list[str]) -> int:
    if len(argv) != 3:
        print("usage: host_split_check.py MANIFEST UPLOAD_LIST", file=sys.stderr)
        return 2
    classes = parse_manifest(Path(argv[1]).read_text(encoding="utf-8"))
    upload = {
        line.strip()
        for line in Path(argv[2]).read_text(encoding="utf-8").splitlines()
        if line.strip() and not line.strip().startswith("#")
    }
    local_only = classes["local-only"]
    server_ok = classes["server-eligible"]
    model_ok = classes["model-visible"]
    errors: list[str] = []
    leaked = sorted(upload & local_only)
    if leaked:
        errors.append("local-only paths in upload set: " + ", ".join(leaked))
    unmarked = sorted(upload - server_ok - local_only)
    if unmarked:
        errors.append("upload paths missing server-eligible mark: " + ", ".join(unmarked))
    bad_model = sorted(model_ok - server_ok)
    if bad_model:
        errors.append("model-visible but not server-eligible: " + ", ".join(bad_model))
    overlap = sorted(local_only & (server_ok | model_ok))
    if overlap:
        errors.append("local-only overlaps another class: " + ", ".join(overlap))
    if errors:
        print("\n".join(errors), file=sys.stderr)
        return 1
    print("upload set ok: %d path(s)" % len(upload))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Illustrative manifest:

# host-split.manifest
local-only:
  fixtures/captures/prod-replay.json
  tests/test_clock_skew.py
server-eligible:
  src/verify_signature.py
  tests/test_digest_synthetic.py
  fixtures/synthetic/good-digest.json
model-visible:
  src/verify_signature.py
  tests/test_digest_synthetic.py

Candidate upload list that should fail:

src/verify_signature.py
tests/test_digest_synthetic.py
fixtures/captures/prod-replay.json

Local command shape, still unexecuted:

python3 host_split_check.py host-split.manifest upload.txt
# non-zero expected: prod-replay.json is local-only

# after a clean upload list, laptop synthetic check:
python3 -m unittest tests.test_digest_synthetic

The free-server command is intentionally not scripted. The pairing rule was to submit one explicit test target after the checker printed upload set ok, then copy the remote exit code into the notes. A model retelling of that log does not replace the code.

Where the free options actually helped

Free model access had one narrow job. It drafted manifest comments and proposed synthetic fixture names. The bypass header and the filename classification were rejected. The useful residue was a shorter list of synthetic files, which the junior then wrote by hand.

The free server option was the place to run tests.test_digest_synthetic once the upload set was clean and the laptop was still busy. It was not the place to run tests.test_clock_skew. It was not evidence that production would behave the same way. A different host can pass because a test was skipped, not because an assertion held.

Remove every product mention and the workflow still stands. Split the suite. Check the upload set. Record two exit codes. The hosted seat only matters after that split exists, and only for the class the manifest allows.

Limitations, and who should skip it

The checker matches path strings. It does not scan contents for secrets. A server-eligible path can still hide a token. Content review stays human.
./fixtures/a.json and fixtures/a.json are different strings here. Do not drop this example into a production upload pipeline without path normalization.
Environment parity is out of scope. Locale, clock, CPU, and network still differ. Clock-bound and network-bound tests stay local-only.
This split is not a data-processing agreement. Teams that must keep customer data, regulated payloads, or signing keys off shared hosts should not treat a free seat as an approved destination.
Numeric allowances, model names, session length, and log retention were not verified for this draft. Availability changes. Current product docs outrank this article.
A solo rush with nobody to initial the manifest should skip the remote half. The second signature is the control. The script is only the latch.
Flaky timing suites and air-gapped repositories are a bad fit. So is any team that wants one green badge to cover both hosts.

What the notes closed on

The hour ended with a rejected upload list, a clean synthetic subset, and two exit codes written on separate lines. The model's paragraph about the verifier was not stored. The initialed manifest was.

Readers who already have a free model seat and a free server option can run this checker on a throwaway repository first. Only a clean synthetic subset should be aimed at the remote host. If current docs disagree with these availability assumptions, the docs win.

Outside the Assertion: The Retry Budget a Pairing Hour Refused to Inline

Quinn Sun — Sun, 11 Oct 2026 20:42:37 +0000

The contract check failed once, then passed on a local rerun. The model in the pairing window offered a helper that retried the assertion three times and returned the first success. A senior on the same call did not accept the patch. The hour ended with a receipt file, a required seed, and a rule that the test process itself may not retry.

The scene is a reconstructed pairing hour, written as a worked example. It is not a production incident report, and it does not claim measured flake rates.

What the screen showed

A service under test exposes GET /orders/:id. The fixture inserts one order, the client waits 200 milliseconds, and the assertion expects status to equal packed. On the first run the body still said picking. The second run, started from the same shell without a clean database, returned packed and exit code 0.

That sequence is a familiar trap in a pairing room. A green rerun feels like proof. Often it is only leftover rows, a warm cache, or a worker that finished between the two commands.

The senior asked three questions before anyone edited the helper.

Which process owned the clock: the test, the app, or the database?
What was the seed, and was it written down before the second run?
If a second machine runs the same command once, does it fail the same way?

The model answered the first question with a guess about eventual consistency. No log line supported that phrase. Unsupported wording stayed out of the patch.

Dead end one: a retry inside the assertion

The first proposal wrapped the check in a loop and treated the first success as the result. The patch looked small. It also moved a timing policy into the test file, where the next editor could raise the limit and call the suite stable.

async function expectPacked(client, id) {
  let last = null;
  for (let attempt = 1; attempt <= 3; attempt += 1) {
    last = await client.get(`/orders/${id}`);
    if (last.body && last.body.status === "packed") return last;
    await new Promise((resolve) => setTimeout(resolve, 200 * attempt));
  }
  throw new Error("status never reached packed");
}

The senior rejected that helper. A retry inside the assertion hides the attempt count from CI and from the pairing notes. The loop also couples a pass to whichever process still holds the previous row. Leftover state is not a contract.

Dead end two: stop because the model was free

The second proposal was to end the hour because a free model cannot be deterministic. That move throws away the only evidence on the table: one failing body, one passing body, and no seed file.

Model tier does not explain a status field. A free model and a paid model can both invent the same wrapper. The decision belongs to the harness. Free access can draft the next patch. It is not a reason to trust a green rerun, and it is not a reason to ignore a real mismatch.

The decision the hour kept

The senior kept a smaller rule. The pair wrote it down before the next edit.

The test process runs the check once.
A retry, if anyone still wants one, is a separate command with a written budget.
Each attempt records the seed, the command, the exit code, and a hash of the output.
A second process must reproduce the failure before application code changes.
The model may propose the receipt shape. It may not mark the receipt optional.

The script that follows is a proposed example for a scratch repository. It was not executed against a live service for this draft. Keep the file as retry-receipt.mjs so top-level await is valid, or set "type": "module" in a local package.json.

// retry-receipt.mjs
// Proposed example. Not a recorded production log.
import { spawn } from "node:child_process";
import { createHash } from "node:crypto";
import { writeFileSync } from "node:fs";

const budget = Number(process.env.RETRY_BUDGET || "1");
const seed = process.env.ORDER_SEED || "seed-not-set";
const cmd = process.env.CHECK_CMD || "node check-order.mjs";

function sha(text) {
  return createHash("sha256").update(text).digest("hex");
}

function runOnce(attempt) {
  return new Promise((resolve) => {
    const child = spawn(cmd, {
      shell: true,
      env: { ...process.env, ORDER_SEED: seed, ATTEMPT: String(attempt) }
    });
    let out = "";
    child.stdout.on("data", (chunk) => { out += chunk; });
    child.stderr.on("data", (chunk) => { out += chunk; });
    child.on("close", (code) => {
      resolve({
        attempt,
        seed,
        exit: code,
        body_sha256: sha(out),
        tail: out.slice(-240)
      });
    });
  });
}

const attempts = [];
for (let i = 1; i <= budget; i += 1) {
  attempts.push(await runOnce(i));
  if (attempts.at(-1).exit === 0) break;
}

const receipt = {
  decided: "retry_lives_outside_the_test",
  budget,
  seed,
  attempts,
  keep: attempts.some((row) => row.exit !== 0)
    ? "investigate_before_patch"
    : "still_not_proof"
};

writeFileSync("retry-receipt.json", JSON.stringify(receipt, null, 2));
console.log(JSON.stringify({ path: "retry-receipt.json", keep: receipt.keep }));

The companion check fails closed when the seed was never set. Scoring a rerun without a seed is the failure mode the hour was trying to kill.

// check-order.mjs — single-shot sketch
const seed = process.env.ORDER_SEED;
if (!seed || seed === "seed-not-set") {
  console.error("seed missing; refusing to score a rerun");
  process.exit(2);
}
console.error(JSON.stringify({ seed, status: "picking" }));
process.exit(1);

Commands the notes kept

Run the receipt writer on the dirty shell first. Then run it again with a clean environment and a budget of one. The second command is the comparison, not a victory lap.

export ORDER_SEED="order-17-worker-not-run"
export RETRY_BUDGET="2"
export CHECK_CMD="node check-order.mjs"
node retry-receipt.mjs

env -i PATH="$PATH" HOME="$HOME" \
  ORDER_SEED="$ORDER_SEED" RETRY_BUDGET=1 \
  CHECK_CMD="node check-order.mjs" \
  node retry-receipt.mjs

How to score the two files

A missing seed must exit 2. That exit means the harness refused to score, not that the product failed.
Budget 2 on a dirty shell may show a later pass. That pass does not authorize an application edit.
Budget 1 on a clean process is the row that matters.
If the clean process fails with picking and the dirty shell passed only on attempt 2, keep leftover state as the finding.
If both processes fail the same way, keep a missing worker step as the finding, and only then ask for a patch.
A receipt whose keep field is still_not_proof is still not approval. Exit code 0 does not upgrade itself.
Do not commit retry-receipt.json when the tail holds tokens, cookies, or customer payloads. Keep the hash. Drop the tail.

Replace the sketch's hard-coded picking line with a real HTTP call only after those seven checks pass on the stub. A live call added too early recreates the original argument in a noisier form.

Where free model access and a free server sit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is the assistant named for this outreach. The operator describes it as an open-source project with free model access and a free server option. Those two availability claims are operator-supplied context for the workflow. This draft does not name a model, a token quota, a machine size, or a duration. Figures like that go stale, and an older number would be the wrong fact to print.

Free model access, in this hour, is a way to draft the receipt schema and the failing-closed check without treating a paid balance as the reason to stop. A free server is a place to run the second process so the pass is not cached on the laptop that just went green. That split is the senior's third question. It is not a product verdict.

A free server is not a production twin. Free model access does not make a generated retry loop correct. If today's project docs do not list those options, drop the product and keep the receipt rule. The rule does not need the assistant.

When the docs do list them, a restrained use is to paste retry-receipt.json and ask which fields break the rule list. The human still sets keep.

Who should not use this

Do not point CHECK_CMD at untrusted input. The sample sets shell: true for a local scratch run, which is a sharp edge.
Do not share a database between the two processes. A second process on the same data volume recreates the trap.
Do not treat two runs as a flake-rate study. No sample size is claimed here.
Teams that need attested hardware, signed builds, or regulated evidence should not substitute a free server session for that evidence.
Teams whose worker is intentionally asynchronous should put the retry budget in the worker contract, with an owner and an alert, not inside expectPacked.
Reviewers who want a second green run to close the ticket should use another workflow. This one is built to refuse that closure.

What stayed in the notes

The model offered a short paragraph about resilience. The senior did not keep the paragraph. The kept decision was the receipt: seed required, budget outside the test, second process on a separate environment, and no application edit until both receipts name the same failure.

That split is enough for a pairing hour. It will not replace a load test, a trace, or a product decision about retries at the edge. It will stop a helper from laundering one lucky rerun into a merged patch.

Readers comparing assistants can check MonkeyCode's current notes on free model access and the free server option, then run the scratch script on a fixture they already trust. If those notes disagree with this paragraph, believe the notes.

The Pairing Hour Kept the Replay Exit Code After the Model Edited the Fixture

Quinn Sun — Sat, 10 Oct 2026 17:08:47 +0000

Liquid syntax error: Unknown tag 'endraw'

Second-Process Probes: What a Pairing Hour Kept After the Session Reset

Quinn Sun — Fri, 09 Oct 2026 13:15:56 +0000

The scene below is a worked example, not a field report. A missing currency code is slipping into a warehouse export, and the Friday cutoff leaves one pairing hour. One engineer drives the keyboard. A senior refuses another model summary, even though the assistant has already offered a tidy rewrite of normalizeRow.

Tidy is not evidence. The hour needs a file that can fail.

The constraint on the table

The change looks small, which is why it is dangerous. Empty currency has to fail closed. The SKU has to survive. Amount parsing has to stay untouched unless a separate probe says otherwise. A green sentence in chat can satisfy none of those constraints.

Before the next prompt, the senior writes the hour's rule on the ticket. An accepted edit needs a probe file in the repo, an expected exit code, and a rerun in a new process. If the session vanishes, the files still have to be enough to replay the decision.

Copy the commands, then break them on an old tree before trusting them. An unexecuted sample is only a paragraph with fences.

What the senior asked for instead of a summary

The pairing notes do not store confidence. They store three answers, each as a path or a command.

The input that must fail closed, checked into probes/fixtures/missing-currency.json.
The assertion that must stay strict, in probes/missing-currency.test.js.
The command that must run outside the chat: node --test probes/missing-currency.test.js.

A short answer with a path beats a long answer with no file. The senior marks a question open until the path exists and a human has opened it. That sounds slow. It is faster than reviewing a rewrite whose only proof is tone.

Dead end: prose that could not fail

The first reply explains the guard clearly. It names an error string and describes the branch. Nothing in that reply has an exit code.

The senior parks the explanation and asks for a failing test on the current function. Explanation can be pasted into a status update while normalizeRow stays wrong. A probe cannot. The hour does not spend a second prompt polishing sentences that cannot go red.

Dead end: the test got easier

The second reply adds a test, then edits the fixture. The bad row is gone. The suite goes green because the case has left the building.

The diff is small enough to miss in a hurry. The senior rejects it for a mechanical reason, not a vibe. The assertion no longer rejects an empty currency. A pairing hour that cannot see that edit will keep shipping softer tests, and the export job will keep the original bug.

Dead end: the session reset

The third pass is the first useful one. The probe fails on the old function. The guard lands. The same probe passes. Then the free server session resets, and the scrollback goes with it.

Nobody reconstructs the chat from memory. A remembered checkmark is treated as lost. The working tree still holds the probe card, the test, and the function diff. That is the only record the senior is willing to trust. Resume starts at those files, not at a retold summary.

The probe card

The card is a YAML note beside the test. It is a decision slip, not a platform. Drop fields that a team will not actually read.

id: billing-export-normalize-014
question: Does a missing currency code fail closed?
fixture: probes/fixtures/missing-currency.json
command: node --test probes/missing-currency.test.js
expect_exit: 0
old_code_must_fail: true
reject_if:
  - the assertion accepts an empty currency
  - the bad row disappears from the fixture
  - amount parsing changes without its own probe
rerun: new process, after every assistant edit

The fixture stays synthetic. Real invoices do not belong in a prompt or a sample repo.

{ "amount": "10.00", "currency": "", "sku": "A1" }

The probe is intentionally dull. Dull probes survive review.

import assert from "node:assert/strict";
import test from "node:test";
import { readFileSync } from "node:fs";
import { normalizeRow } from "../src/normalize-row.js";

test("missing currency fails closed", () => {
  const row = JSON.parse(
    readFileSync(new URL("./fixtures/missing-currency.json", import.meta.url))
  );
  assert.equal(row.currency, "");
  assert.throws(() => normalizeRow(row), /currency/);
});

The function sketch below is an unexecuted example. It exists so the probe has a target, not so a reader can paste it into production.

export function normalizeRow(row) {
  if (!row || !row.currency) {
    throw new Error("currency required");
  }
  return {
    sku: row.sku,
    amount: row.amount,
    currency: row.currency,
  };
}

Commands that kept the hour honest

Run the probe on the old code first. A failure is the signal that the bug is still real. A pass on unmodified code means the probe is already too weak, and the rewrite should wait.

node --test probes/missing-currency.test.js
echo "before_exit=$?"

After the edit, open a new shell. Do not accept a pass that only the assistant pane can see.

node --test probes/missing-currency.test.js
echo "after_exit=$?"
git diff -- probes/missing-currency.test.js probes/fixtures/missing-currency.json src/normalize-row.js

Review the diff with a fixed list, not a fresh debate.

The fixture still has "currency": "".
The assertion still throws on that row.
amount handling is unchanged, or a second probe covers the change.
The card's command still matches the command just run.

A small table stops the hour from sliding back into chat scores.

Evidence in hand	What the senior did
Prose only	Parked the rewrite
Green result only inside the same chat	Parked it
Fixture or assertion softened	Rejected the edit
Session gone and probe file missing	Rebuilt the probe before any further draft
Old code failed, new process printed `after_exit=0`, assertion unchanged	Kept the patch, discarded the transcript

The decision that survived

The kept decision is the probe card plus the second-process exit code. The assistant summary does not survive review. The in-chat pass does not survive review. The story of what the lost scrollback probably said does not survive review either.

That decision is narrow on purpose. One probe does not cover unknown currency codes, header drift, or a caller that trims fields before normalizeRow runs. The hour refuses a broader claim. It refuses to merge a rewrite that has not failed in the open, on disk, and then passed again outside the chat.

A pairing script other hours can reuse

The script is short enough to pin in a ticket. It does not mention a vendor, because the rule is about evidence.

Write the failing input before asking for a rewrite.
Confirm the probe fails on unmodified code.
Let the assistant draft. Limit the ask to the function and the probe.
Read the diff for a softer assertion before reading the prose.
Rerun in a new process. Record before_exit and after_exit on the ticket.
If the session resets, resume from the files. Do not resume from memory.

Teams that skip step 2 will eventually bless a probe that never failed. Teams that skip step 5 will bless a pass that lived only in a pane. Both skips look like speed until the export job runs again. The senior's job in the hour is to keep those two steps from being negotiated away.

Where a free assistant fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In this worked example, the draft and the first failing probe come from MonkeyCode's free model access, used with the free server option. Those two availability claims are operator-supplied for this draft. They are not a token quota, a model name, a hardware size, an uptime figure, or a statement that the free option is permanent.

The assistant earns a place in two spots. It produces a first sketch of normalizeRow quickly, and it proposes a probe the senior can reject. It does not earn the merge. A reset makes the boundary obvious. A free server session is a convenience, not the system of record. The repo is.

Readers who want the same split can point the probe command at any assistant they already use. Anyone comparing this free model access or free server option should read MonkeyCode's current repository and docs, then rerun the probe on their own tree. The workflow on this page does not require an account to be useful, and a sprint should not be planned around numbers this draft did not verify.

Who should skip it

Some rooms should not borrow this pattern.

Incident response that needs a contractual SLA, a pinned build, or retained prompts for audit.
Hours where no one can execute node --test outside the assistant.
Plans whose only copy sits in a free session that can disappear.
Reviews that would treat one green probe as a substitute for reading the diff.

A free coding server is also the wrong place for production credentials, customer extracts, or live warehouse files. The fixture here is fake. Keep it that way. If the only way to explain the bug is to paste a real row, stop and build a synthetic one first.

Limitations worth keeping on the ticket

No token allotment, latency, or pass rate appears here because none was measured for publication. Free access can change after this page is written. Check the current docs before a sprint depends on it. A worked example that invents a quota would be the same class of failure as a softened assertion.

The card catches a specific cheat: a test or fixture edited so the bug can no longer fail. It misses a rewrite that passes this probe and breaks a caller the card forgot. Unlisted questions stay open. Add a probe per kept question, or leave the question marked open on the ticket.

Node's built-in runner is only the sample command. A Python or Go shop can swap the command and keep the slip. The decision does not depend on a product feature, and it does not get stronger because a chat window says the suite is green. Run it again in a clean process. Then keep the exit code, not the story.

Three Dead Ends in an AI Pairing Session, and the Replayable Claim Ledger That Ended It

Quinn Sun — Thu, 08 Oct 2026 09:27:30 +0000

Three Dead Ends in an AI Pairing Session, and the Replayable Claim Ledger That Ended It

The bug was small: a webhook handler that double-charged on retried deliveries. Two engineers sat down with a coding assistant, and inside twenty minutes the model had produced a patch, an explanation, and four confident sentences about backward compatibility.

The tests passed. The senior still would not merge. Not because the patch looked wrong, but because three of the four claims in the explanation had never been executed by anything.

This is the story of the dead ends that followed, and the one artifact the session kept: a claim ledger that turns model prose into rows a machine can falsify.

The setup: one handler, one assistant, four claims

The reproduction was ordinary. A signed webhook arrived twice because the sender retried after a slow 200. The handler wrote a ledger row on the first delivery and again on the second, so the customer saw two charges.

The assistant proposed an idempotency key, a lookup before insert, and a small migration. Its explanation contained four assertions:

The change is backward compatible with rows written before the migration.
Retried deliveries with the same key are dropped.
Retried deliveries with different keys still process.
The extra read adds no meaningful latency to the happy path.

Claim 2 was covered by a new unit test. Claims 1, 3, and 4 were not covered by anything. They were prose.

Dead end 1: reviewing the transcript instead of the diff

The first pass was to read the explanation line by line and nod. The prose was well written. It referenced the right files, used the right nouns, and described the migration in the right order.

The problem is that a transcript is unfalsifiable by construction. A reviewer can agree with a sentence, but agreement is not a verdict. Nothing in the session could turn claim 1 into a true-or-false value, so the discussion looped: the senior asked for evidence, the assistant rewrote the same sentence with more confidence.

Agreement is not evidence. A claim becomes evidence only when a command exits non-zero on it.

Dead end 2: re-rolling the prompt

The second pass was to ask again with sharper wording. The engineers added constraints, asked for tests in the same response, and pasted the failing payload.

The assistant returned a different explanation with a functionally identical diff. Three re-rolls produced three phrasings of claim 4, each with a different adjective and none with a measurement. The session burned tokens and converged on nothing, because the prompt was never the bottleneck.

Re-rolling changes how a claim is worded. It does nothing about whether the claim is checked.

Dead end 3: reading a passing suite as evidence

The third pass was the tempting one: the new unit test for claim 2 passed, so the suite was green, so the patch was good.

That inference is where most pairing sessions go wrong. A suite proves exactly the set of assertions inside it. Claim 2 was inside the suite. Claims 1, 3, and 4 were outside it, and no amount of green pixels reaches them.

The senior wrote the four claims on a whiteboard and drew a box around the one with a test under it. The other three were the actual review surface.

The decision that survived: a claim ledger

The kept decision was small and boring. Before the next patch, the pair wrote every assertion the assistant made into a JSONL file, one row per claim, each with a command that would fail if the claim were false.

What goes in a row

id — a stable label so the same claim can be tracked across sessions.
claim — the sentence, quoted, not paraphrased.
cmd — the command that decides it. Exit code is the verdict.
expect_exit — usually 0; 1 when the claim predicts a failure.
expect_stdout_includes — an optional substring, for claims about output rather than status.

The runner

The version the session ended with is short enough to read in one sitting. It replays each row against a pinned revision and appends a verdict to an output file.

#!/usr/bin/env node
// ledger.mjs — replay pairing-session claims against one fixed revision.
// Node 20+. Usage:
//   node ledger.mjs claims.jsonl --rev "$(git rev-parse HEAD)" --out ledger.out.jsonl
import { readFile, writeFile, appendFile } from "node:fs/promises";
import { execFile } from "node:child_process";
import { promisify } from "node:util";

const pexec = promisify(execFile);
const flag = (name, fallback) => {
  const i = process.argv.indexOf(`--${name}`);
  return i === -1 ? fallback : process.argv[i + 1];
};

const claimsPath = process.argv[2];
const rev = flag("rev", "unpinned");
const out = flag("out", "ledger.out.jsonl");
const timeoutMs = Number(flag("timeout", "60000"));

const rows = (await readFile(claimsPath, "utf8"))
  .split("\n")
  .map((l) => l.trim())
  .filter(Boolean)
  .map((l) => JSON.parse(l));

await writeFile(out, "");
let falsified = 0;

for (const row of rows) {
  const started = Date.now();
  const result = await pexec("/bin/sh", ["-c", row.cmd], {
    cwd: row.cwd ?? ".",
    timeout: timeoutMs,
    maxBuffer: 4 * 1024 * 1024,
    env: { ...process.env, CI: "1" },
  }).then(
    (v) => ({ code: 0, stdout: v.stdout, stderr: v.stderr }),
    (e) => ({
      code: e.code ?? 1,
      stdout: e.stdout ?? "",
      stderr: e.stderr ?? String(e),
    })
  );

  const expected = row.expect_exit ?? 0;
  let verdict = result.code === expected ? "holds" : "falsified";
  if (verdict === "holds" && row.expect_stdout_includes) {
    verdict = result.stdout.includes(row.expect_stdout_includes)
      ? "holds"
      : "falsified";
  }
  if (verdict === "falsified") falsified += 1;

  await appendFile(
    out,
    JSON.stringify({
      id: row.id,
      claim: row.claim,
      rev,
      cmd: row.cmd,
      expect_exit: expected,
      exit: result.code,
      verdict,
      ms: Date.now() - started,
      stderr_tail: result.stderr.slice(-400),
    }) + "\n"
  );
  console.log(`${verdict.padEnd(10)} ${row.id}  ${row.claim}`);
}

console.log(`\n${rows.length - falsified} hold, ${falsified} falsified @ ${rev}`);
process.exitCode = falsified > 0 ? 1 : 0;

A claims file from the webhook session

{"id":"c1","claim":"rows written before the migration still read","cmd":"node --test test/migration.legacy.test.js","expect_exit":0}
{"id":"c2","claim":"same key on retry is dropped","cmd":"node --test test/webhook.idempotency.test.js","expect_exit":0}
{"id":"c3","claim":"different key still processes","cmd":"node --test test/webhook.distinct-key.test.js","expect_exit":0}
{"id":"c4","claim":"happy path adds no second write","cmd":"node --test test/webhook.single-write.test.js --reporter=tap","expect_stdout_includes":"inserts: 1","expect_exit":0}

Running it, and pinning the revision

git rev-parse HEAD                      # the revision every verdict is tied to
node ledger.mjs claims.jsonl --rev "$(git rev-parse HEAD)" --out ledger.out.jsonl

grep -c '"verdict":"falsified"' ledger.out.jsonl   # count the misses
git diff | sha256sum                                 # fingerprint the patch itself

The point of writing the revision into every row is that a verdict is only meaningful for one tree. A ledger from last Tuesday says nothing about today's branch, and pretending otherwise is how a harness quietly becomes a ritual.

What the ledger does not prove

A green ledger is a narrow instrument, and the session was explicit about its edges.

It proves only the claims someone wrote down. A missing row is invisible, not verified.
It says nothing about design quality. A patch can satisfy four claims and still be the wrong shape for the codebase.
It cannot check claims that need production data, load, or human judgment to settle.
A row's command is arbitrary shell. Running a command list derived from a model response is a real supply-chain risk; it belongs in a container or a disposable VM with no credentials mounted, not on a laptop with cloud keys in the environment.
Repeated passes on an unchanged tree are waste. The ledger is a decision tool, not a watch loop.

Where free model access and a free server actually help

The session above is token-hungry in a specific way: many short exchanges, each producing claims that need checking. That pattern is a poor fit for per-seat pricing and a good fit for trying a tier before committing budget.

That is where MonkeyCode entered the picture for this account. The project advertises free access to its model pool and a free server option, and the operator states the free allowance is on the order of ten million tokens. Free-tier numbers move, so treat that as a starting figure to verify against the project's current terms rather than a planning constant.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What made it relevant was not the price. It was that the ledger loop needs a place to run commands that should not touch a developer's main environment, and a free server option covers that without a procurement conversation. The ledger itself is plain Node and would run anywhere.

A fit table

Situation	Hosted free tier fits	Hosted free tier does not fit
Exploring claim-ledger workflow for the first time	Yes
Short, high-frequency exchanges that generate many testable claims	Yes
Data that cannot leave a controlled network		Yes
Work needing guaranteed throughput or uptime commitments		Yes
Team already standardized on a self-hosted runner		Yes

Who should not use this approach

The claim ledger is not for everyone, and the honest answer is that it costs more up front than it saves for some work.

Teams shipping throwaway prototypes: writing rows slows a patch that will be deleted next week.
Compliance-bound work where data residency is fixed and cannot be changed by a tier choice.
Anyone who would read a green ledger as merge approval. It is an input to review, never a replacement for it.
Workloads that need contractual uptime. A free tier is the right place to experiment, not the right place to promise availability.

What the senior kept

The kept decision was not the model, the tier, or the patch. It was the row format: a sentence, a command, an expected exit code, and a revision stamp.

One soft note for teams who want to try the loop before buying anything: free model access and a free server are enough to run a first ledger, and the JSONL file is the part worth carrying to whatever infrastructure comes next.

DEV Community: Quinn Sun

A Pairing Hour on Retries: The Failure Taxonomy We Kept and the Three Loops We Deleted

The question that reframed the hour

The taxonomy we kept

Dead end 1: one wrapper to retry them all

Dead end 2: retry at the HTTP client

Dead end 3: dedupe on a payload hash

The decision: retry the job, not the call

Key and ledger

Classifier and policy

The runner

Tests you can run now

Fault-injection matrix

Where the free tier actually helped

Limitations and who should skip this

The closing checklist

The Pairing Hour That Replaced a Webhook Queue With 60 Lines of Admission Control

The constraint nobody wrote down

Question 1: what happens when forty PRs land in ninety seconds?

Question 2: what does the endpoint do when it is slow rather than down?

Question 3: can you reconstruct a failure from the log alone?

The artifact: an admission gate, not a queue

Reproducing the burst without a network

The decision table we kept

Where the free tier actually enters the design

Limitations and who should not use this

What the senior kept

The Pairing Hour That Split the Review Bot: Browser Prefilter, Server Verdict, One Token Budget

Three questions written on the whiteboard

Dead end 1: move the whole bot into the browser

Dead end 2: send every diff to the shared endpoint

Dead end 3: cache verdicts by prompt hash

The decision kept: a split, plus a budget checked before enabling

Where the free tier entered the session

Limitations and who should skip this

A checklist for the next pairing hour

A Pairing Hour on Token Budgets: Three Dead Ends Before We Kept the Usage Ledger

The question that started the hour

Dead end 1: the dashboard aggregate

Dead end 2: estimating with len(text) / 4

Dead end 3: wall-clock time as a cost proxy

The artifact we kept: a usage ledger and a fail-closed gate

The canary that keeps the ledger honest

The decision table the pair wrote down

Where a free tier and a free server actually fit

Limitations and who should skip this

Time the Cold Start Before You Trust a Free Server With Your Review Bot

The question nobody in the room could answer

Five questions the senior wrote on the whiteboard

Two dead ends before the first line of code

Where the free model access and the free server fit

Artifact one: a service that describes its own state

Artifact two: idle, then measure, then print CSV

The decision table, written before the run

What was kept, and what was dropped

Limitations, and who should skip this

A Pairing Hour on a Timezone-Flaky Test: Three Dead Ends and the One Seam We Kept

The first ten minutes were questions, not code

Reproduce before you read the stack trace

Dead end 1: add retries to the test

Dead end 2: a global fake timer

Dead end 3: a production freeze flag

The decision we kept: one clock seam

What the seam costs

Where a free model and a disposable server fit the hour

A decision table for the next flake

Limits, and who should skip this

What went into the pairing notes

Red Before Green: A Probe Gate for Free-Tier Model Output

Three dead ends that ate the hour

The question that ends an argument in one run

A probe that has never been red is a description, not a test

The artifact: a probe gate in about sixty lines

Running it

A log shape worth copying

Routing questions in a pairing hour

Where free model access and a free server change the economics

Limitations, and who should skip this

What the hour kept

The Upload Set a Senior Signed Before Any Free-Server Run

Dead end 2: estimating with `len(text) / 4`