The Retry Budget Needed a Pairing Signature, Not More Jitter

The junior had already pasted a p95 screenshot into the chat when the senior sat down. The graph was ugly. The uncommitted diff was uglier: a retry wrapper, jitter, a longer timeout, and a comment that promised the latency would settle. The senior did not review the jitter. They asked for the last request id and for a classification of the failure, not another generated backoff story.

This write-up reconstructs that pairing desk. It is a worked example, not a report of a named outage, a customer, or a benchmark. The useful part is the freeze they kept.

What was actually on the table

The service was a small Node handler behind an internal gateway. Clients called POST /v1/jobs and waited for 202 or a typed error. After a deploy, dashboards showed more client-side timeouts. The junior's first move was familiar: wrap fetch and retry.

That move is how retry storms start. A retry is a loan against someone else's CPU. Pairing had to decide whether the loan was allowed.

The senior pinned the conversation on three numbers:

• the current per-attempt timeout
• the current maximum attempts
• the total wall-clock budget the client could burn

Until those three numbers lived in a freeze file, no model output was a candidate patch. Generated retry wrappers are cheap to obtain in 2026. Owning the queue they create is not.

Questions the senior asked out loud

The senior did not ask whether the model felt confident. They asked for evidence the pairing could re-run.

1. Whether the client was aborting at 2s while the handler still returned at 2.4s.
2. Whether the gateway was returning 504 while the handler succeeded.
3. Whether a 429 from a neighbor service was being retried as if it were a transient blip.
4. Whether anyone had checked that retries were amplifying the original queue.

The junior did not have those answers. The chat window already contained a generated wrapper.

Dead end 1: retry every fetch

The first pasted helper looked like this:

// example only — rejected in pairing
async function fetchWithRetry(url, options, attempts = 5) {
  let lastError;
  for (let i = 0; i < attempts; i++) {
    try {
      return await fetch(url, options);
    } catch (err) {
      lastError = err;
      await new Promise((r) => setTimeout(r, 200 * 2 ** i));
    }
  }
  throw lastError;
}

Pairing rejected it for mechanical reasons:

• attempts defaulted to 5 with no freeze file
• every throw retried, including programmer errors wrapped as exceptions
• backoff ignored the product's 2s UX budget

The senior had the helper deleted from the working tree before the next pass. Style was not the argument. The missing classifier was.

Dead end 2: a global timeout bump

The second pass hid the latency:

// example only — rejected in pairing
const TIMEOUT_MS = 10_000;

Ten seconds made any dashboard that measured success-after-wait look calmer. It made the queue worse for everyone else. Pairing recorded the dead end in one line: timeout inflation is not a classification.

A longer wait is a product change. It needed a pairing signature, not a silent constant edit.

Dead end 3: a suite that stayed green

The third pass kept the 2s timeout and retried only on TypeError. Unit tests stayed green because they stubbed fetch to succeed on the second call. Nobody asserted that the budget could not grow.

That was the hole. The suite rewarded a retry the freeze would have forbidden. Green tests without a freeze are how an agent-shaped patch sneaks a product change through pairing.

The artifact pairing kept

They added a tiny freeze file and a classifier. The freeze is the pairing contract. The classifier is the only thing allowed to decide whether a retry is even legal.

{
  "$comment": "retry-budget.lock.json — bump only with pairing sign-off",
  "endpoint": "POST /v1/jobs",
  "perAttemptTimeoutMs": 2000,
  "maxAttempts": 2,
  "totalBudgetMs": 2500,
  "retryable": ["GATEWAY_TIMEOUT", "HTTP_429", "CONNECT_RESET"],
  "notRetryable": ["HTTP_400", "HTTP_401", "HTTP_403", "HTTP_404", "HTTP_422", "HANDLER_5XX"]
}

HANDLER_5XX is not retryable on this desk. If the handler is on fire, more attempts feed the fire. Gateway timeouts and 429s are different. They can represent shed load, and a single retry can be rational.

The classifier is deliberately boring:

// timeout-classifier.js — example harness
function classify({ httpStatus, timedOut, connectReset }) {
  if (timedOut) return "GATEWAY_TIMEOUT";
  if (connectReset) return "CONNECT_RESET";
  if (httpStatus === 429) return "HTTP_429";
  if (httpStatus >= 500) return "HANDLER_5XX";
  if (httpStatus >= 400) return `HTTP_${httpStatus}`;
  return "OK";
}

function mayRetry(code, freeze) {
  return freeze.retryable.includes(code);
}

function budgetFor(attempt, freeze) {
  if (attempt >= freeze.maxAttempts) {
    return { allowed: false, reason: "maxAttempts" };
  }
  const used = attempt * freeze.perAttemptTimeoutMs;
  if (used >= freeze.totalBudgetMs) {
    return { allowed: false, reason: "totalBudgetMs" };
  }
  return { allowed: true, timeoutMs: freeze.perAttemptTimeoutMs };
}

module.exports = { classify, mayRetry, budgetFor };

A test then pins the freeze. If a model or a human raises maxAttempts or totalBudgetMs, CI fails until pairing writes a one-line sign-off in the lock comment.

// retry-budget.test.js — example
const assert = require("node:assert/strict");
const freeze = require("./retry-budget.lock.json");
const { classify, mayRetry, budgetFor } = require("./timeout-classifier");

assert.equal(freeze.maxAttempts, 2);
assert.equal(freeze.perAttemptTimeoutMs, 2000);
assert.equal(freeze.totalBudgetMs, 2500);

assert.equal(
  mayRetry(classify({ httpStatus: 504, timedOut: true }), freeze),
  true
);
assert.equal(
  mayRetry(classify({ httpStatus: 500, timedOut: false }), freeze),
  false
);
assert.equal(
  mayRetry(classify({ httpStatus: 400, timedOut: false }), freeze),
  false
);

assert.equal(budgetFor(0, freeze).allowed, true);
assert.equal(budgetFor(2, freeze).allowed, false);

console.log("retry freeze held");

Run it locally:

node retry-budget.test.js

The pairing log was equally small. They kept it next to the lock so the next reviewer did not have to reconstruct the argument from chat.

# pairing-log.txt
2026-09-18
decision: keep maxAttempts=2, totalBudgetMs=2500
rejected: universal fetch retry
rejected: TIMEOUT_MS=10000
rejected: tests that stub success on attempt 2 without freeze asserts
sign-off: senior + junior

Decision table the desk used

Observation	Legal move	Illegal move
Client abort at 2s, handler 2.4s	Measure the handler; consider UX copy	Silent 10s timeout
Gateway 504, handler 200	One retry if the freeze allows	Five jittered retries
Handler 500	Page the handler; no retry	Backoff loop
429	Honor Retry-After or the freeze	Ignore 429 and hammer
Tests green, freeze changed	Fail CI	Merge as a refactor

The table is the pairing. Models may propose code that respects it. They may not edit the table quietly.

Where a spare runner fitted

Once the freeze existed, pairing still needed a machine that was not also compiling the main app.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that, as supplied for this write-up, offers free model access and a free server option. The desk used that combination narrowly: classifier tests ran on the spare server so the laptop could keep rebuilding, and a free model was asked only to propose a classifier patch that did not edit retry-budget.lock.json. If a suggestion touched the lock, pairing dropped it. Desks that already freeze a budget can park the same harness on that free server. Desks with no freeze should not start by asking a model for jitter.

The product is optional. The freeze is not. Remove every product name from this article and the pairing still holds: classify, freeze, then generate.

#!/bin/sh
# pairing-gate.sh — example
set -e
node retry-budget.test.js
if ! git diff --exit-code -- retry-budget.lock.json; then
  echo "lock file changed; pairing signature required" >&2
  exit 1
fi

Make it executable and run it before anyone looks at a generated diff:

chmod +x pairing-gate.sh
./pairing-gate.sh

That last check is the whole gate. If git diff --exit-code -- retry-budget.lock.json fails, pairing is not looking at a classifier. It is looking at a budget raid.

Limitations

The freeze does not know about mesh-level retries, browser connection pools, or a mobile client's own timeout. It does not prove p95. It only forbids silent widening.

Status-code classification is crude. A 500 from a truly idempotent read is a different animal than a 500 from POST /v1/jobs that may have created a record. This desk treated job creation as not retryable on handler 5xx. Another desk might split read and write. They would need a different lock.

A spare remote server will not share production RTT. Use it to run the contract, not to certify latency. Model output remains untrusted. The gate is git diff on the lock plus the tiny Node test. If those are skipped, the workflow is theater.

Who should not use this

• Incident commanders mid-page who need a rollback, not a new classifier
• Teams whose retries already live in Envoy, an SDK, or a mesh policy they do not own
• Anyone hoping a model will make p95 look better without a freeze
• Suites that cannot run a 20-line Node script in CI

Those groups need a different artifact. This one is for a pairing desk that is about to accept an agent-shaped retry and wants a reason to say no.

What they kept

They kept maxAttempts at 2. They kept the 2.5s total budget. They kept HANDLER_5XX off the retry list. The jitter poem did not merge.

The junior still wanted the graph to look kinder. The senior agreed the graph mattered. The signature on the lock was how they stayed engineers while a model sat in the room.