When a Free Model Must Not Classify Retryability

On a Monday deploy, a refund agent started auto-retrying ledger writes. Each 503 response was labeled retryable by a helper model. The queue doubled while the ledger stayed dark.

Customers then saw duplicate authorization holds by noon. The model had treated unavailability as safe to replay. Retryability was never a natural-language task for agents.

Retryability is a control-plane decision

A retry changes money, inventory, and on-call load. The classifier sits on the hottest failure path. Latency, determinism, and auditability all matter on that path.

A free model can draft fixtures and comments. It must not vote on any production retry. The live vote belongs to pinned status rules.

Tool-calling agents make this failure mode common. The model already chooses arguments for an HTTP tool. Teams then let it also label the tool error. That second vote is how duplicate POSTs are born.

Red flags before the model is wired in

Stop the integration if any item below is true.

• The outbound call mutates money, inventory, or access rights.
• The upstream documents Retry-After or required idempotency keys.
• Duplicate execution is user-visible, billable, or legally binding.
• The helper model is not pinned, logged, or replayable.
• Backoff state lives only on an ephemeral free server.
• On-call cannot explain a retry without reading a prompt.

A yes on any row means keep the model off the path. Incident chat is not an exception to that rule. Comfort text can wait until the worker has failed closed.

What the classifier is allowed to read

The live classifier may read only transport facts.

• HTTP status code from the failed upstream attempt
• Retry-After header when present and numerically valid
• A caller-supplied idempotency key for mutating methods
• Static method allowlist covering GET and keyed PUT
• Per-route maximum attempt budget stored in config

The classifier must not read the response body as prose. It must not send that body to a model for labeling. Body text stays untrusted on the hot path. Status codes remain the only allowed retry contract.

Artifact: a pinned retry classifier

The following Node module is a proposal for local tests. It is not a production SLA for vendors. Teams should adapt the status map for each vendor.

// retryability.mjs
export const RETRYABLE_STATUS = new Set([408, 425, 429, 502, 503, 504]);

export const MUTATING = new Set(["POST", "PATCH", "DELETE"]);

export function parseRetryAfter(header, nowMs = Date.now()) {
  if (header == null || header === "") return null;
  const raw = String(header).trim();
  const asInt = Number.parseInt(raw, 10);
  if (Number.isFinite(asInt) && String(asInt) === raw) {
    if (asInt < 0 || asInt > 120) return null;
    return asInt * 1000;
  }
  const when = Date.parse(raw);
  if (Number.isNaN(when)) return null;
  const delta = when - nowMs;
  if (delta < 0 || delta > 120_000) return null;
  return delta;
}

export function classifyRetry(input) {
  const {
    method,
    status,
    retryAfterHeader,
    attempt,
    maxAttempts,
    idempotencyKey,
    source,
  } = input;

  if (source && source !== "pinned-map") {
    return { retry: false, reason: "unpinned-source", delayMs: 0 };
  }
  if (attempt >= maxAttempts) {
    return { retry: false, reason: "budget-exhausted", delayMs: 0 };
  }
  if (!RETRYABLE_STATUS.has(status)) {
    return { retry: false, reason: "status-not-retryable", delayMs: 0 };
  }
  if (MUTATING.has(method) && !idempotencyKey) {
    return { retry: false, reason: "mutating-without-idempotency", delayMs: 0 };
  }

  const hinted = parseRetryAfter(retryAfterHeader);
  const delayMs = hinted ?? Math.min(1000 * 2 ** attempt, 30_000);
  return { retry: true, reason: "pinned-status", delayMs };
}

The source guard against unpinned maps is the lesson. A model adapter cannot sneak in without failing closed. Backoff remains a numeric delay for every retry. Prose comments and hidden tool calls stay out.

Tests that fail if a model joins the path

Run the file with Node's built-in test runner.

// retryability.test.mjs
import { test } from "node:test";
import assert from "node:assert/strict";
import { classifyRetry, parseRetryAfter } from "./retryability.mjs";

const base = {
  method: "POST",
  status: 503,
  retryAfterHeader: null,
  attempt: 0,
  maxAttempts: 3,
  idempotencyKey: "refund-9f3c",
  source: "pinned-map",
};

test("503 with idempotency key may retry", () => {
  const out = classifyRetry(base);
  assert.equal(out.retry, true);
  assert.equal(out.reason, "pinned-status");
});

test("POST without an idempotency key must not retry", () => {
  const out = classifyRetry({ ...base, idempotencyKey: "" });
  assert.equal(out.retry, false);
  assert.equal(out.reason, "mutating-without-idempotency");
});

test("404 is never retryable", () => {
  const out = classifyRetry({ ...base, status: 404 });
  assert.equal(out.retry, false);
});

test("409 conflict is never retryable", () => {
  const out = classifyRetry({ ...base, status: 409 });
  assert.equal(out.retry, false);
});

test("unpinned source fails closed", () => {
  const out = classifyRetry({ ...base, source: "free-model" });
  assert.equal(out.retry, false);
  assert.equal(out.reason, "unpinned-source");
});

test("Retry-After seconds are capped", () => {
  assert.equal(parseRetryAfter("3"), 3000);
  assert.equal(parseRetryAfter("9999"), null);
});

test("response body never reaches classifyRetry", () => {
  const keys = Object.keys(base);
  assert.equal(keys.includes("body"), false);
  assert.equal(keys.includes("prompt"), false);
});

node --test retryability.test.mjs

A green run does not prove production safety. It only proves the classifier rejects unpinned sources. That is the contract this article cares about.

Decision table

Signal	Live retry vote	Offline model use
503 + idempotency key	Pinned map may retry	Draft extra fixtures
409 conflict	Never retry	Summarize runbooks
429 with Retry-After	Honor capped delay	Not consulted
500 on POST, no key	Never retry	Generate negative tests
Timeout, unknown status	Never retry	Propose new map rows
Free server preemption	Fail closed	Rebuild the harness

The model column stays as optional offline work. The left column remains the only production vote. Do not merge the columns to save a round trip.

How a model sneaks onto the path

Most leaks arrive as a small helper during an incident. Someone pastes the 503 body into a prompt for comfort. The helper then gets imported into the worker loop.

The next module is an anti-pattern for review drills only. Do not ship it. Do not wrap it.

// do-not-ship.mjs — anti-pattern, review drills only
export async function classifyWithModel(res, promptClient) {
  const text = await res.text();
  const vote = await promptClient.complete(
    `Retry this ${res.status}? ${text.slice(0, 500)}`
  );
  return String(vote).toLowerCase().includes("yes");
}

That function reads prose and returns an unlogged boolean. It ignores idempotency, budgets, and Retry-After caps. Reviewers should reject any import of this shape.

A second leak is schema drift inside the tool layer. The agent regenerates an OpenAPI snippet after a 503. Required idempotency headers disappear from the new snippet. The next call looks unique to the ledger. Pin the tool schema the same way the status map is pinned.

Load tests will lie if retries are model-voted

Soak tests assume an authored traffic mix. A model-voted retry loop rewrites that mix under failure. One planted 503 can multiply a 100 POST script into four hundred attempts.

Percentiles then describe the retry storm, not the service. Error budgets then describe the storm as well. Keep the classifier pinned so the load profile stays authored.

Record attempt counts as a metric beside latency. Record reason from classifyRetry on every reject. A rising unpinned-source count is a ship-stop, not a dashboard curiosity.

# proposal: fail a soak when retry reasons drift
node soak.mjs --max-retry-ratio 0.05 --forbid-reason unpinned-source

The command is a local proposal, not a vendor benchmark. Teams still need their own traffic files and SLOs. The point is to measure retries as a first-class output.

Pull request checklist

Reject the change if any box fails.

• Function arguments still exclude body and prompt fields.
• New status codes include a vendor doc link in comments.
• Mutating methods still require an idempotency key.
• Tests cover 404, 409, 429, and unpinned source.
• No new network client is imported by the classifier.
• Worker hosting is not a scratch free server.

A checklist is slower than a prompt during a 503 burst. That slowness is the feature. Retry policy should be boring under pressure.

Where a free model still helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those lanes fit fixture drafting and isolated harness runs. They do not fit live retry classification at all. For that offline fixture loop, those free lanes are enough to try.

A useful split of duties looks like this.

1. Freeze the status map in source control today.
2. Ask a free model to propose missing negative tests.
3. Run those tests on a free server scratch box.
4. Copy accepted tests into the pinned suite afterward.
5. Keep the production worker on a stable host.

The free model never sees live 503 bodies. The free server never stores the attempt budget. Human reviewers still read every new map row.

Where the free server must not sit

A retry coordinator holds attempt counts and lease timers. A free server can vanish between two 503s. Lost timers become either silence or a stampede.

Keep the following components off the free server.

• The worker that increments attempt counters on each 503
• The durable store for idempotency keys still in flight
• The clock used to honor Retry-After without drift
• The publisher that re-queues delayed jobs after backoff

Scratch generation of tests can live there. The retry control plane cannot live there. Cold starts are acceptable for fixture jobs. Cold starts are not acceptable for Retry-After math.

Exit criteria

Move the live path off free lanes when any trigger fires.

• Duplicate side effects appear in staging replay logs.
• Source logs show any value except pinned-map.
• p95 classify time exceeds the worker ack deadline.
• On-call cannot recite the status map from memory.
• A model-written test was merged without a human diff.
• Backoff state was lost after a scratch host recycle.

Exit means code and hosting, not a prompt tweak. Restore the pinned map in the worker first. Restore a durable worker for the retry loop. Then delete the adapter that called the model.

Do not keep the adapter behind a feature flag for later. Flags drift. The next incident will flip the flag under fatigue. Deletion is the only honest exit.

Who should not use this approach

Teams should skip this split in some cases.

• Purely local CLIs with no networked side effects at all
• Read-only crawlers that already treat every error as fatal
• Teams without a test runner or a review path
• Protocols that already pin retryability in-band today

Those groups need less machinery, not a model vote. Do not add a classifier only for fashion. A fatal-on-any-error client is already a pinned policy.

Limitations

The status map above is incomplete on purpose. gRPC codes, WebSocket drops, and exactly-once queues need their own tables. Retry-After parsing here rejects long delays rather than scheduling them.

These tests do not measure any vendor uptime. They do not claim a model quality score. They only lock the hot path to deterministic inputs.

Status maps drift whenever vendors change error contracts. Revisit the table when an upstream publishes a new error catalog. Do not let a model apply that catalog live.

Idempotency keys have their own lifetime rules. This article does not define key expiry, hashing, or storage. A pinned retry vote still fails if keys collide or vanish.

Closing

Retry storms often start as helpful agent features. They end as duplicate charges and angry ledgers. Keep classification boring, pinned, and out of the prompt.

Free models can still write the tests that keep it boring. The production vote stays in the module above.