Postmortem: A Retried Model Call Applied One Patch Three Times

A retry loop around a model call turned one patch into three commits. The gate reported success each time. The duplicate reached main because retries were keyed on the request, not on the patch content.

This post walks the timeline, the four contributing factors, and the durable fix. The artifact is small: a content-addressed idempotency key, a usage ledger built from provider-reported numbers, and a replay test that fails before the fix and passes after.

Scope of this postmortem

The timeline below was reconstructed in a local replay harness, not captured from production traffic. Task ids, timings, and attempt counts are illustrative. The failure mode is the subject; the numbers are scaffolding.

Readers who already run a patch gate can map the harness onto their own retry wrapper in an afternoon.

The setup

The gate had three stages: generate a patch, run tests, apply the patch. Generation used free model access. Application ran on a free server option to keep the rehearsal environment cheap.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. The operator states the free allowance is 10 million tokens. Nothing in the rest of this post depends on that number; the failure would occur at any budget size.

Timeline (UTC, replay harness)

Time	Event
09:14:02	Worker picks task t-4417, generation succeeds, patch stored
09:14:03	Tests pass on the stored patch
09:14:03	Apply call returns HTTP 502 from an upstream proxy
09:14:03	Retry wrapper classifies 502 as transient, sleeps 1s
09:14:04	Attempt 2 regenerates the patch from the same prompt
09:14:04	Attempt 2 applies successfully, worker crashes before ack
09:14:05	Queue redelivers the task after the visibility timeout
09:14:06	Attempt 3 regenerates and applies again
09:31:40	Reviewer sees three near-identical commits on one branch
10:02:15	Branch frozen, dedupe table added

Two details matter more than the rest. Every retry regenerated the patch instead of reusing the stored one. Every apply wrote a new row with no uniqueness constraint on content.

Why the tests stayed green

The test stage never saw a duplicate. It ran once per attempt against one candidate patch, in isolation.

The gate's success signal was apply_exit_code == 0. Three successful applies produce three zeroes. Nothing in the pipeline compared the applied set against the intended set.

Green tests answered a narrow question: does this patch pass. They never answered: was this patch applied exactly once.

Contributing factors

1. Retries keyed on the request. The wrapper used a fresh attempt id per call. Two identical patches looked like two different jobs.
2. Regeneration inside the retry scope. Logging, model sampling, and patch assembly sat inside the loop. Only the HTTP send needed to be there.
3. No uniqueness constraint on apply. The apply table accepted duplicates by design, because nobody asked what a duplicate would look like.
4. Ack after side effect. The worker applied the patch, then acked. A crash in that window guarantees redelivery.

Factor four is the trigger. Factors one and three turn a redelivery into a duplicate commit.

The durable fix, part 1: content-addressed keys

Derive the key from what is being applied, not from when it was requested. Then the same patch always collides with itself.

import hashlib

def idempotency_key(task_id: str, patch_bytes: bytes, scope: str) -> str:
    normalized = b"\n".join(line.rstrip() for line in patch_bytes.splitlines())
    digest = hashlib.sha256(normalized).hexdigest()
    return hashlib.sha256(f"{task_id}:{digest}:{scope}".encode()).hexdigest()

Normalizing whitespace first keeps cosmetic diffs from minting new keys. Keep the task id in the key so two tasks producing byte-identical patches still apply separately.

The apply path becomes an insert guarded by the key:

CREATE TABLE applied_patch (
  idem_key   TEXT PRIMARY KEY,
  task_id    TEXT NOT NULL,
  applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

INSERT INTO applied_patch (idem_key, task_id)
VALUES ($1, $2)
ON CONFLICT (idem_key) DO NOTHING
RETURNING idem_key;

Zero rows returned means the patch was already applied. Treat that as success, not as an error. The worker should then ack and move on.

The durable fix, part 2: a ledger from provider numbers

Estimated token counts drift. Provider-reported usage does not. Record the reported numbers, and mark the task unknown when they are absent.

from dataclasses import dataclass

@dataclass(frozen=True)
class Usage:
    task_id: str
    prompt_tokens: int
    completion_tokens: int
    source: str  # "provider_reported" or "unknown"

def record(ledger, task_id, raw_usage):
    if raw_usage is None:
        ledger.append(Usage(task_id, 0, 0, "unknown"))
        return
    ledger.append(Usage(
        task_id,
        raw_usage.prompt_tokens,
        raw_usage.completion_tokens,
        "provider_reported",
    ))

Rule: a task with source == "unknown" cannot proceed to apply. Quarantine it and reconcile first. Two reasons matter here. An unmeasured retry loop can burn a free allowance silently. And unknown usage usually means the stream died mid-response, which is a separate class of bug.

The retry decision table

Replace if status >= 400: retry with an explicit table. Four rows cover most patch gates.

Condition	Action	Key
429 or 5xx before first token	Retry with backoff, capped at 3 attempts	Reuse stored patch
Stream aborted after first token	Discard partial patch, reconcile usage, requeue fresh	New key
Usage field missing	Fail closed, quarantine task	No key issued
Apply returned unknown state	Read applied_patch first, then decide	Existing key

The second row is the one teams skip. A truncated stream is not a transient transport error. Retrying it re-samples the model and produces a different patch from the one under test.

Reproducible test plan

Run this against a fake model server. No product access is needed, and the test is deterministic.

1. Start fake_model.py with a scripted response sequence: [502, 502, 200].
2. Point the gate at it and set the worker to crash after apply on attempt 2.
3. Redeliver the task once, simulating a visibility timeout.
4. Assert three things: one row in applied_patch, one nonzero usage record, zero unknown records.

def test_retry_applies_exactly_once(gate, fake_model):
    fake_model.script([502, 502, 200])
    gate.crash_after_apply = 2
    gate.run(task_id="t-4417", prompt="fix the off-by-one")
    gate.redeliver("t-4417")

    assert gate.applied_count() == 1
    assert gate.ledger.unknown_usage() == 0
    assert gate.ledger.total_tokens() > 0

Run the same test with the key function replaced by a counter. It should fail with applied_count() == 3. A regression test that cannot fail is decoration.

Verification checklist before merging the fix

• The key is derived from normalized patch bytes and task id.
• The apply insert uses ON CONFLICT DO NOTHING and treats zero rows as success.
• The ack happens after the apply transaction commits.
• Regeneration is outside the retry scope.
• Unknown usage blocks apply rather than defaulting to zero.
• The replay test fails on the pre-fix branch.

Limitations

This fix prevents duplicates. It does not improve patch quality, and it adds one database write to the apply path.

Content-addressed keys break when formatting churn is high. Normalize aggressively, or key on a semantic diff instead of raw bytes.

The dedupe table also grows without a retention policy. Prune by task age once the queue's redelivery window has passed.

Who should not use this

Teams with a human review step before every merge already have a dedupe layer. A named reviewer catches three identical commits quickly.

Single-worker pipelines with no retries gain little. The table costs more than it saves.

Regulated environments that require audited model vendors should settle that question first. Idempotency does not make an unapproved vendor approved.

Where the free tier fit

The gate in this post ran on MonkeyCode's free model access and the free server option. That combination made a replay harness cheap enough to keep running in CI.

If the duplicate-apply failure mode feels familiar, the rehearsal is the useful part. A fake server and a four-row decision table will tell you more than another green run.

Takeaways

Key retries on content, not on requests. Keep regeneration outside the loop. Ack after the commit, never before. And measure with reported numbers, because a silent retry loop spends budget whether or not anyone is watching.