48-Hour Field Notes: The Retry That Billed the Same Diff Twice

At 07:40 the CI job posted the third identical review comment on the same pull request. Nothing in the code had changed. The upstream API had returned 504 twice, the webhook retried, and each retry walked the same diff through the same prompt. The ledger I keep for AI calls showed three entries, three prompt counts, one diff.

That was the whole bug. Not model quality, not latency. Our AI calls were not idempotent, and a timeout was quietly turning into a second and third charge.

These are my notes from 48 hours of fixing that on a single small box, using MonkeyCode's free model access and the free server option to keep the experiment cheap enough to be honest. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free tiers are finite by definition, and check the current terms yourself — allowances and limits change, and I am not quoting a number here because I do not want you budgeting against a figure I cannot guarantee tomorrow.

Hours 0–8: name the failure before you fix it

A retry is not a retry. Inside one process, a for attempt in range(3) loop costs you three model calls and you can see it in the logs. Across a job re-dispatch — webhook redelivery, queue visibility timeout, k8s restarting a pod that looked dead — you get a fresh process with no memory that the work already happened. That second kind is the expensive one, because nothing in the process knows it is repeating itself.

The fix is the same thing payment processors do with terminal timeouts: derive a stable key from the request, reserve it before doing the work, and refuse to do the work twice for the same key.

# ledger.py — the guard, about 40 lines
import hashlib, json, sqlite3, time

DDL = """
CREATE TABLE IF NOT EXISTS calls (
  key TEXT PRIMARY KEY,
  status TEXT NOT NULL,          -- pending | done
  result TEXT,
  prompt_tokens INTEGER,
  output_tokens INTEGER,
  created_at REAL NOT NULL
);
"""

def key_for(model, prompt, schema_version="v3"):
    raw = json.dumps([schema_version, model, prompt], sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()

def reserve(conn, key):
    try:
        conn.execute("INSERT INTO calls(key,status,created_at) VALUES(?,?,?)",
                     (key, "pending", time.time()))
        conn.commit()
        return True
    except sqlite3.IntegrityError:
        return False

def commit(conn, key, result, usage):
    conn.execute(
        "UPDATE calls SET status='done', result=?, prompt_tokens=?, output_tokens=? WHERE key=?",
        (json.dumps(result), usage["prompt"], usage["output"], key))
    conn.commit()

def get_done(conn, key):
    row = conn.execute("SELECT result FROM calls WHERE key=? AND status='done'", (key,)).fetchone()
    return json.loads(row[0]) if row else None

The key is a hash of (schema_version, model, prompt). Hashing the model name matters: if you route the same prompt to a different model, that is genuinely different work and deserves its own row. Hashing the schema version matters more than people expect, because prompt templates drift and a cached answer from an older template is worse than no answer.

Hours 8–24: the test that proves one charge

The rule I wanted was "N upstream attempts may happen, one billed result may be recorded." So the test asserts both numbers, not just one.

class FlakyClient:                      # test-only fake, no network
    def __init__(self): self.calls = 0
    def complete(self, model, prompt):
        self.calls += 1
        if self.calls < 3:
            raise TimeoutError("upstream 504")
        return {"text": "LGTM", "usage": {"prompt": 812, "output": 41}}

def test_retries_record_one_result(tmp_path):
    conn = sqlite3.connect(tmp_path / "budget.db", timeout=5)
    conn.executescript(DDL)
    client = FlakyClient()

    out, src = review(conn, client, "free-tier-model", "diff...")
    assert (out, src) == ("LGTM", "model")
    assert client.calls == 3            # upstream saw three attempts
    assert spent(conn) == 853           # the ledger recorded one

    out2, src2 = review(conn, client, "free-tier-model", "diff...")
    assert src2 == "cache" and client.calls == 3

Run it with python -m pytest -q -k idempotency, then read the ledger the way you would read a bank statement:

sqlite3 budget.db "SELECT status, COUNT(*), SUM(prompt_tokens+output_tokens) FROM calls GROUP BY status;"

The 812/41 numbers are fixture values, not a benchmark. What I actually learned from the first run was less flattering. The cache row appeared, client.calls stayed at 3 — and the pending row from the two failed attempts sat there forever. My guard had no way back from a crash.

Hours 24–48: the zombie row and the honest limits

A pending row written before a call is a landmine if the process dies mid-call. It blocks every future retry of that diff, which turns a cost guard into an availability bug. The correction was a lease, not a lock: reclaim pending rows older than a window, and let the next attempt take over.

def reclaim(conn, key, stale_after=900):
    conn.execute("DELETE FROM calls WHERE key=? AND status='pending' AND created_at < ?",
                 (key, time.time() - stale_after))
    conn.commit()

Fifteen minutes was arbitrary and I would tune it against your own p95 latency rather than copy it. The same is true of the ledger's growth: add a TTL sweep or the SQLite file becomes a slow leak.

What this approach does not fix, stated plainly. It does not stop in-process retries from hitting the model — those still cost, and only the recorded result is deduplicated. It is best-effort across restarts, because on a free-tier single instance the disk may be ephemeral; verify persistence before you trust it as a source of truth. It assumes the work is deterministic enough that a stored answer stays valid, which fails for sampling with nonzero temperature or for prompts that embed timestamps. And it does nothing for tool calls inside the loop that are themselves non-idempotent, like opening a ticket or pushing a commit.

Skip this if your reviews must be re-read fresh on every run, if you deliberately sample multiple candidates, or if each call carries unique runtime context that makes hashing pointless. Two processes writing concurrently also need PRAGMA journal_mode=WAL and BEGIN IMMEDIATE, or you will trade duplicate billing for database is locked.

What I would repeat: build the ledger before the prompt, assert the billed count in a test, and treat every timeout as a potential second charge. The free tier and the free server were enough to run all of this in an afternoon; the discipline was the part that mattered, and it would have mattered on a paid account too. If you try it, keep the ledger next to the model config — that is the file that tells you the truth.

Comments

[Gautam K]

The cross-process retry is the one that bites. A key of commit SHA plus prompt version plus model, checked in a shared store before the call, makes the second and third run a cache hit. Bonus: the review comment can be keyed the same way, so a redelivered webhook edits the existing comment instead of posting a third.