A Retry Is Not a Repeat: Five Idempotency Myths Behind a Model-Backed Endpoint

Your endpoint returned one 200. How many times did the model actually run?

That question is the whole article. Retries are invisible in your logs. They show up in your data.

I see the same five claims in code review. All five sound reasonable. Four of them are wrong in ways that corrupt state.

Here is a lab you can run in ten minutes. It counts effects, not responses.

Myth 1: "A timeout means the request never arrived"

No. A timeout is a client-side observation. It says nothing about the server.

The request may be queued, half-processed, or finished with the response lost on the way back. RFC 9110 separates safe, idempotent, and non-idempotent methods for exactly this reason.

Your client gave up at 5.0s. The model finished at 5.4s. You have just scheduled a duplicate.

Corrected model: a timeout is an unknown, not a failure. Treat it as "maybe happened".

Myth 2: "A 200 means the work happened once"

A 200 means this attempt succeeded. It says nothing about earlier attempts.

Three timed-out attempts plus one success is four model calls and one 200. Your dashboard shows a healthy request. Your database shows three writes.

Ask yourself: what in my system can tell the difference between one effect and four?

If the answer is "nothing", you do not have retries. You have duplication with extra steps.

Myth 3: "Idempotency keys are a nice-to-have"

They are the smallest mechanism that turns an unknown into a decision. Generate one per logical operation, not per HTTP attempt.

The common bug is key placement. People generate the key inside the retry loop, which guarantees a fresh key each time. Then they wonder why dedupe never fires.

Rules I would keep:

1. The key is created before the first attempt.
2. Every retry reuses the exact same key.
3. The server stores the key with the finished response and a TTL.
4. A key seen but unfinished returns a conflict, not a second execution.

Myth 4: "Retry on 5xx and ignore everything else"

429 is the one status that explicitly asks you to retry later. Ignoring it is how you turn a rate limit into an outage.

Read the Retry-After header when it exists. Fall back to exponential backoff with jitter when it does not. AWS's builders' library covers why jitter matters more than the exact base.

And stop retrying the things that will never work. A 400 or 422 means your prompt or schema is wrong. Retrying it five times just logs the same failure five times.

Myth 5: "Model calls are cheap, so retries are cheap"

Free model access changes the cost per attempt. It does not change the number of side effects.

Token spend drops toward zero. Wall-clock does not. Duplicate writes do not. Streamed partials do not.

So budget your retries in time and effects, not dollars. That reframing is the useful part. A free attempt is still an attempt.

The artifact: a stub endpoint that lies to you

Two files. Python 3.9+. Standard library only. No API keys, no network calls.

File 1: flaky_model.py

A stub that always outlives the client timeout, counts real executions, and dedupes by key.

# flaky_model.py - slow stub with effect counting and key dedupe.
# Run: python flaky_model.py
import json, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

LOCK = threading.Lock()
SEEN = {}          # idempotency key -> response body, or None while in flight
EFFECTS = 0        # how many times the "model" really ran
REQUESTS = 0
SLOW = 2.0         # always slower than the client timeout below


class H(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def _send(self, code, payload, extra=None):
        body = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        for k, v in (extra or {}).items():
            self.send_header(k, v)
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self):
        if self.path == "/stats":
            with LOCK:
                return self._send(200, {"effects": EFFECTS, "requests": REQUESTS})
        self._send(404, {"error": "not found"})

    def do_POST(self):
        global EFFECTS, REQUESTS
        if self.path != "/generate":
            return self._send(404, {"error": "not found"})

        key = self.headers.get("Idempotency-Key")
        with LOCK:
            REQUESTS += 1
            if key and key in SEEN:
                cached = SEEN[key]
                if cached is None:
                    return self._send(409, {"error": "in flight"})
                return self._send(200, cached, {"X-Replay": "true"})
            if key:
                SEEN[key] = None
            EFFECTS += 1
            out = {"text": "answer #%d" % EFFECTS, "effect": EFFECTS}
        time.sleep(SLOW)
        with LOCK:
            if key:
                SEEN[key] = out
        self._send(200, out, {"X-Replay": "false"})


if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8787), H).serve_forever()

File 2: probe.py

A client with a short timeout and bounded backoff. It runs the same logical request twice: once without a key, once with one.

# probe.py - same prompt, two clients, one effect counter.
import json, time, urllib.error, urllib.request, uuid

BASE = "http://127.0.0.1:8787"
TIMEOUT = 0.8      # shorter than the stub's 2.0s, so attempt 1 always fails


def stats():
    with urllib.request.urlopen(BASE + "/stats", timeout=2) as r:
        return json.loads(r.read())


def post(prompt, key=None, attempts=3):
    for i in range(1, attempts + 1):
        req = urllib.request.Request(
            BASE + "/generate",
            data=json.dumps({"prompt": prompt}).encode(),
            headers={"Content-Type": "application/json"},
            method="POST",
        )
        if key:
            req.add_header("Idempotency-Key", key)
        try:
            with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
                return r.status, r.headers, json.loads(r.read())
        except urllib.error.HTTPError as e:
            if e.code == 409 or e.code == 429 or 500 <= e.code < 600:
                time.sleep(1.5 * 2 ** (i - 1))
                continue
            return e.code, e.headers, json.loads(e.read())
        except (TimeoutError, urllib.error.URLError):
            time.sleep(1.5 * 2 ** (i - 1))
            continue
    return None, {}, {"error": "gave up"}


if __name__ == "__main__":
    for label, key in (("A: no key", None), ("B: with key", str(uuid.uuid4()))):
        before = stats()
        code, headers, body = post("write a haiku about retries", key=key)
        after = stats()
        print(label)
        print("  status:", code, "body:", body)
        print("  replay header:", headers.get("X-Replay"))
        print("  effects delta:", after["effects"] - before["effects"])

Run it

python flaky_model.py &
python probe.py
curl -s localhost:8787/stats

Read the output

Case A should show an effects delta of 2. One logical request. Two real executions. Nobody noticed.

Case B should show a delta of 1, plus X-Replay: true on the retry. The second attempt got the stored answer instead of running the work again.

Two lines of output. That is the whole difference between a retry and a repeat.

A retry decision table

Signal	Retry?	Same key?	Note
Read timeout	Yes, bounded	Yes	Assume the server may have finished
429	Yes, honor Retry-After	Yes	Backoff with jitter, cap attempts
5xx	Yes, bounded	Yes	1s, 2s, 4s plus jitter
409 in flight	Yes, after a delay	Yes	Your own earlier attempt is still running
400 / 422	No	n/a	Fix the request, not the transport
Stream cut mid-way	Only if deduped	Yes	Otherwise you have partial effects

Where this lab actually runs

I run the stub locally. When I want the same probe against a real endpoint, I put both files on a throwaway box and point BASE there.

MonkeyCode's free model access and free server option are the two availability claims I am relying on here, as provided by the operator. I am not making claims about model names, quotas, hardware, or how long any of it lasts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you swap the stub for a free-model endpoint, keep the key logic identical. The transport changes. The failure modes do not.

Limitations, and who should skip this

• The dedupe store is in memory. A restart forgets every key.
• One process only. Two instances need a shared store and a real TTL.
• Keys need scope. Same key, different prompt, should be a conflict.
• Streaming makes "finished" ambiguous. Define it before you dedupe.
• The stub has no auth. Do not expose it on a public URL.

Skip this approach if you move money, send email, or call a non-idempotent third party. There, a shared dedupe store is not optional, and "probably once" is not an answer.

The one question to keep

Before you add a retry, ask what happens if it succeeds twice.

If you cannot answer in one sentence, you are not ready to retry yet.

Run the probe. Read the delta. Then decide.

Comments

[Steven Mock]

The key-placement bug is the one I'd highlight for anyone skimming this. Generating the key inside the retry loop looks harmless and quietly defeats the whole mechanism — I've seen that exact mistake in production code. Counting effects rather than responses is the right frame, and the closing question is the one to ask before adding any retry: what happens if this succeeds twice? If you can't answer that in one sentence, you're not ready.