Retries Are Not Free: Measuring Duplicate Side Effects in an Agent Loop

Your agent calls a tool. The call times out. Your loop retries it.

Now you have two writes and one intent. Which one was the bug? Neither. The bug is the missing dedup decision.

I keep seeing the same four claims about retries in agent code. Each one sounds reasonable. Each one is wrong in a way you can measure in about ten minutes.

Let me show you the harness first. Then the myths, because the harness is the evidence.

The artifact: a duplicate ledger you can run today

This is a self-contained simulation, not a real payment API. It models one specific failure: the write lands, then the response never arrives.

# retry_ledger.py - stdlib only. Simulates a 'commit, then crash' server.
import http.server, threading, time, json, uuid, urllib.request, urllib.error

LOCK = threading.Lock()
LEDGER = []        # every write that actually landed
ATTEMPTS = {}      # idempotency key -> times the server saw it

class Handler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        key = self.headers.get('Idempotency-Key', '')
        n = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(n).decode()
        with LOCK:
            seen = ATTEMPTS.get(key, 0) + 1
            ATTEMPTS[key] = seen
            if key in [row['key'] for row in LEDGER]:
                status, note = 200, b'replayed'            # safe replay
            elif seen == 1:
                LEDGER.append({'key': key, 'body': body})
                status, note = 500, b'commit-then-crash'   # write landed anyway
            else:
                LEDGER.append({'key': key, 'body': body})
                status, note = 201, b'created'
        self.send_response(status)
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()
        self.wfile.write(note)

    def log_message(self, *args):
        pass

def call(url, payload, key_factory, max_attempts=2):
    for attempt in range(1, max_attempts + 1):
        req = urllib.request.Request(
            url,
            data=json.dumps(payload).encode(),
            headers={'Content-Type': 'application/json',
                     'Idempotency-Key': key_factory(attempt)},
            method='POST')
        try:
            with urllib.request.urlopen(req, timeout=2) as r:
                return r.status
        except urllib.error.HTTPError as e:
            if e.code < 500 or attempt == max_attempts:
                raise
            time.sleep(0.1 * attempt)

def run(label, key_factory):
    global LEDGER
    with LOCK:
        LEDGER.clear(); ATTEMPTS.clear()
    try:
        status = call('http://127.0.0.1:%d/charge' % PORT, {'amount': 100}, key_factory)
        note = 'status %s' % status
    except Exception as e:
        note = 'raised %s' % e
    with LOCK:
        writes = len(LEDGER)
        tries = sum(ATTEMPTS.values())
    print('%-12s attempts=%d writes=%d (%s)' % (label, tries, writes, note))

if __name__ == '__main__':
    server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Handler)
    PORT = server.server_address[1]
    threading.Thread(target=server.serve_forever, daemon=True).start()
    run('stable-key', lambda a: 'order-42')
    run('fresh-key', lambda a: str(uuid.uuid4()))

Run it:

python3 retry_ledger.py

You should see this:

stable-key   attempts=2 writes=1 (status 200)
fresh-key    attempts=2 writes=2 (raised HTTP Error 500: Internal Server Error)

Same payload. Same retry loop. One version charges your user twice.

Now the four myths.

Myth 1: "A retry is just a second request"

The server counts attempts. The ledger counts writes. Those are different numbers.

In the run above, the fresh-key case shows attempts=2 writes=2. The client never observed a success, and the server still wrote twice. Your logs show zero successes. Your database shows two rows.

Correction: delivery is at-least-once, so deduplication is yours.

Myth 2: "PUT and DELETE are idempotent, so I am safe"

RFC 9110 defines idempotency at the HTTP method level: repeating a request has the same effect on server state. That is useful. It is also narrower than what you care about.

Your side effect may sit outside server state entirely:

• An email sent through a third-party API.
• A webhook fired at a customer endpoint.
• A file appended to object storage under a new key each time.

The HTTP method cannot protect a side effect the HTTP server does not own. Dedup has to live where the side effect lives.

Myth 3: "A timeout means the request never landed"

A timeout means you stopped waiting. It says nothing about what the server did.

The commit-then-crash branch above is exactly this case. The write committed, the response died, the client saw 500. If your retry policy reads 500 as "nothing happened", you duplicate on every cold start.

Read the two failure classes separately:

• 4xx: usually a real rejection. Retrying will not help.
• Timeout and 5xx: outcome unknown. Retry is safe only with a dedup key.

Myth 4: "The model should decide what to retry"

Retry policy is deterministic infrastructure, not a prompt. A model that sees a 500 may retry, rewrite the payload, or invent a new one. Rewriting the payload is the worst outcome, because it defeats your idempotency key.

Keep the model on the outside. It proposes a tool call. Your client library decides whether to resend it, with the same key, and how many times.

A decision table you can paste into a design review

Operation	Safe to auto-retry?	Key required	Notes
Read a record by id	Yes	Not needed	GET is idempotent per RFC 9110
Create with a client key	Yes	Yes	Server must store the key with the row
Create with a server-generated id	No	No	Each retry makes a new row
Send an email or SMS	No	Yes, provider-side	Providers vary; read their docs
Charge a card	No	Yes, provider-side	Never retry a bare POST here
Append to a log stream	Yes	Not needed	Duplicate lines are cheap
Fire a webhook to a customer	No	Yes	Customers dedupe badly, if at all

Where a free endpoint fits, and where it does not

The harness above runs in one process. You can run it anywhere Python 3 runs. If you want to test this without a card on file, MonkeyCode's free model access and free server option are two operator-supplied availability claims. Treat them the way you treat any other availability claim: verify them against your own workload before relying on them. A simulation that writes nothing real is a reasonable first workload for a free server, because a wrong result costs you nothing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not use this approach if any of these describe you:

• Your tool calls are pure reads. You have no duplicates to prevent.
• You already run a durable outbox with exactly-once delivery. The harness adds nothing.
• You cannot change the client. Fix the retry policy before adding a key.
• Your provider offers a native idempotency key and documents its replay window. Use theirs.

One warning: idempotency keys expire on some providers. A key stored for a few hours will not save a retry tomorrow. Check the window before you depend on it.

What to check before you ship a retry

1. Count writes, not successes. Add a ledger table or a counter.
2. Give every mutating call a key that survives a process restart.
3. Send the same key on every attempt of the same intent.
4. Retry 5xx and timeouts only. Fail fast on 4xx.
5. Cap attempts and add backoff. Unbounded loops multiply writes.
6. Log both the key and the attempt number.
7. Test the commit-then-crash path. It is the only one that duplicates silently.

The last item matters most. A clean 500 with no write is easy to handle. A 500 after a write is the one that reaches your users twice.

Start with the ledger, not the blog post. If you run this on a free tier, tell me your two numbers.