Retry Is Not a Reset: Four Myths About Duplicate Side Effects in Agent Loops

My agent created the same ticket four times last week.

One call timed out. The loop retried. The upstream was flaky for eleven seconds. I ended up with four tickets and three apologies.

The bug was not in the model. It was in my mental model of failure. So here are four myths I used to believe, the evidence that killed them, and the corrected model I use now.

Myth 1: A failed call means nothing happened

No. Transport failure and effect failure are different events.

A request can do three things. It can never arrive. It can arrive and commit the effect. Or it can commit the effect and lose the response on the way back. That third case is the worst one, because it tells you nothing.

Ask yourself a blunt question. If the network drops right after the database commit, what does your client see? A timeout. Exactly what it sees when the request never left your laptop.

Can you tell those two apart from the client side? You cannot. Ever.

Myth 2: Retries are free because the endpoint is free

The endpoint might be free. The retry is not free.

Every duplicate retry costs something real:

• A duplicated side effect: a ticket, a commit, an email, a row.
• A step from your loop budget, not a cent from your wallet.
• A corrupted audit log, where step counts stop matching effects.

People hear free tier and think no cost. Wrong axis. The currency of a free endpoint is steps, wall-clock time, and trust in your own logs. I wrote about step budgets before; this is the sibling problem.

Myth 3: Idempotency keys are a payments thing

Payments popularized the pattern. They do not own it.

Every non-read tool call in your agent loop needs a stable key:

• create_ticket, send_email, push_commit, insert_row
• Anything that appends to a queue
• Anything that writes a file you would notice twice

Reads are naturally idempotent. Writes are not. Sort your tools into those two buckets before you write a single line of retry policy. That sort takes ten minutes and saves weekends.

Myth 4: A better prompt will prevent duplicates

Prompts do not change transport semantics.

Telling the model do not retry if unsure just adds a new failure mode. The agent now abandons calls that would have succeeded. And it still cannot see through a dropped acknowledgement.

Retry logic belongs in code. Not in the system prompt.

The corrected model: track three states, not two

Stop modelling a call as success or failure. Model it as three states:

1. Attempted — the client sent bytes.
2. Committed — the server changed state.
3. Acknowledged — the client learned about state 2.

The client only ever observes states 1 and 3. So the server must enforce exactly-once, and the client must supply a stable key. Rules I follow now:

• Generate the key before the first attempt, not inside the retry loop.
• Persist the key next to the tool call, so a restart reuses it.
• Retry transport errors, 5xx, and 429 with backoff plus jitter.
• Never retry a 4xx. The request was understood and rejected.
• Dedupe on the server, keyed on that key, backed by a real store.

A runnable harness: 60 lines, standard library only

The code below starts a deliberately hostile endpoint. It commits the effect, then returns 503 before acknowledging. Then it runs the same client twice: once against a naive server, once against a deduping one.

# retry_ledger.py - Python 3 standard library only
import json
import threading
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer

STATE = {'effects': 0, 'seen': {}, 'fail_next': True, 'dedupe': False}


class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def _reply(self, code, payload):
        body = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        payload = json.loads(self.rfile.read(length) or b'{}')
        key = payload.get('idempotency_key', 'missing')

        if STATE['dedupe'] and key in STATE['seen']:
            return self._reply(200, STATE['seen'][key])

        STATE['effects'] += 1              # the side effect commits here
        result = {'effect_no': STATE['effects'], 'key': key}
        STATE['seen'][key] = result

        if STATE['fail_next']:             # and the ack dies here
            STATE['fail_next'] = False
            return self._reply(503, {'error': 'upstream'})

        return self._reply(200, result)


def call(payload, retries=3):
    req = urllib.request.Request(
        'http://127.0.0.1:8765/write',
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json'},
        method='POST',
    )
    for _ in range(retries):
        try:
            with urllib.request.urlopen(req, timeout=2) as r:
                return json.load(r)
        except urllib.error.HTTPError as e:
            if e.code < 500:
                raise   # 4xx: retrying will not help
        except urllib.error.URLError:
            pass        # transport died; effect may still exist
    raise RuntimeError('gave up after retries')


def run(dedupe):
    STATE.update({'effects': 0, 'seen': {}, 'fail_next': True, 'dedupe': dedupe})
    server = HTTPServer(('127.0.0.1', 8765), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    try:
        call({'idempotency_key': 'ticket-7f3a', 'action': 'create_ticket'})
    finally:
        server.shutdown()
    return STATE['effects']


if __name__ == '__main__':
    print('naive server, side effects:', run(dedupe=False))
    print('deduping server, side effects:', run(dedupe=True))

Save it as retry_ledger.py and run it:

python3 retry_ledger.py

Output from my local run:

naive server, side effects: 2
deduping server, side effects: 1

Two side effects from one logical operation. That is the whole lesson in two lines. Change the port if 8765 is busy. These numbers come from a local simulation, not from any hosted endpoint.

Decision table: what to do when a call fails

What you observed	Effect may have committed?	Safe action
Timeout, connection reset	Yes	Retry with the same key
429	Usually no	Back off, then retry with the same key
500 / 502 / 503	Yes	Retry with the same key
400 / 422	No	Fix the request, do not retry
409 conflict	Yes	Treat it as already done

Notice how often retry with the same key appears. That repetition is the point, not sloppy writing.

Where the key lives when the loop runs on one host

If your model call and your tool call run next to each other, keep the key in one place and pass it down. Do not let them invent keys independently.

MonkeyCode offers free model access and a free server option, which is one way to run a loop like this without provisioning a second machine. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat both as vendor availability claims, not as guarantees about quotas, throughput, or permanence. The harness above does not depend on either one, and you can run it against any endpoint.

One trap if you co-locate everything: a restart wipes your in-memory dedupe dict. Write the key and the outcome to disk before you send the request. Otherwise your free server becomes a duplicate factory.

Limitations and who should skip this

Be honest about the boundaries:

• This harness is a local simulation, not a benchmark or a load test.
• Server-side dedupe only works when the effect can be keyed at all.
• A process-local dict is not a durable store. Production needs a shared one with a retention window.
• Keying everything doubles write volume and adds a lookup on every call. That cost is real.

Do not reach for this if your tools are all reads. Do not reach for it if re-running the entire job from scratch is cheap and clean. And do not reach for it if duplicates are genuinely harmless in your domain. In those cases, retries without keys are fine.

The one line I would keep

If you keep a single habit from this post, keep this one: generate the idempotency key before the first attempt, and reuse it for every retry.

That line is dull. It is also the difference between four tickets and one.

If you want to reproduce the failure honestly, the harness plus any endpoint you can point it at is enough. Watching the naive server report 2 is more convincing than any argument I can write here.