AI Wrote Your Idempotency Layer? Replay Requests Until It Breaks

The fastest way to find out whether an AI-generated idempotency layer actually works is to stop reading the generated code and start replaying requests against a real disposable server, because most failures hide in ordering, retries, and collision handling rather than syntax.

Idempotency is often treated as a binary property—either a handler can run twice without changing the result or it cannot—but in production it behaves like a stack of implied contracts. The handler must know which fields participate in the identity key, what to do when a retry carries the same key and a different body, whether a timeout followed by a retry should return a stored result or execute the action again, and what happens when the process restarts between attempts. Generated code often satisfies the happy path and quietly encodes the wrong assumption about one of those transitions, so the only reliable way to judge it is to make the transition happen.

Instead of asking a model to explain why the code is correct, you can put the model to work at the edges of the test loop. First, let one model generate or review a small HTTP service that processes a charge request under an idempotency key. Then deploy that service to a temporary server, because local mocks tend to hide real HTTP semantics, process lifecycles, and the kind of network retry that confuses poorly written handlers. MonkeyCode's free server option gives you a place to run that service; its free model access gives you a second model for producing adversarial request sequences. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Start with a deliberately small service. The version below is illustrative, not production, and keeps state in memory so the interesting behavior is visible.

from fastapi import FastAPI
from uuid import uuid4

app = FastAPI()
store = {}

@app.post('/charges/{key}')
def charge(key: str, body: dict):
    if key in store:
        if store[key]['amount'] != body['amount']:
            return {'error': 'key_conflict'}, 409
        return {'id': store[key]['id'], 'amount': store[key]['amount']}, 200
    record = {'id': str(uuid4()), 'amount': body['amount']}
    store[key] = record
    return record, 201

A small replay harness is enough to turn those contracts into a pass or fail signal. It sends a sequence of requests for each case and prints the status code and response body next to the expected result.

import requests

BASE = 'https://your-throwaway-server.example'

cases = {
    'duplicate_same_body': ('invoice-1', [{'amount': 100}, {'amount': 100}], [201, 200]),
    'same_key_new_body': ('invoice-2', [{'amount': 100}, {'amount': 101}], [201, 409]),
    'different_key': ('invoice-3', [{'amount': 100}, {'amount': 100}], [201, 201]),
}

for name, (key, bodies, wants) in cases.items():
    print(f'--- {name} ---')
    for body, want in zip(bodies, wants):
        r = requests.post(f'{BASE}/charges/{key}', json=body, timeout=5)
        print(r.status_code, r.json(), 'expected', want)

Each row gives you one concrete contract. A duplicate with the same body should return the stored result with a 200, a retry with a different body should return a 409, and a new key should create another record with a 201. If the generated service returns 201 for the second request in the second case, you have found the exact place where the handler treats key existence as the whole story.

Once the first three cases run, prompt the free model with the request schema and ask it for ten more sequences that are likely to confuse an idempotent endpoint. You will usually get useful categories: trailing whitespace in the key, equivalent but differently typed fields, a retry after a client timeout, a duplicate that arrives before an earlier request has completed, and a crash between write and response. Add each suggestion to the harness only if you can state the expected outcome in advance; cases without a clear oracle add noise.

Most generated implementations fail on the same-key-different-body case, because they check existence before content. Some fail on restart, because in-memory state disappears and a second duplicate creates a new record. Others fail on concurrent duplicates, because two requests both miss the store and both create a row. You may not catch the concurrency race with sequential replay, but you can catch the first two faults in seconds.

When the free server returns a connection reset or a 502 during a replay, resist the temptation to blame the handler immediately; rerun the case against a local instance or wait for a stable window. A free server is a convenience, not a controlled benchmark environment, and network noise will otherwise teach you false lessons about the generated code.

This approach is not a replacement for transaction-level tests if the operation touches payments, inventory, or another system where a duplicate write is expensive. In those settings, test with the actual database, isolation level, and locking strategy you use in production, and treat the throwaway loop only as an early design check. Similarly, if your endpoint's idempotency depends on a database unique constraint rather than an in-memory store, the minimal service above will not reveal constraint-violation races.

After you fix the handler, save the replay cases in a file next to the code. The next time a model changes the logic, run the same file first. If a generated patch cannot pass the sequences that previously failed, you have a much better reason to reject it than the vague feeling that the code looks suspicious.