Make Free Model Requests Idempotent Before You Add Retries

#ai #api #security #devops

Add retries only after the request knows what it already did. Free model endpoints and free servers fail in the worst place: after the work may have run but before you receive the response.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I rely on two operator-supplied availability claims only: MonkeyCode has free model access and a free server option.

The workflow below is a client-side guard, not a provider guarantee.

Why retry is not a default

A free server worker can be recycled mid-request.
A free model endpoint can time out after generation.
A retry can duplicate a downstream write or return a mixed JSON payload.
A plain timeout does not tell you whether the provider already ran the request.

The rule is boring but useful: keep request intent, raw response, and accepted result separate. A retry is safe only when the accepted result is absent and the journal shows no partial downstream side effect.

The idempotent request journal

The artifact is a small Python journal. It hashes the prompt and request parameters into a key, appends events to a JSONL file, and atomically writes a result only after the payload parses as complete JSON.

import hashlib
import json
import os
import time
from pathlib import Path

def request_key(prompt, params):
    canonical = json.dumps({'prompt': prompt, 'params': params}, sort_keys=True)
    return hashlib.sha256(canonical.encode()).hexdigest()[:16]

class Journal:
    def __init__(self, root='.reqjournal'):
        self.root = Path(root)
        self.root.mkdir(parents=True, exist_ok=True)

    def append(self, key, event, detail=None):
        line = {'ts': time.time(), 'event': event, 'detail': detail}
        with (self.root / f'{key}.jsonl').open('a') as f:
            f.write(json.dumps(line) + '\n')

    def last_event(self, key):
        path = self.root / f'{key}.jsonl'
        if not path.exists():
            return None
        return json.loads(path.read_text().strip().splitlines()[-1])['event']

    def result_path(self, key):
        return self.root / f'{key}.result.json'

def validate(raw):
    data = json.loads(raw)  # no partial JSON accepted
    if not isinstance(data, dict) or 'choices' not in data:
        raise ValueError('unexpected envelope')
    return data

def request_once(journal, key, call):
    journal.append(key, 'SENT')
    try:
        raw = call()
    except TimeoutError:
        journal.append(key, 'TIMEOUT')
        raise
    payload = validate(raw)
    tmp = journal.result_path(key).with_suffix('.tmp')
    tmp.write_text(json.dumps(payload))
    os.replace(tmp, journal.result_path(key))
    journal.append(key, 'COMPLETE')
    return payload

def request(journal, key, call, max_attempts=3):
    result = journal.result_path(key)
    if result.exists():
        journal.append(key, 'CACHE_HIT')
        return json.loads(result.read_text())

    for attempt in range(1, max_attempts + 1):
        before = journal.last_event(key)
        journal.append(key, 'ATTEMPT', {'n': attempt, 'after': before})
        try:
            return request_once(journal, key, call)
        except TimeoutError:
            if attempt == max_attempts:
                journal.append(key, 'DEAD_LETTER')
                raise
        except ValueError as exc:
            journal.append(key, 'INCOMPLETE', {'error': str(exc)})
            if attempt == max_attempts:
                raise
    raise RuntimeError('unreachable')

os.replace is the important line. It swaps the temporary file into place only after validation succeeds. A caller can never read a half-written result.

What runs where

Use the free server option as the disposable worker. Point the journal at storage you control, not at a temp directory that disappears with the worker. The free model access is the call target.

This split matters because the free server can die and reappear, while the journal can still answer one question: what did we accept before the timeout?

Test plan

Run these checks before adding more retry logic.

Same key and same prompt: the second run returns the cached result and logs CACHE_HIT.
Kill the process after SENT is written but before COMPLETE: the result file is absent, so a retry can run.
Feed truncated JSON: no result file is created and the journal logs INCOMPLETE.
Feed valid JSON: the temp file is renamed into place, so no caller sees a partial result.

The decision table stays small.

Last journal event	Result file exists?	Retry?
none	no	yes
`SENT` or `TIMEOUT`	no	yes, same key
`INCOMPLETE`	no	yes, do not reuse partial payload
`COMPLETE`	yes	no, return cache
`DEAD_LETTER`	no	no, inspect journal

Limitations

This is at-most-once for downstream work, not provider-side exactly-once. A timeout after SENT can still cause a duplicate provider call.
If the provider bills per request or token, retries can still cost extra. Check your plan before enabling retry.
The journal is only as durable as the storage path. Keep it on persistent or object storage if the free server is ephemeral.
This does not validate model content, only the envelope shape. Add a schema or contract probe for strict response formats.
If model output writes directly to the filesystem or shell, gate those writes before execution. The journal does not sandbox side effects.

Who should not use this

You need strict exactly-once delivery. Use a database outbox or a provider idempotency key.
Your free quota is tiny. Retrying may waste it; fail fast instead.
Your worker has no durable place for the journal. In-memory state lasts exactly as long as the free server process.

The useful version is not a retry loop; it is a record of what already happened. Add the journal first, then let retry read it.

Try this once: kill the worker between SENT and COMPLETE, then restart and watch the journal continue instead of guessing.

DEV Community