Jordan Huang

Posted on Sep 2

That Free AI Call Probably Went Through. Stop Blind Retries

#ai #testing #api #python

Your free-tier AI endpoint times out. Your first reflex: press retry. But did the request arrive? Did it run? Did it burn quota? You don't know yet.

Blind retries make that mystery worse. This post is a myth-busting field guide for retrying free AI calls. It covers three myths, a three-minute probe, and a decision table you can use today.

The three myths

Myth 1: Timeout means the request never went out

A request is not one atomic action. It has phases: DNS, TCP, TLS, headers, payload, response. A read timeout often happens in the last phase, long after the payload reached the server.

So the server may have processed your prompt. You lost contact, not evidence. The timeout is a connection statement, not a server statement.

Myth 2: Retrying again is harmless

Retrying on a non-idempotent API creates a second side effect. With free access, that side effect is usually quota consumption. More requests, more tokens, more cost.

Retry is not a noun. It's an action with a consequence you can't see.

Your client timeout is a design decision. Too aggressive timers cause ghost retries. Too loose timers make the user wait forever. You need to calibrate against the actual endpoint, not against a guide from last year.

Myth 3: Exponential backoff fixes everything

Backoff reduces the storm. It queues your retry. It does not de-duplicate. If your first attempt actually ran, backoff just delays the duplicate. Now you have one big success and one ghost success.

Backoff helps servers. It doesn't protect your quota.

A three-minute probe for your endpoint

You can't trust docs forever. Endpoints evolve. Write a tiny probe.

import sys
import requests
from requests.exceptions import ConnectTimeout, ConnectionError, ReadTimeout

endpoint = sys.argv[1]
payload = {"prompt": "Say 'duplicate' once."}

try:
    r = requests.post(endpoint, json=payload, timeout=(3, 20))
    print("phase: response", r.status_code, "after", round(r.elapsed.total_seconds(), 2), "s")
except ConnectTimeout:
    print("phase: connect_timeout  -> request probably never sent")
except ReadTimeout:
    print("phase: read_timeout     -> server may have processed payload")
except ConnectionError:
    print("phase: connection_error -> socket failed before completion")

Run it a few times. If you see read_timeout, your retry risk is high. Connect timeouts are safer to retry. Read timeouts are the gray zone.

Curl also looks easy:

curl --max-time 5 --retry 2 --retry-delay 1 \
  -X POST "$ENDPOINT" \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"hi"}'

Curious? Curl has --retry. And curl will happily issue multiple requests under the hood. But it won't tell you whether the server already saw the first one.

Decision table: can I retry?

Failure phase	Retry?	Rule
DNS / TCP / connect timeout	Yes	Payload never sent. Safe to retry.
Read timeout	No*	Server may have processed payload.
HTTP 408, 429, 5xx	Yes	Server failed explicitly. Use backoff.
HTTP 200, malformed body	Maybe	Generation may have succeeded. Save the body and inspect.

*Only retry read timeouts if you can de-duplicate with a request ID or stored state.

The corrected mental model

Calling this operation "retry" is a lie. You are making a new attempt when you cannot confirm failure. Free AI endpoints make that lie expensive, because quota disappears in both attempts.

Do this instead:

Log before sending: prompt hash, timestamp, request ID.
On timeout, capture the phase from the probe.
If it's a read_timeout, wait, then check your own store: "Did this prompt already run?"
If you have no deduplication, retry only once. Mark the record as possibly_duplicate.

Treat retries as jobs with unknown side effects, not a magic button.

Where to run this probe?

Your laptop works. A cron job works too. You can also run it on a free server and test against a free model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access plus a free server option, which is enough for this kind of experiment. Use it as a sandbox. Not for a production SLA.

Who should skip this

You should skip this approach if you already have explicit idempotency keys. You should also skip it if you need strong consistency in a critical pipeline. And you should skip it if you're trying to benchmark raw speed. This probe teaches failure phases, not throughput.

Then the next time a timeout appears, you already know your failure phase. You won't press retry on a coin flip.

Until your probe says otherwise, every blind retry is a wager. Bet with evidence.

Prefer to try? Check MonkeyCode's free model access when you want a dedicated spot for these probes. Then design your retry policy from data, not hope.

DEV Community