Three Retry Myths That Make Your Free-Tier Model Calls Worse

Retry is a reflex. It is not a strategy.

You send a request. It times out. You hit retry. The second call is slower. Why? Because you joined an overloaded queue.

This is common on free-tier model servers. Developers repeat three retry myths. Each myth makes the problem worse.

Myth #1: Retry immediately after a failure

Immediate retry feels natural. One failure. Try again. The math is simple.

Free-tier servers do not follow simple math. Failures cluster. When a shared host is saturated, many clients fail at once. Your retry joins that same saturated queue.

Evidence: run a probe. Errors usually appear in bursts. A retry after one millisecond often hits the same burst.

Corrected mental model: retry is a backoff control, not a second chance. Add a base delay. Add jitter. Exponential backoff with jitter is not folklore. It is basic control theory.

Myth #2: Timeouts should match the median latency

Read your latency report. The p50 is 400ms. Nice. The p95 is eight seconds. That is the number that breaks users.

A common mistake is setting one timeout from the median. Free-tier long tails make that dangerous. You cut off healthy slow calls. Or you wait forever on dead ones.

Split the timeout into phases. A connection can succeed in 100ms. Then the server can sit on the prompt for twenty seconds. That is queueing, not connection failure.

Use separate connect and read timeouts. Measure each phase. The probe below does that.

Myth #3: Every HTTP error deserves a retry

HTTP 400 means your payload is wrong. Retrying it makes you persistent, not correct. HTTP 429 means you are too fast. Respect Retry-After. HTTP 503 might be transient. But not always.

Blind retries waste quota. They also add load to an already overloaded free tier.

Corrected mental model: classify errors before touching the retry button. 4xx is permanent. 429 is rate limit. 5xx is maybe transient. Treat retries as a state machine.

The artifact: a failure classifier

Here is a small script. It records the phases of every request. It does not reveal the server's secrets. It reveals where your own request loses time.

Save it as probe.py. Change HOST, PATH, and PAYLOAD to match your endpoint. Run it with Python 3.8+.

import collections
import http.client
import json
import time

def probe(host, path, payload, timeout=10):
    t0 = time.monotonic()
    try:
        conn = http.client.HTTPSConnection(host, timeout=timeout)
        conn.request('POST', path, body=json.dumps(payload),
                     headers={'Content-Type': 'application/json'})
        t1 = time.monotonic()
        resp = conn.getresponse()
        t2 = time.monotonic()
        body = resp.read()
        t3 = time.monotonic()
        return {
            'status': resp.status,
            'connect_send': t1 - t0,
            'ttfb': t2 - t1,
            'read': t3 - t2,
            'total': t3 - t0,
        }
    except Exception as exc:
        return {'error': type(exc).__name__, 'message': str(exc)}

def classify(res):
    if 'error' in res:
        return 'connection_error'
    if res['status'] >= 500:
        return 'server_error'
    if res['status'] == 429:
        return 'rate_limited'
    if res['status'] >= 400:
        return 'client_error'
    if res['ttfb'] > 2:
        return 'slow_ttfb'
    return 'ok'

counts = collections.Counter()
for _ in range(100):
    result = probe('your-model-host', '/v1/complete', {'prompt': 'ping'})
    counts[classify(result)] += 1
    print(result)

print(counts)

No third-party packages needed. http.client is part of the standard library.

What the output looks like

A healthy endpoint may print a clean 200 each time. A saturated one may print a mix.

{'status': 429, 'connect_send': 0.12, 'ttfb': 1.4, 'read': 0.02, 'total': 1.54}
{'error': 'Timeout', 'message': 'timed out'}
Counter({'slow_ttfb': 41, 'ok': 30, 'rate_limited': 14, 'server_error': 9, 'connection_error': 6})

This tells a story. The endpoint accepts connections. Many requests wait in a queue. The 429s appear after slow ttfb values. That is the fingerprint of saturation.

How to read the output

• connection_error burst: network or client limits. Check your side.
• server_error cluster: the host is saturated. Do not retry blindly.
• rate_limited: you are too aggressive. Add jitter and backoff.
• slow_ttfb: queueing, not model speed. Reduce payload size or concurrency.
• client_error: fix the payload. Never retry.

Also watch connect_send. If it stays flat while ttfb grows, the network is fine. The server is the bottleneck.

Turn the output into a policy

• If rate_limited is above 5%, increase the base backoff.
• If connection_error is above 20%, inspect the client. Do not blame the model.
• If server_error clusters, write a small alert. Retry will not fix it.
• If slow_ttfb is above 30%, check your input size. Big prompts spend more time in queues.

Limitations and who should skip this

This classifier does not measure model quality. It cannot see the scheduler. It only sees your client's perspective.

Do not use this as a real-time solution. If you need low latency, this is already too late. You need a queue or a reserved server.

Do not use a single prompt as your sample. Different payload sizes create different failure patterns.

Finally, do not trust a ten-call run. Run at least one hundred calls. Sample size matters.

Where this fits in practice

Once you know the failure class, retry policy becomes simple. Some errors get backoff. Some get no retry. Some get a longer timeout.

I run this kind of probe on low-cost endpoints. If you are testing against MonkeyCode's free model access or its free server option, this is a cheap and safe first step. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The method is generic. You can use it with any HTTP model API. Try it once. Then your retry button can stay quiet.