The Retry Was Working as Intended, and the Free Tier Kept Failing

It started with the kind of complaint that makes you question your career choices: the model got worse after lunch. A small summarization service on a free server was rejecting every third request, and the failures always began right after one slow response. I was ready to blame the provider, the network, or the alignment of the planets. The truth was more embarrassing, because the attacker was my own retry logic.

The setup was painfully ordinary. A Python service on a free server called a free model endpoint, summarized a document, and stored the result. Disclosure: This article was prepared as part of MonkeyCode's product outreach; I ran the experiment on MonkeyCode's free model access and free server option. The HTTP client had retries because free infrastructure is free for a reason, and transient timeouts are part of the deal.

The symptom was a wave pattern. One request timed out, three retries followed, and then every legitimate request for the next minute came back with a 429. The provider's status page was green, my logs showed a healthy mix of successes, and the service recovered on its own. So why did real users keep hitting errors right after a timeout?

The Logs Only Told Me the Ending

I was logging the final outcome of every call: success or failure. Retries were invisible because the HTTP client swallowed them, which is the first debugging lesson. If you do not log decisions, you cannot debug decisions. I added one line per retry attempt — attempt number, status code, wait time, reason — and the pattern jumped out immediately.

Anatomy of a Retry Storm

Here is what the ledger revealed, in three acts:

1. A single request hit a slow model response and timed out after ten seconds.
2. My client retried three times, and each retry landed inside the same sixty-second rate-limit window.
3. Those retries consumed the quota that real requests needed, so the next minute of traffic failed with 429s.

The backoff was polite in isolation: one second, two seconds, four seconds, plus jitter. But the rate-limit window was sixty seconds, and the quota was small, so three retries from one request used what three separate requests should have used. Worse, the endpoint was not idempotent, which meant every retry re-created the same summarization job. A single timeout produced duplicate records and a rate-limit penalty.

Three Changes That Fixed It

Idempotency keys. Every request now carries a hash of its payload, so the server can recognize a retry and return the original result instead of doing the work again.

Retry-After respect. On a 429, the server explicitly tells you how long to wait, and my backoff was ignoring that number. The header is now the first source of truth, and exponential backoff is only a fallback.

A circuit breaker. After five failures in sixty seconds, the client stops retrying entirely and fails fast. A quick error is better than three slow retries that make the whole tier angrier.

The Simplified Code

This is a minimal version of the client I shipped, and it is not the production code. It reproduces the fix well enough to run against any HTTP endpoint.

import hashlib
import json
import random
import time
from datetime import datetime, timezone

import httpx

RETRYABLE_STATUS = {408, 429, 502, 503, 504}


class PoliteRetryClient:
    def __init__(self, max_attempts=3, base_delay=1.0, failure_threshold=5):
        self.max_attempts = max_attempts
        self.base_delay = base_delay
        self.failure_threshold = failure_threshold
        self.failure_times = []
        self.ledger = []

    def _circuit_open(self):
        now = time.monotonic()
        self.failure_times = [t for t in self.failure_times if now - t < 60]
        return len(self.failure_times) >= self.failure_threshold

    def _key_for(self, payload):
        raw = json.dumps(payload, sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(raw.encode()).hexdigest()

    def _backoff(self, attempt):
        return self.base_delay * (2 ** (attempt - 1)) + random.uniform(0, 0.5)

    def _log(self, attempt, status, reason, key, waited):
        self.ledger.append({
            "attempt": attempt,
            "status": status,
            "reason": reason,
            "waited": round(waited, 2),
            "idempotency_key": key,
            "at": datetime.now(timezone.utc).isoformat(),
        })

    def call(self, url, payload):
        if self._circuit_open():
            raise RuntimeError("circuit open: too many recent failures")

        key = self._key_for(payload)
        headers = {"Idempotency-Key": key}

        for attempt in range(1, self.max_attempts + 1):
            try:
                response = httpx.post(url, json=payload, headers=headers, timeout=10.0)
            except httpx.TimeoutException:
                wait = self._backoff(attempt)
                self._log(attempt, None, "timeout", key, wait)
                self.failure_times.append(time.monotonic())
                time.sleep(wait)
                continue

            if response.status_code == 200:
                self._log(attempt, 200, "success", key, 0)
                return response.json()

            if response.status_code not in RETRYABLE_STATUS:
                self._log(attempt, response.status_code, "non_retryable", key, 0)
                response.raise_for_status()

            retry_after = response.headers.get("Retry-After")
            wait = float(retry_after) if retry_after else self._backoff(attempt)
            self._log(attempt, response.status_code, "retryable", key, wait)
            self.failure_times.append(time.monotonic())
            time.sleep(wait)

        raise RuntimeError(f"gave up after {self.max_attempts} attempts")

Three parts of that file matter most:

• The idempotency key is derived from the sorted payload, so retries of the same request look identical to the server. If the server stores that key, duplicate work dies at the door.
• The Retry-After header overrides the backoff whenever the server knows better. The exponential curve is now a fallback, not a policy.
• The circuit breaker checks a sliding sixty-second window before every attempt. When the tier is clearly struggling, the client says no instead of piling on.

Debugging Techniques That Actually Found the Bug

Reproduce with the smallest loop. One request succeeded, two requests succeeded, and three requests in the same window failed. The bug only appeared when requests shared a time slice.

Log the decision, not just the result. The retry ledger turned an invisible storm into a readable story with timestamps and reasons. That one change saved more debugging time than any other.

Read the headers. Retry-After existed the entire time, and my client simply ignored it. The server was telling me exactly what to do.

Question the obvious culprit. The provider's status page was green because the provider was fine. I was the source of my own outage, and the retry log proved it.

Where This Pattern Breaks

If your endpoint has side effects you cannot make idempotent, do not retry automatically. Fail fast and let a human decide, because a duplicate payment is worse than a timeout.

If your rate limit is extremely tight, even three polite retries can starve other requests. Consider a queue or a single-flight pattern instead of retrying in place.

A circuit breaker with a low threshold can mask a real outage and turn a slow degradation into a hard failure. Tune the threshold against your actual traffic, not a guess.

None of this fixes model quality. If the model returns a confident but wrong summary, no retry policy will save you, and that is a completely different debugging story.

The model never got worse that afternoon, and the free tier was not flaky. My retry policy was eating the quota, and the only reason I found it was the ledger. Now every retry decision is logged, every request carries an idempotency key, and the circuit breaker knows when to stop. Before your next incident, audit your retry policy — it is cheaper than debugging it live.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.