Retries, timeouts and model fallback for OpenAI-compatible APIs in ~40 lines of Python

Disclosure: I work on the APIGOTO team. APIGOTO is an independent OpenAI-compatible API gateway and is not affiliated with OpenAI, Anthropic or Google. The pattern below works with any OpenAI-compatible endpoint, ours or not.

When a prototype calls an LLM API, the first production problem is rarely the prompt. It's a 429, a slow response that hangs your worker, or one model being temporarily unavailable. Here is the small wrapper I use to handle all three.

The idea

1. Set a hard timeout on every request.
2. Retry rate-limit and server errors with exponential backoff.
3. If a model keeps failing, move on to the next model in a list.

Code

import os, time
from openai import OpenAI, APIStatusError, APITimeoutError, APIConnectionError

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL", "https://api.china-aiapi.com/v1"),
    api_key=os.environ["LLM_API_KEY"],
    timeout=30,      # never wait forever
    max_retries=0,   # we handle retries ourselves
)

RETRYABLE = {429, 500, 502, 503, 504}

def chat(messages, models, attempts=3):
    last_err = None
    for model in models:
        for i in range(attempts):
            try:
                r = client.chat.completions.create(model=model, messages=messages)
                return model, r.choices[0].message.content
            except APIStatusError as e:
                last_err = e
                if e.status_code not in RETRYABLE:
                    break          # 400/401/404: retrying won't help, try next model
            except (APITimeoutError, APIConnectionError) as e:
                last_err = e
            time.sleep(min(2 ** i, 8))
    raise RuntimeError(f"all models failed: {last_err}")

model, text = chat(
    [{"role": "user", "content": "Summarize HTTP 429 in one sentence."}],
    models=["your-primary-model", "your-backup-model"],
)
print(model, text)

Notes

• Use the model IDs your provider actually lists (client.models.list()), not names you assume exist.
• A 401 or 404 means fix your key or model name; don't retry it.
• Log which model answered. When the fallback fires often, you want to know.
• Because only base_url and the key change, you can point this at OpenAI directly, a self-hosted server, or a gateway.

If you want to try it against our gateway, the base URL is https://api.china-aiapi.com/v1, and pricing is pay-as-you-go: www.china-aiapi.com. Feedback on the pattern is welcome in the comments.