GWEN

Posted on Jul 30

Your AI App Needs a Timeout Strategy, Not Just a Better Model

#ai #api #llm #python

Most AI apps handle successful requests well.

They send a prompt, wait for the response, and display the result. The problem starts when the model becomes slow or temporarily unavailable.

A request that normally takes two seconds may suddenly take fifteen. Without a timeout strategy, users are left waiting, and your application appears broken.

Set task-specific timeouts

Different tasks need different limits:
This is the TokenBay API endpoint I use at work. If you want to try it, there is currently a discount available:

[Try TokenBay]https://www.tokenbay.com/?utm_source=devto&utm_medium=community_content&utm_campaign=week1_free_content

TIMEOUTS = {
    "chat": 8,
    "classification": 5,
    "document_summary": 30,
}

Use the timeout when sending the request:

response = client.chat.completions.create(
    model="gpt-5.4-mini",
    messages=messages,
    timeout=TIMEOUTS["chat"],
)

A real-time chat request should fail quickly. A document summary may reasonably need more time.

Retry temporary failures

Do not retry every error. Invalid API keys and bad requests will not fix themselves.

Retry only temporary failures such as timeouts, rate limits, and server errors:

import time

def generate_response(messages, model):
    for attempt in range(2):
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
                timeout=8,
            )
        except TimeoutError:
            if attempt == 1:
                raise
            time.sleep(2)

Retries should be limited. Otherwise, a slow provider can create more traffic, higher costs, and an even worse outage.

Add a fallback model

For important user-facing features, use a fallback:

MODELS = [
    "gpt-5.4-mini",
    "claude-sonnet-4.6",
]

def generate_with_fallback(messages):
    for model in MODELS:
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
                timeout=8,
            )
        except TimeoutError:
            continue

    raise RuntimeError("All models failed")

An OpenAI-compatible gateway makes this easier because your application can keep the same client structure while testing different models.

For example:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokenbay.com/v1",
    api_key="YOUR_TOKENBAY_API_KEY",
)

You can change the model without adding another SDK or rewriting your request logic.

Monitor the right metrics

Track more than average latency:

Timeout rate
Retry rate
Fallback rate
P95 latency
Cost per successful request
Failure rate by model

A model that is fast most of the time but occasionally takes twenty seconds may still create a poor user experience.

Final thought

A better model cannot fix an application that waits forever or retries blindly.

Production AI apps need clear timeouts, limited retries, tested fallbacks, and latency monitoring.

The goal is simple:

Fail quickly when necessary, recover when possible, and keep the user experience predictable.

DEV Community