DEV Community: AIFast Hub

Your AI Agent Got a 200. The Tool Call Still Failed.

AIFast Hub — Fri, 21 Aug 2026 11:43:10 +0000

AI agents rarely fail in the place where the log says they failed.

A request can return HTTP 200, the model can produce a perfectly valid response, and the workflow can still be wrong because a tool call was rejected, timed out, or executed against stale arguments. The application then continues with an empty result and writes a confident answer.

I have seen this happen in two forms:

the model returns a tool call, but the application never executes it;
the application executes the tool, gets an error, and sends the next model request without preserving that failure.

Both bugs look like model quality problems. They are usually integration problems.

This guide shows a small failure-aware loop for OpenAI-compatible APIs. It uses ordinary Python, the official OpenAI client, and a deliberately unreliable tool. The point is not to build a framework. The point is to make every transition visible enough to test.

Start with the HTTP boundary

Before adding an agent framework, prove that the API boundary works. Keep the base URL and model ID configurable:

export OPENAI_BASE_URL="https://www.aifast.hk/v1"
export OPENAI_API_KEY="replace-with-your-key"
export OPENAI_MODEL="copy-an-exact-model-id-from-your-provider"

Check model discovery first:

curl -sS "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Accept: application/json" | jq -r '.data[].id'

Then send the smallest chat request:

curl -sS "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$OPENAI_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK\"}],\"temperature\":0}"

Do not put the real key in a shell history, issue, or screenshot. The examples use an environment variable so the credential stays outside the request body.

A successful chat request proves very little about tools. Model discovery, chat completions, streaming, structured output, and tool calls can fail independently. Test the feature you plan to use.

The broken loop

Here is the kind of loop that creates silent failures:

response = client.chat.completions.create(
    model=model,
    messages=messages,
    tools=tools,
)

message = response.choices[0].message
messages.append(message)

for call in message.tool_calls or []:
    result = run_tool(call.function.name, call.function.arguments)
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": result,
    })

# The next request happens even when run_tool() returned an exception string.

There are three problems here.

First, the code does not distinguish a tool result from a tool failure. Second, it does not validate that the requested tool exists. Third, it assumes that a model response containing tool_calls is enough to prove that the requested action happened.

The model only suggested an action. Your application owns execution.

Make failures part of the message history

A safer version records a structured result for every call. The model can then decide whether to retry, ask for a missing input, or explain that the task cannot be completed.

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL", "https://www.aifast.hk/v1"),
)

model = os.environ["OPENAI_MODEL"]

TOOLS = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for one city.",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"}
            },
            "required": ["city"],
            "additionalProperties": False
        }
    }
}]


def get_weather(city):
    # Replace this with a real provider call.
    if city.lower() == "timeout-city":
        raise TimeoutError("weather provider timed out")
    return {"city": city, "temperature_c": 21, "source": "demo"}


def execute_tool(call):
    name = call.function.name
    try:
        arguments = json.loads(call.function.arguments)
    except json.JSONDecodeError as exc:
        return {
            "ok": False,
            "error": "invalid_arguments",
            "message": str(exc),
        }

    if name != "get_weather":
        return {
            "ok": False,
            "error": "unknown_tool",
            "message": f"Tool {name!r} is not registered.",
        }

    if not isinstance(arguments.get("city"), str) or not arguments["city"].strip():
        return {
            "ok": False,
            "error": "validation_error",
            "message": "city must be a non-empty string",
        }

    try:
        return {"ok": True, "data": get_weather(arguments["city"])}
    except TimeoutError as exc:
        return {
            "ok": False,
            "error": "upstream_timeout",
            "retryable": True,
            "message": str(exc),
        }
    except Exception as exc:
        return {
            "ok": False,
            "error": "tool_error",
            "retryable": False,
            "message": str(exc),
        }

The important detail is not the weather function. It is the shape of the returned object. A caller can now tell the difference between successful data, malformed arguments, an unknown tool, and a retryable upstream error.

Preserve the tool call ID

The next model request must contain the assistant message with its tool call and a tool message carrying the same tool_call_id.

def run_agent(user_text, max_rounds=4):
    messages = [
        {
            "role": "system",
            "content": "Use tools when needed. Never invent tool results. "
                       "If a tool fails, explain the failure or try again only "
                       "when the error is retryable.",
        },
        {"role": "user", "content": user_text},
    ]

    for round_no in range(max_rounds):
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=TOOLS,
            temperature=0,
        )
        assistant = response.choices[0].message
        messages.append(assistant)

        calls = assistant.tool_calls or []
        if not calls:
            return assistant.content or "The model returned no text."

        for call in calls:
            result = execute_tool(call)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(result, ensure_ascii=False),
            })

    raise RuntimeError("agent exceeded the tool-call round limit")

Do not silently drop the assistant tool-call message. Do not create a tool message with a made-up ID. Providers may reject the sequence, while other providers or wrappers may hide the mistake and produce confusing output.

Test the failure path, not only the happy path

Most agent tests ask for a successful answer. That misses the bug.

You need at least these cases:

the model answers without a tool;
the model requests a valid tool and the tool succeeds;
the model sends malformed JSON arguments;
the model requests a tool that is not registered;
the tool returns a retryable timeout;
the provider returns an API error before the tool loop starts;
the model emits multiple tool calls in one response;
the loop reaches its maximum number of rounds.

A minimal test should assert the message sequence, not just the final text:

def test_tool_failure_is_preserved():
    result = execute_tool(type("Call", (), {
        "function": type("Function", (), {
            "name": "get_weather",
            "arguments": '{"city":"timeout-city"}',
        })(),
    })())

    assert result["ok"] is False
    assert result["error"] == "upstream_timeout"
    assert result["retryable"] is True

For an integration test, use a fake model response and inspect the second request. It should include the original tool call ID and a JSON error object. If the second request contains only the user's original question, your application discarded the evidence the model needs.

Retry only what is safe to retry

A timeout from an idempotent read operation can usually be retried with a limit and backoff. A payment, email, database write, or deployment cannot be retried safely just because the network timed out. The server may have completed the action before the client lost the response.

Keep retry policy with the tool, not in a blanket agent wrapper:

RETRYABLE = {"upstream_timeout", "connection_reset", "rate_limited"}


def should_retry(result, attempt):
    return (
        result.get("ok") is False
        and result.get("error") in RETRYABLE
        and attempt < 2
    )

For rate limits, respect the provider's retry headers when present. For 401, fix credentials or configuration; retrying the same request will not repair an invalid key. For 400, inspect the payload and model ID. For 5xx, capture the request ID and response body, then decide whether the operation is safe to repeat.

The status code is a starting point, not a diagnosis.

Log evidence without logging secrets

When a tool loop fails, record:

request ID, if the provider returns one;
model ID and API operation;
tool name and a hash or redacted form of arguments;
round number and tool-call ID;
HTTP status and provider error code;
elapsed time for the model and the tool;
whether the operation was read-only or mutating.

Do not record API keys, full authorization headers, private prompts, or user data by default. A log that proves the failure but leaks the credential is not a successful debugging setup.

What I check before calling an agent reliable

I run one real request through the same base URL used by the application. I verify the exact model ID from /models. Then I force a tool timeout and confirm that the next model request contains the failure, not an empty success placeholder.

That last test catches more bugs than another happy-path demo.

I also test multiple tool calls in one model response. The executor records each call ID separately and returns one tool message for every call, even when one succeeds and another fails. Collapsing those results into a single message makes it hard to tell which action produced which output.

For production incidents, I keep the provider request ID beside the tool-call ID. They answer different questions: the request ID helps the API provider trace the model request, while the tool-call ID connects the model's proposed action to the result my application returned. If either identifier disappears between log lines, debugging turns into guesswork.

Write operations need one more guard. Before retrying a payment, deployment, email, or database mutation, attach an idempotency key or check the target state. A timeout means the client did not receive a response; it does not prove the server abandoned the operation. Blind retries can turn one uncertain action into two real actions.

An OpenAI-compatible endpoint gives you a familiar request shape. It does not guarantee identical models, tool behavior, limits, or error details. Keep the wire-level test, tool executor, retry policy, and message history visible. When something breaks, you want evidence from the boundary instead of a polished answer that happens to be wrong.

References

The examples use an OpenAI-compatible API endpoint and a demo tool. Test the exact model and tool capabilities in your own environment before relying on them in production.

Debugging 401, 429, and 5xx Errors in OpenAI-Compatible APIs: A Practical Field Guide

AIFast Hub — Thu, 13 Aug 2026 08:43:41 +0000

A failed AI request often gets reduced to one line in an application log:

openai.APIStatusError: Error code: 502

That line is almost useless on its own.

A request may pass through an SDK, DNS, TLS, a reverse proxy, an API gateway, and an upstream model before the response reaches your code. A 401, 429, or 502 tells you where to start, but it does not tell you what happened.

The fastest debugging method I know is to stop treating every failure as "the API is down." First capture the raw response. Then classify the failure. Only after that should you decide whether a retry makes sense.

Capture the wire evidence first

SDK exceptions are convenient, but they hide details you need during an incident. Reproduce one failing call with curl before changing application code:

export OPENAI_BASE_URL="https://www.aifast.hk/v1"
export OPENAI_API_KEY="replace-with-your-key"
export OPENAI_MODEL="copy-an-exact-model-id-from-your-provider"

curl -sS \
  -D /tmp/ai-response.headers \
  -o /tmp/ai-response.body \
  -w $'http=%{http_code}\ndns=%{time_namelookup}\nconnect=%{time_connect}\ntls=%{time_appconnect}\nfirst_byte=%{time_starttransfer}\ntotal=%{time_total}\n' \
  "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$OPENAI_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK\"}],\"max_tokens\":8}"

This leaves you with three useful artifacts:

/tmp/ai-response.headers
/tmp/ai-response.body
DNS, connection, TLS, first-byte, and total timings

Do not paste the complete Authorization header into a ticket. Keep the status, request ID, rate-limit headers, error code, and timestamps. Redact credentials and user data.

A small shell helper makes the capture repeatable:

printf '%s\n' '--- headers ---'
sed -E 's/^(set-cookie:|authorization:).*/\1 [REDACTED]/I' /tmp/ai-response.headers
printf '%s\n' '--- body ---'
python3 -m json.tool /tmp/ai-response.body 2>/dev/null || cat /tmp/ai-response.body

If curl fails before it receives an HTTP status, you are not debugging a 401, 429, or 5xx yet. Check DNS, TLS, proxy variables, and network reachability first.

dig +short "$(python3 - <<'PY'
from urllib.parse import urlparse
import os
print(urlparse(os.environ['OPENAI_BASE_URL']).hostname)
PY
)"

curl -Iv "$OPENAI_BASE_URL/models"

A 401 is an identity or routing problem

A 401 means an HTTP server received the request and refused the credentials. It is not proof that the network is broken.

I check these in order:

Did the request go to the expected host?
Does OPENAI_BASE_URL include the required /v1 prefix exactly once?
Did the shell actually export the key used by this process?
Does the key belong to this provider, project, or organization?
Is an intermediate gateway stripping or replacing Authorization?
Does the account or key have an IP allowlist?

Check the variables without printing the secret:

python3 - <<'PY'
import hashlib
import os
from urllib.parse import urlparse

base = os.environ.get("OPENAI_BASE_URL", "")
key = os.environ.get("OPENAI_API_KEY", "")

print("scheme:", urlparse(base).scheme)
print("host:", urlparse(base).hostname)
print("path:", urlparse(base).path)
print("key_present:", bool(key))
print("key_length:", len(key))
print("key_fingerprint:", hashlib.sha256(key.encode()).hexdigest()[:12] if key else "missing")
PY

The fingerprint lets two operators confirm they are using the same key without exposing it.

Next, test the lightest authenticated endpoint the provider exposes:

curl -sS -i "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Interpret the result carefully:

/models returns 401: focus on the key, account, host, and gateway.
/models works but generation returns 401 or 403: inspect model permissions and routing policy.
The response is HTML: you may have hit a website route, WAF page, login page, or proxy error instead of the API.
The response is 404: verify the path before rotating keys.

Rotating a key before confirming the host is a common waste of time. A perfect key sent to the wrong service still fails.

A 429 is not one problem

Many applications treat every 429 as "requests per minute exceeded." That is too coarse.

A 429 can mean:

a short-window request-rate limit
a token-rate limit
a concurrency limit
an account usage or spend limit
depleted balance or quota
a provider-specific safety throttle

Read the response body and headers before retrying:

python3 - <<'PY'
import json
from pathlib import Path

body = Path("/tmp/ai-response.body").read_text(errors="replace")
try:
    data = json.loads(body)
except json.JSONDecodeError:
    print(body[:1000])
else:
    error = data.get("error", data)
    for key in ("type", "code", "message", "param"):
        print(f"{key}: {error.get(key)}")
PY

grep -iE '^(retry-after|x-ratelimit-|ratelimit-|x-request-id|request-id):' \
  /tmp/ai-response.headers

The error code matters more than the broad HTTP status. Retrying a temporary rate limit can work. Retrying an exhausted balance or enforced spend limit cannot restore access.

A safe retry loop respects Retry-After, adds jitter, and stops after a small number of attempts:

import email.utils
import random
import time
from datetime import datetime, timezone

import requests


def retry_delay(response, attempt):
    value = response.headers.get("retry-after")
    if value:
        try:
            return max(0.0, float(value))
        except ValueError:
            try:
                target = email.utils.parsedate_to_datetime(value)
                return max(0.0, (target - datetime.now(timezone.utc)).total_seconds())
            except (TypeError, ValueError):
                pass

    return min(20.0, (2 ** attempt) + random.random())


def post_with_bounded_retry(url, headers, payload, attempts=4):
    for attempt in range(attempts):
        response = requests.post(url, headers=headers, json=payload, timeout=(5, 90))

        if response.status_code != 429:
            response.raise_for_status()
            return response.json()

        try:
            code = response.json().get("error", {}).get("code", "")
        except ValueError:
            code = ""

        permanent = {
            "insufficient_quota",
            "organization_spend_limit_exceeded",
            "project_spend_limit_exceeded",
            "organization_usage_limit_exceeded",
        }
        if code in permanent or attempt == attempts - 1:
            response.raise_for_status()

        time.sleep(retry_delay(response, attempt))

One trap: the official SDK already retries some failures. If you add another retry layer around it, four application attempts can turn into many more wire requests. During debugging, set SDK retries explicitly so the count is observable:

import os

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_BASE_URL"],
    max_retries=0,
)

Then decide where retry ownership belongs. I prefer one retry layer, not an SDK loop hidden inside a job-worker loop hidden inside a queue retry.

Split 5xx by where it originated

5xx means a server failed, but "server" may refer to your reverse proxy, the API gateway, or the upstream model provider.

Start with the response shape:

JSON in the provider's normal error schema suggests the request reached the API layer.
An nginx or CDN HTML page suggests an edge or gateway failure.
An empty body with a gateway-branded header often means the upstream connection failed.
A request ID from the provider is evidence the provider saw the call.

Then compare timing:

Large DNS or TLS time: network path or certificate issue.
Fast 502: gateway rejected or could not open the upstream connection.
Response near a fixed duration such as 30 or 60 seconds: proxy or load-balancer timeout is likely.
Long first-byte time followed by 504: upstream generation exceeded a gateway deadline.

Run two controls before declaring an outage:

# Control 1: authenticated discovery
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' \
  "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY"

# Control 2: minimal non-streaming generation
curl -sS -o /tmp/minimal.json -w '%{http_code} %{time_total}\n' \
  "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$OPENAI_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"OK\"}],\"max_tokens\":4,\"stream\":false}"

If discovery works and only one model fails, the gateway is reachable. Test another model listed as available by the provider before changing DNS or TLS settings. If every model fails with the same gateway-generated response, collect evidence and check the provider's live status information.

For AIFast, the English model compatibility and status checker is one way to separate a model-specific maintenance event from a client-side configuration failure. Treat it as a provider-specific example; the same method applies to any gateway with a live model catalog and status source.

Streaming needs a separate test

A non-streaming request can succeed while streaming fails through a buffering proxy.

Use curl -N so output is not buffered by the client:

curl -N -sS \
  "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$OPENAI_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Count from 1 to 5, one number at a time\"}],\"stream\":true}"

What to watch:

All chunks arrive at once: a proxy may be buffering Server-Sent Events.
The connection closes without a terminal marker: inspect idle timeouts and upstream resets.
Non-streaming succeeds but streaming gets 502: compare gateway buffering and timeout policies.
Chunks are valid but your SDK crashes: save the raw event stream and inspect parser assumptions.

Do not use a single 200 non-streaming response as proof that an OpenAI-compatible endpoint supports your production workload.

Log failures in a form another person can use

A useful incident record answers six questions:

{
  "timestamp_utc": "2026-08-13T07:30:00Z",
  "operation": "chat.completions",
  "model": "provider-model-id",
  "stream": false,
  "http_status": 502,
  "error_code": "upstream_error",
  "request_id": "req_redacted",
  "attempt": 1,
  "duration_ms": 31240,
  "base_url_host": "api.example.com"
}

Keep the API key, prompt text, cookies, and personal data out of the log. If prompts are needed for reproduction, store a sanitized fixture separately.

In Python, catch connection failures separately from status failures:

import os

import openai

try:
    response = client.chat.completions.create(
        model=os.environ["OPENAI_MODEL"],
        messages=[{"role": "user", "content": "Reply with OK"}],
        max_tokens=8,
    )
    print("request_id:", response._request_id)
except openai.APIConnectionError as exc:
    print("transport_error:", repr(exc.__cause__))
    raise
except openai.APIStatusError as exc:
    print("status:", exc.status_code)
    print("request_id:", exc.request_id)
    print("body:", exc.response.text[:1000])
    raise

The request ID is the best bridge between your logs and a provider's logs. Record it on failures and successful canary requests.

The decision tree I use

When a production request fails, I follow this order:

No HTTP status: inspect DNS, TLS, proxy, and connection errors.
401: verify host, /v1, key fingerprint, project, and gateway header forwarding.
429: read error.code and rate headers. Retry only temporary limits.
5xx: identify the response owner, compare timings, run a second model and a non-streaming control.
Streaming-only failure: inspect buffering, idle timeout, and event framing.
Preserve the request ID, UTC time, model, operation, status, duration, and sanitized body.

The main lesson is boring but useful: status codes are categories, not diagnoses. A blind retry may hide a transient failure, multiply traffic during an outage, or delay the discovery that an account has no usable quota.

Capture the wire evidence first. The fix usually becomes much smaller after that.

Sources

Migrating to an OpenAI-Compatible API Without Breaking Production

AIFast Hub — Thu, 13 Aug 2026 05:40:33 +0000

Changing an OpenAI integration often looks like a two-line job:

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL", "https://www.aifast.hk/v1"),
)

That is enough for a smoke test. It is not enough for production.

I have seen migrations pass a simple chat request and then fail on model discovery, streaming, tool calls, retries, or a client that quietly appends the wrong path. The phrase "OpenAI-compatible" describes a protocol family, not a guarantee that every endpoint and feature behaves identically.

This guide uses the official OpenAI Python SDK and ordinary curl commands. The checks work with a self-hosted gateway, a local model server, or a managed API gateway.

Start with the wire, not your framework

Do not begin the migration inside an agent framework. Test the HTTP boundary first. Frameworks add their own model aliases, retries, fallbacks, and error wrapping, which makes a basic URL mistake much harder to see.

Set three environment variables:

export OPENAI_BASE_URL="https://www.aifast.hk/v1"
export OPENAI_API_KEY="replace-with-your-key"
export OPENAI_MODEL="copy-an-exact-model-id-from-your-provider"

Keep the model ID outside the code. Model catalogs change, and copying an ID from an old tutorial is a reliable way to get a 404.

First, check model discovery:

curl -sS "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Accept: application/json"

To print only the model IDs when jq is installed:

curl -sS "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY" | \
  jq -r '.data[].id'

A successful response proves four small but useful things: DNS works, TLS works, the base path is plausible, and the credential reaches an authenticated endpoint. It does not prove that chat, streaming, tools, or images work.

Now send the smallest useful chat request:

curl -sS "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"$OPENAI_MODEL\",
    \"messages\": [{\"role\": \"user\", \"content\": \"Reply with OK\"}],
    \"temperature\": 0
  }"

Use the exact catalog ID returned by your provider. A brand name such as "Claude" or "Gemini" is not an API model ID.

The `/v1` trap

Most migration failures I see are path construction mistakes.

The official Python SDK accepts a base_url. If the value already ends in /v1, your request becomes:

https://gateway.example/v1/chat/completions

That is usually correct. Problems start when a wrapper adds another version segment:

https://gateway.example/v1/v1/chat/completions

The opposite mistake also happens: a client expects the version in base_url, receives only https://gateway.example, and calls /chat/completions without /v1.

Before blaming the credential, log the final request URL. In Python, an httpx event hook is a clean way to do it without printing the key:

import os

import httpx
from openai import OpenAI


def log_request(request: httpx.Request) -> None:
    print(f"-> {request.method} {request.url}")


http_client = httpx.Client(
    event_hooks={"request": [log_request]},
    timeout=httpx.Timeout(60.0, connect=5.0),
)

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL", "https://www.aifast.hk/v1"),
    http_client=http_client,
    max_retries=0,
)

models = client.models.list()
print(models.data[0].id if models.data else "No models returned")

I disable retries during diagnosis. One request should produce one observable result. Once the integration is correct, retries can go back on.

A 401 is not a connectivity failure

These status codes point to different layers:

401: the server did not accept the credential. Check the header format, whitespace, key scope, and whether the application loaded the environment variable you expected.
403: the credential may be valid but lacks permission, or a security layer blocked the request.
404: check the final URL, endpoint family, and exact model ID. Some gateways return 404 for an unsupported endpoint.
429: you may have hit a request or token limit, exhausted quota, or reached upstream capacity. Inspect the response body and headers before retrying.
5xx: the gateway or an upstream service failed. Retry only if the operation is safe to repeat.

Do not turn every failure into a retry. Retrying a malformed request or bad key only creates noise.

Do not rely on SDK retry defaults without checking the version you deploy. If your application also wraps the SDK in a tenacity loop, one logical operation can produce far more requests than you intended. Set the retry count explicitly so the behavior stays predictable across upgrades.

Use explicit settings:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL", "https://www.aifast.hk/v1"),
    timeout=60.0,
    max_retries=2,
)

SDK timeout defaults can change between releases and may be much longer than a synchronous web request should tolerate. Set an explicit timeout for your workload and test the failure path before rollout.

Test streaming separately

A non-streaming response can succeed while streaming fails through a proxy, CDN, ingress controller, or application server. Buffering is the usual culprit.

Test the raw event stream:

curl -N "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"$OPENAI_MODEL\",
    \"messages\": [{\"role\": \"user\", \"content\": \"Count from 1 to 5 slowly\"}],
    \"stream\": true
  }"

curl -N disables its output buffering. You should see incremental data: events rather than one large response at the end.

If the SDK streams correctly on your laptop but the browser receives everything at once, inspect every hop between the application and the client. Reverse proxies may buffer responses, serverless platforms may impose execution limits, and your own web framework may consume the iterator before returning it.

Chat Completions and Responses are not interchangeable

A gateway can support /v1/chat/completions and still reject /v1/responses. The request bodies differ too: Chat Completions uses messages, while Responses commonly uses input and has a different event model.

Check the endpoint your client actually uses. A tool that says "OpenAI-compatible" may have migrated to Responses while your gateway only implements Chat Completions. A 404 in that situation is a protocol mismatch, not proof that the whole gateway is down.

Keep the first migration test on Chat Completions unless your application specifically requires Responses. Then add a separate acceptance test for Responses instead of assuming support.

Tool calls need their own acceptance test

Text output says nothing about tool-call compatibility. Test a tiny function with a strict schema:

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL", "https://www.aifast.hk/v1"),
    timeout=60.0,
    max_retries=0,
)

response = client.chat.completions.create(
    model=os.environ["OPENAI_MODEL"],
    messages=[
        {"role": "user", "content": "What is the weather in Tokyo? Use the tool."}
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Return weather for one city",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "city": {"type": "string"}
                    },
                    "required": ["city"],
                    "additionalProperties": False,
                },
            },
        }
    ],
)

message = response.choices[0].message
print(json.dumps(message.model_dump(), indent=2, ensure_ascii=False))

Verify the function name, JSON arguments, finish reason, and behavior while streaming. Some adapters flatten tool calls into text or emit arguments in partial chunks that your parser does not expect.

Preserve the evidence you will need later

During rollout, log:

the final request path, without secrets;
HTTP status and provider error type;
the response model field;
request IDs from response headers or official SDK properties;
latency and retry count;
whether the request streamed;
the selected route or upstream, if the gateway exposes it.

Never log API keys, full authorization headers, or user prompts containing private data.

Request IDs matter when a provider investigates a failure. Read them from the response. Do not invent an ID on the client and present it as the provider's request ID.

A rollout sequence that catches real failures

I use this order:

Call /models with curl.
Send one non-streaming Chat Completions request.
Repeat it with the official SDK and retries disabled.
Test SSE streaming with curl -N.
Test tool calls if the application uses them.
Test Responses only if the client needs that endpoint.
Restore deliberate timeout and retry settings.
Send a small percentage of production traffic to the new route.
Compare status codes, latency, token usage, and output handling.
Keep the old route available until the observation window is clean.

A browser-based gateway check can help with the early protocol checks: inspect an endpoint and read the report. Treat black-box results as compatibility signals, not proof of the underlying model's identity.

I maintain this guide as part of the AIFast team, so the examples use AIFast's documented OpenAI-compatible base URL, https://www.aifast.hk/v1. Treat it as a provider-specific example. Run the same acceptance tests against any gateway you evaluate, including AIFast.

What a successful migration means

A successful migration is not "the first prompt returned text." It means every feature your application depends on has an explicit test, failures are observable, retries are bounded, and rollback is still possible.

The two-line configuration change is real. The rest of the work is proving that those two lines did not hide a protocol mismatch.