An LLM request that repeatedly stalls for roughly three minutes is a useful debugging clue, not a diagnosis. A client deadline, gateway timeout, upstream delay, buffering, or retries can all contribute. Neither a 178–194 second duration nor a message such as TRANSPORT: Connection error identifies the failing component on its own.
Check the actual transport first
Streaming HTTP chat completions typically deliver server-sent events (SSE). WebSocket support and automatic transport selection depend on the particular API, client, and version. There is no universal OpenAI-compatible SDK setting named transport: sse.
Keys such as api: openai-completions, streamIdleTimeoutMs, and retryPolicy belong to particular applications or wrappers; copying them into an unrelated SDK may do nothing or raise a configuration error. Use your client's documented options and inspect its request logs. If a wrapper really does negotiate WebSockets first, test its documented HTTP/SSE option rather than treating that behavior as common to every SDK.
The official Python SDK documents streaming, timeout controls, and retry behavior separately in its README. Check the documentation for the version you installed, including whether it already retries requests, before adding another retry layer.
Separate the timeout types
A connection timeout limits establishment of a connection. A read timeout limits a wait for incoming data; it is not necessarily a deadline for the whole response. A stream that keeps sending bytes can outlive a read timeout. A correctly implemented read timeout can also fire before the first body chunk, even if headers have arrived.
HTTPX distinguishes connect, read, write, and pool timeouts in its timeout documentation. Choose limits for the expected workload: a shorter timeout reduces waiting but may also interrupt a valid long-running generation. A fixed 90-to-180-second ratio is not generally appropriate.
Check buffering at each proxy
A proxy can delay forwarding upstream chunks. For an nginx HTTP proxy location serving SSE, this is a starting point, not a complete server configuration:
# Inside the existing location that proxies to the model backend:
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 90s; # example: maximum gap between upstream reads
proxy_read_timeout is measured between successive reads, not across the entire response. nginx can also honor an upstream X-Accel-Buffering: no response header unless configured to ignore it. Other proxies, CDNs, compression layers, or the application itself may still buffer. Changing nginx alone does not guarantee token-by-token delivery. See the nginx proxy module documentation.
Retry by error type, with a total deadline
String matching an exception for "502" or "timeout" can misclassify failures. Inspect transport exceptions and HTTP status codes. Repeating an unchanged malformed request or invalid credential is generally not useful; some authentication failures require refreshing credentials, not blindly resending them.
The following Python 3.11+ example uses HTTPX directly, not SDK-specific configuration. It demonstrates a non-streaming request so that the entire body is consumed inside the deadline. Configure BASE, KEY, and MODEL in your environment; BASE is the API base including /v1 when required.
import asyncio
import os
import random
import httpx
RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}
async def ask_with_retry(client, *, attempts=3, budget_s=60):
if attempts < 1 or budget_s <= 0:
raise ValueError("attempts and budget_s must be positive")
# Includes attempts, response reads, and backoff.
async with asyncio.timeout(budget_s):
for attempt in range(attempts):
try:
response = await client.post(
os.environ["BASE"].rstrip("/") + "/chat/completions",
headers={"Authorization": f"Bearer {os.environ['KEY']}"},
json={
"model": os.environ["MODEL"],
"messages": [{"role": "user", "content": "hi"}],
"stream": False,
},
)
response.raise_for_status()
return response.json()
except httpx.HTTPStatusError as exc:
if exc.response.status_code not in RETRYABLE_STATUS:
raise
if attempt == attempts - 1:
raise
except (httpx.TimeoutException, httpx.NetworkError):
if attempt == attempts - 1:
raise
await asyncio.sleep(min(0.8 * 2 ** attempt, 6) + random.uniform(0, 0.3))
async def main():
async with httpx.AsyncClient(
timeout=httpx.Timeout(30, connect=5, read=20)
) as client:
print(await ask_with_retry(client))
if __name__ == "__main__":
asyncio.run(main())
The status list and delays are example policy, not instructions to retry every server error. Production handling should respect a valid Retry-After within the remaining budget. A timed-out POST may already have reached the upstream, so repeating it can duplicate work or charges. Do not blindly replay a stream after delivering partial output or executing tool side effects.
For streaming, keep both the request and the full stream-consumption loop inside the deadline and close the stream on cancellation. Async timeouts use cooperative cancellation, not a process-level kill switch; blocking code or suppressed cancellation can delay cleanup. See Python's timeout documentation.
Measure before changing configuration
This shell example writes response headers and body to local files so an HTTP error is not mistaken for a successful generation. Set the three environment variables first and inspect the captured body without sharing credentials or sensitive content.
curl -sS -N --connect-timeout 10 --max-time 60 \
-D response.headers -o response.body \
-w 'http=%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n' \
"$BASE/chat/completions" \
-H "Authorization: Bearer $KEY" \
-H 'Content-Type: application/json' \
-d "$(python -c 'import json,os; print(json.dumps({"model":os.environ["MODEL"],"messages":[{"role":"user","content":"hi"}],"stream":True}))')"
time_connect includes time from the start until connection establishment; DNS and TLS have their own timing fields. time_starttransfer measures the first response byte, often a header, not the first generated token. These are cumulative timings, not independent phase durations. -N disables curl's output buffering, not a proxy's. See the curl manual.
A fast test only shows that one request completed; it does not prove the client is at fault. Record HTTP status, headers, body/error events, curl's exit status, and arrival times of actual SSE events. Compare direct and proxied requests where available. A model-list endpoint can help, but presence in a list alone does not establish that the model is callable with your credentials.
The checklist
- Identify the API, client version, and transport actually in use.
- Set documented connect/read limits and an end-to-end deadline.
- Inspect buffering across the entire response path.
- Avoid stacked retries, classify failures, and account for duplicate work.
- Correlate client measurements with proxy and upstream logs before assigning a cause.
Related guide: Your OpenAI-compatible endpoint hangs for 180 seconds.
If you want the whole set: I packaged all 30 of these recipes — pacing, JSON repair, failover ladders, evals, cost accounting — into one markdown bundle. It's $19, paid in USDT, download unlocks the moment the transfer confirms, no account needed.
Top comments (0)