Production Streaming for OpenAI-Compatible LLM APIs: SSE, Timeouts, and Retries

Streaming is not just a user-interface trick. It changes how a service manages connections, cancellation, retries, and observability. A chat UI that waits for a complete response can appear broken during a long generation; a UI that streams every token can leave half-open connections and duplicate work if the client retries carelessly.

This guide shows a small, production-oriented streaming client using Python and an OpenAI-compatible endpoint. The examples use the model identifiers currently visible in AIWave's public pricing catalog on 2026-08-03. Rates are included only as a dated reference:

Model	Input / output per 1M tokens	Good fit
deepseek-v4-flash	$0.206 / $0.412	Interactive chat and extraction
qwen3-coder-480b-a35b-instruct	$0.12 / $0.36	Code explanation and generation

Check the AIWave pricing page before budgeting. The endpoint is OpenAI-compatible, so an existing OpenAI SDK integration can keep its message format while using https://aiwave.live/v1.

What SSE actually guarantees

Server-Sent Events (SSE) is a long-lived HTTP response whose body contains events separated by blank lines. The browser EventSource API is one consumer, but Python services can consume the same wire format with an HTTP client. An event normally looks like data: {json}\n\n; a final data: [DONE] marker tells the client that the model has finished.

SSE is one-way. The client cannot send a new message over the same stream. To cancel generation, close the HTTP connection and make cancellation visible in application telemetry. The MDN SSE reference describes the framing rules; your provider's API documentation defines the JSON fields inside each event.

A minimal streaming client

The OpenAI Python SDK handles the event parsing for Chat Completions. Keep the key as a placeholder in examples and load a real key through your deployment secret manager.

from openai import OpenAI

client = OpenAI(
    base_url="https://aiwave.live/v1",
    api_key="YOUR_API_KEY_HERE",  # Create a key at https://aiwave.live/
    timeout=60.0,
)

stream = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[
        {"role": "system", "content": "Answer in short, testable steps."},
        {"role": "user", "content": "Explain why database indexes speed up reads."},
    ],
    stream=True,
    max_tokens=500,
)

for chunk in stream:
    text = chunk.choices[0].delta.content or ""
    print(text, end="", flush=True)
print()

The first token is a useful latency metric, but it is not the same as total request latency. Record both time_to_first_token and time_to_last_token. A model can start quickly and still take a long time to finish a large response.

Forwarding a stream from FastAPI

If your application exposes its own endpoint, do not buffer the provider response in memory. Yield each chunk as it arrives and preserve the SSE content type. The example below uses httpx and emits a small JSON envelope to the browser.

import json
import httpx
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()


async def events(prompt: str):
    payload = {
        "model": "qwen3-coder-480b-a35b-instruct",
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "max_tokens": 800,
    }
    headers = {"Authorization": "Bearer YOUR_API_KEY_HERE"}
    async with httpx.AsyncClient(timeout=httpx.Timeout(60.0, connect=10.0)) as client:
        async with client.stream(
            "POST", "https://aiwave.live/v1/chat/completions", json=payload, headers=headers
        ) as response:
            response.raise_for_status()
            async for line in response.aiter_lines():
                if not line.startswith("data: "):
                    continue
                data = line[6:]
                if data == "[DONE]":
                    yield "data: [DONE]\n\n"
                    return
                event = json.loads(data)
                delta = event.get("choices", [{}])[0].get("delta", {})
                text = delta.get("content") or ""
                if text:
                    yield "data: " + json.dumps({"text": text}) + "\n\n"


@app.get("/chat")
async def chat(prompt: str):
    return StreamingResponse(events(prompt), media_type="text/event-stream")

In production, add a disconnect check so work stops when the browser leaves. FastAPI's request object can be passed into the generator and polled between chunks. Also set proxy buffering off for the route; otherwise Nginx or a CDN may hold several events and defeat streaming.

Timeouts and cancellation

Use separate connection and read timeouts. A connection timeout protects you from a dead upstream; a read timeout protects you from a stream that stops producing tokens. The right value depends on your workload. Interactive chat might use 10 seconds to connect and 60 seconds to read, while a code-generation job may need several minutes.

Cancellation must be idempotent. Closing the client stream is safe, but retrying immediately can create two generations for one user action. Give every request an internal ID and record whether the client disconnected, the provider ended normally, or your timeout terminated the stream.

Retry only before output begins

Retries are safest before the first token. Once output has reached the user, a retry can duplicate text or cause a side effect twice if the model is calling tools. A simple policy is:

1. Retry connection failures and 429/5xx responses before the first event.
2. Use exponential backoff with jitter and a small maximum attempt count.
3. Never blindly retry a stream after partial output; return the partial result with a retryable status to the caller.

For tool-calling agents, persist tool-call IDs and make downstream actions idempotent. Streaming improves perceived latency, but it does not remove the need for distributed-systems discipline.

Measure the economics of streaming

Streaming changes perceived latency, not token pricing. At the dated rates above, a request with 3,000 input tokens and 600 output tokens costs about $0.000865 on DeepSeek V4 Flash and $0.000576 on Qwen3 Coder. The formula is (input_tokens × input_rate + output_tokens × output_rate) / 1,000,000; refresh rates from AIWave pricing before a finance report.

Track these fields per request: model, first-token milliseconds, total milliseconds, input tokens, output tokens, completion status, disconnect reason, retry count, and estimated USD cost. Keep prompt text out of logs unless it has been explicitly scrubbed. A request ID is enough to join application logs with provider metrics.

Checklist

• Use the exact model ID from the AIWave model catalog.
• Set separate connect and read timeouts.
• Disable buffering on streaming proxy routes.
• Stop upstream work when the client disconnects.
• Retry only before the first token, with a bounded backoff.
• Treat tool calls and other side effects as idempotent.
• Keep YOUR_API_KEY_HERE in public examples and rotate real keys through a secret manager.

The Chat Completions documentation covers request parameters and streaming behaviour. Start with a small evaluation set, measure first-token and completion latency separately, and then tune timeouts using observed production distributions rather than a single benchmark.