"Streaming through an OpenAI-compatible proxy: 6 bugs that only show up in production"

I run a small OpenAI-compatible gateway. Non-streaming requests worked on day one. Getting streaming right took me about three weeks of bug reports from my own tools.

The annoying part: every bug below passes unit tests. They only show up with a real client, a real network, and a model that thinks for 30 seconds before saying anything.

Here's what broke, and the code that fixed it.

1. chunk["choices"][0] throws IndexError

If you pass stream_options: {"include_usage": true}, OpenAI sends a final chunk that carries token counts and an empty choices array:

{"choices":[],"usage":{"prompt_tokens":412,"completion_tokens":88,"total_tokens":500}}

Any code that assumes choices[0] exists dies right there, at the very end of the stream, after the user already read the full answer. Worst possible moment to 500.

async for chunk in upstream:
    if not chunk.get("choices"):
        if chunk.get("usage"):
            record_usage(user, chunk["usage"])
        continue
    delta = chunk["choices"][0].get("delta", {})

Second half of this bug: not every upstream supports include_usage. SiliconFlow honors it, some others accept the field and silently drop it. If billing depends on that final chunk, you need a fallback. I run tiktoken on the accumulated text when no usage chunk shows up, and mark the record as estimated.

2. TCP doesn't care about your event boundaries

An SSE event is data: {...}\n\n. A single read() gives you half an event, or three events glued together. If you do line.split("data: ")[1], sooner or later you hand truncated JSON to json.loads and the whole stream dies on one packet boundary.

Buffer until you see the blank line:

buf = ""
async for raw in upstream.content.iter_any():
    buf += raw.decode("utf-8", "ignore")
    while "\n\n" in buf:
        event, buf = buf.split("\n\n", 1)
        for line in event.splitlines():
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                return
            yield json.loads(payload)

That buf variable is the entire trick. Two details: some upstreams send [DONE] without a trailing newline, so flush the leftover buffer on exit; and decode on a raw bytes chunk can split a multi-byte UTF-8 character in half. Use an incremental decoder or accept "ignore" and lose the occasional character in a Chinese or emoji response. I lost a few characters before switching to codecs.getincrementaldecoder("utf-8")().

3. tool_calls arrive in pieces and you merge by index, not id

This one cost me a weekend. When the model calls a function, arguments stream in as fragments:

{"delta":{"tool_calls":[{"index":0,"id":"call_a1","type":"function","function":{"name":"get_weather","arguments":""}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"ci"}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ty\":\"Xi"}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"'an\"}"}}]}}

id and name appear only in the first fragment. Later fragments carry index and nothing else. Key your accumulator on id and you end up with one tool call whose id is None and a JSON string you can't parse.

Key on index:

calls = {}
for tc in delta.get("tool_calls", []):
    i = tc.get("index", 0)
    slot = calls.setdefault(i, {"id": "", "name": "", "arguments": ""})
    if tc.get("id"):
        slot["id"] = tc["id"]
    fn = tc.get("function") or {}
    if fn.get("name"):
        slot["name"] = fn["name"]
    slot["arguments"] += fn.get("arguments") or ""

Then json.loads(slot["arguments"]) once the stream ends. And note that parallel tool calls use index 0, 1, 2. If you keep a single slot you silently drop the second and third call. Coding agents hit this path constantly, which is why "the model said it edited the file but nothing changed" is usually a proxy bug, not a model bug.

4. You cannot retry after you've sent the headers

My failover logic originally retried on any error. But once you've written 200 OK and streamed 40 tokens to the client, there is no retry. The status line is already gone. You can drop the connection or emit a fake error event, and both look like a crash to the user.

So I added a first-token gate: don't commit the response until the first byte from upstream arrives.

async def open_upstream(body):
    for ep in (PRIMARY, FALLBACK):
        try:
            r = await client.post(ep, json=body, timeout=8)
            first = await asyncio.wait_for(r.content.__anext__(), timeout=8)
            return r, first          # now it's safe to send 200
        except Exception as e:
            log.warning("upstream %s died before first byte: %s", ep, e)
    raise HTTPException(502, "all upstreams down")

Before this, mid-stream failures ran around 1 in 300 requests. After, failures moved to the window before any user-visible output, which the OpenAI SDK retries on its own. Broken answers went to effectively zero.

One caveat. Reasoning models can sit quiet for 20 to 30 seconds while they think, so a flat 8-second gate kills them. I use 8s for models I know start fast and 30s for the reasoning ones. Measure yours, don't copy mine.

5. Idle connections get killed by whatever sits in front of you

Long think time with zero bytes on the wire means someone hangs up. Nginx defaults to a 60-second read timeout. Cloudflare is stricter. The user sees a truncated answer and blames your service.

Fix: send an SSE comment line every 15 seconds. It's a spec-valid no-op and the OpenAI SDK skips it.

HEARTBEAT = b": ping\n\n"

Race the upstream read against a 15s timer and write the comment when the timer wins. Also set X-Accel-Buffering: no on your own response. Without that header nginx buffers your carefully streamed chunks into one blob, and the user waits for the entire answer anyway. One header, and the stream suddenly feels fast.

6. When the client leaves, cancel the upstream

The user closes the tab halfway through. Starlette gives you await request.is_disconnected(), but while you're iterating the upstream body you never check it. The upstream keeps generating, you keep paying, and it holds a connection slot.

async for chunk in upstream_stream:
    if await request.is_disconnected():
        await upstream_stream.close()
        break
    yield chunk

I found a few hundred of these zombie reads in one day of logs, after a frontend bug spammed reconnects every second. Cheap to fix, real money at scale.

The 30-second way to see all of it yourself

Skip the SDK and look at raw bytes:

curl -N https://keheai.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-ai/DeepSeek-V3","stream":true,
       "stream_options":{"include_usage":true},
       "messages":[{"role":"user","content":"count from 1 to 5"}]}'

-N disables curl's own buffering. You'll see the chunk boundaries as they land, plus that empty-choices usage chunk at the end. Twenty seconds of this teaches more than a day of reading SDK source.

---

Every bug on this list lives in the gap between "OpenAI-compatible" and actually compatible. The spec is loose, the SDKs are forgiving, and failures only appear when someone's workflow depends on the stream finishing.

I built this gateway into keheai.com. It's the same ~200 lines of Python, now serving a few small teams, free tier is 330k tokens if you want to poke at it.