DEV Community

Cover image for Your OpenAI streaming loop works in the demo. It breaks on the last chunk.
Versyka
Versyka

Posted on AI-assisted

Your OpenAI streaming loop works in the demo. It breaks on the last chunk.

#programming #ai #openai #python

If you moved from the OpenAI Python SDK 0.28 to 1.x, your non-streaming code probably broke loudly: openai.ChatCompletion.create became client.chat.completions.create, and every response["choices"][0]["message"]["content"] became response.choices[0].message.content. You fixed those in an afternoon.

Streaming code is different. It usually doesn't break at all. It just gets slightly wrong, on the very last chunk, after the user has already read the answer.

Trap 1: content is None

In 0.28, chunks were dicts, and most people wrote:

text += chunk["choices"][0]["delta"].get("content", "")
Enter fullscreen mode Exit fullscreen mode

A missing key gave you an empty string. In 1.x, chunks are typed objects, and the straight translation is:

text += chunk.choices[0].delta.content   # TypeError on the final chunk
Enter fullscreen mode Exit fullscreen mode

delta.content is None on the final chunk (the one that carries finish_reason) and on chunks that carry tool calls instead of text. If you build the text with an f-string or str(), you don't get an error. You get the word "None" at the end of the user's answer.

Trap 2: the chunk with no choices

Turn on usage reporting:

stream = client.chat.completions.create(
    model=MODEL, messages=messages, stream=True,
    stream_options={"include_usage": True},
)
Enter fullscreen mode Exit fullscreen mode

and the last chunk carries usage with an empty choices list. chunk.choices[0] raises IndexError on the final chunk, after the user has already seen a complete answer. In logs it looks like a random failure at the end of a request.

The guard 

text = ""
for chunk in stream:
    if not chunk.choices:          # usage-only final chunk
        continue
    text += chunk.choices[0].delta.content or ""
Enter fullscreen mode Exit fullscreen mode

or "" covers the None, and continue skips the usage chunk.

Why your tests didn't catch it

Most test suites mock the LLM with one hand-written chunk. Both traps live on the first or last chunk of a real stream, so a mock never reaches them. The only test I trust for streaming reads a real stream (or a recorded one) to the end and checks the joined text.

Finding every place it hides

Streaming loops tend to get copy-pasted: a chat endpoint, a background summarizer, a CLI script. Search for .delta.content and choices[0] across the whole repository, not only the file you remember. That search, plus rewriting what it finds and running your tests before and after, is what Versyka automates when an SDK ships a breaking change.

What's the strangest bug a streaming response has given you? Tell me in the comments.

Top comments (0)