DEV Community

chen qin
chen qin

Posted on

Stop Waiting for the Full AI Response: Stream Tokens in Python

Most AI applications wait for the model to generate the complete answer before showing anything to the user.

For short answers, that may be acceptable. For longer responses, it can make the application feel slow—even when the model is already generating tokens.

Streaming solves this by displaying each part of the response as soon as it arrives.

The non-streaming version

A standard OpenAI-compatible request may look like this:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_API_KEY"],
    base_url=os.environ["AI_BASE_URL"],
)

response = client.chat.completions.create(
    model=os.environ["AI_MODEL"],
    messages=[
        {
            "role": "user",
            "content": "Explain API gateways in three sentences.",
        }
    ],
)

print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

This works, but nothing is printed until the complete response has arrived.

Stream the response

Enable streaming by adding stream=True:

stream = client.chat.completions.create(
    model=os.environ["AI_MODEL"],
    messages=[
        {
            "role": "user",
            "content": "Explain API gateways in three sentences.",
        }
    ],
    stream=True,
)
Enter fullscreen mode Exit fullscreen mode

The request now returns a sequence of chunks instead of one completed response.

Loop through those chunks and print the available content:

for chunk in stream:
    content = chunk.choices[0].delta.content

    if content:
        print(content, end="", flush=True)

print()
Enter fullscreen mode Exit fullscreen mode

The user can now see the answer while it is being generated.

Complete example

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_API_KEY"],
    base_url=os.environ["AI_BASE_URL"],
)

stream = client.chat.completions.create(
    model=os.environ["AI_MODEL"],
    messages=[
        {
            "role": "user",
            "content": "Explain API gateways in three sentences.",
        }
    ],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content

    if content:
        print(content, end="", flush=True)

print()
Enter fullscreen mode Exit fullscreen mode

Keeping the API key, base URL, and model name in environment variables also makes it easier to change providers without rewriting the application logic.

When streaming is useful

Streaming is especially helpful for:

  • AI chat interfaces
  • Coding assistants
  • Long-form generation
  • Command-line tools
  • Applications where perceived latency matters

Remember that model capabilities and streaming formats can vary between providers. Verify support for your selected model and handle empty chunks, connection failures, and interrupted streams before using this pattern in production.

I tested this pattern with an OpenAI-compatible endpoint through APIHubRelay.

What should the next example cover: streaming in Node.js, error handling, or automatic retries?

Top comments (0)