Streaming Responses From Azure OpenAI

#ai #machinelearning #tutorial #llm

Streaming from Azure OpenAI is the OpenAI wire format with one extra participant in it. If your tokens arrive in blocks rather than individually, nothing is broken — a content filter is standing between the model and your socket.

Making the call

Set stream: true. The deployment name goes where the model name would; everything else is the familiar chat completions body.

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://mg-openai-weu.openai.azure.com/",
    api_key="<key>",
    api_version="2024-10-21",
)

stream = client.chat.completions.create(
    model="chat-default",           # deployment name
    messages=[{"role": "user", "content": "Explain a private endpoint."}],
    max_tokens=400,
    stream=True,
)

for chunk in stream:
    if not chunk.choices:
        continue                     # annotation-only frames have no choices
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

The if not chunk.choices guard is not defensive padding. Azure emits frames with an empty choices array — the prompt annotation frame at the start, and content-filter annotation frames throughout — and code that indexes choices[0] unconditionally raises on the very first one.

What comes down the wire

The transport is server-sent events: one data: line per JSON object, blank-line separated, terminated by the literal data: [DONE] sentinel. Azure sends a distinctive first frame carrying prompt_filter_results with an empty choices array, then role and content deltas, then a final content-bearing frame with a finish_reason.

data: {"id":"","object":"","created":0,"model":"","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"}}}],"choices":[],"usage":null}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1692913344,"model":"gpt-4.1","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1692913344,"model":"gpt-4.1","choices":[{"index":0,"finish_reason":null,"delta":{"content":"A"}}],"usage":null}

data: [DONE]

finish_reason is null on every intermediate frame and takes a value on the last one. The values you must handle are stop (the model ended naturally), length (it hit max_tokens), content_filter (the completion was blocked mid-flight) and tool_calls (it wants to call a tool). Treating a length finish as a complete answer is how truncated JSON reaches a parser.

Why your tokens arrive in clumps

This is the behaviour that makes Azure’s stream feel different from OpenAI’s, and it is deliberate. Microsoft’s content streaming documentation describes the default: completion content is buffered, the guardrail system runs over the buffered content, and the content is released only if it passes. The stated consequence is that content is not returned token-by-token but in “content chunks” of the buffer size.

So the perceived time-to-first-token on a default Azure deployment includes a filtering pass that the raw model latency does not. If you are comparing streaming latency between providers and the numbers do not match your expectations, this is usually why — and it is a configuration difference, not a hardware one.

If the completion is blocked mid-stream, you do not get an HTTP error: the connection has already returned 200. You get a final frame with "finish_reason": "content_filter" and the categories that fired. Microsoft also documents that you are billed for the prompt and for the completion tokens generated before filtering, so a filtered stream costs money and produces nothing.

Asynchronous Filter and its offsets

The alternative is an Asynchronous Filter configuration, enabled in the Streaming section of a content filter configuration in the Foundry portal and applied at deployment level. Filters then run alongside the stream instead of in front of it: no buffering, token-by-token delivery, and a filtering signal that arrives late.

Microsoft is specific about how late. The signal is guaranteed within a roughly 1,000-character window of the offending content, which means harmful text can reach the user before it is flagged. The documented expectation is that your application implements redaction on receipt of a delayed signal. This is a real safety trade, not a performance toggle.

Annotation frames in this mode carry a content_filter_offsets object with three fields, all counted in characters from 0 at the beginning of the prompt:

check_offset — how much text has been fully moderated. It never decreases, and it is an exclusive lower bound on future end_offset values.
start_offset and end_offset — the range this particular annotation applies to. Multiple annotations can refer to the same text.

Tracking check_offset is what lets a UI show which part of a streamed answer has been cleared and which is still provisional. Microsoft notes that Asynchronous Filter requires API version 2024-02-01 or later, and that content retroactively flagged as protected material may not be eligible for Customer Copyright Commitment coverage — which is a legal consequence of a latency setting, and the reason this is a decision rather than a default.

Getting token counts from a stream

A streamed response has no usage object by default; every frame above shows "usage": null. Request it explicitly:

stream = client.chat.completions.create(
    model="chat-default",
    messages=[{"role": "user", "content": "Explain a private endpoint."}],
    stream=True,
    stream_options={"include_usage": True},
)

With that set, a final frame arrives after the last content frame, carrying a populated usage object and an empty choices array — another reason the guard in the loop matters. Note that stream_options is not accepted on every code path: it is a standard chat-completions parameter and is rejected as an extra input when combined with the data_sources extension used by Azure OpenAI On Your Data, which is a common surprise when a retrieval-augmented endpoint is added to a working streaming client.

Without it, the only way to attribute cost to a streamed request is to count tokens client-side, which will not match the service’s count. Turn it on before you need the number rather than after.

Tool calls, truncation and proxies

Three things break streaming clients that handled plain text perfectly, and none of them is Azure-specific in cause even though all three show up here first.

Tool call arguments arrive in fragments. When the model calls a tool, the deltas carry a tool_calls array rather than content, and the JSON argument string is split across many chunks with no guarantee about where the splits land — mid-key, mid-value, mid-escape. Each entry carries an index, and the correct handling is to accumulate the argument fragments per index and parse only once finish_reason arrives as tool_calls. Attempting to parse an accumulated buffer on every chunk works right up until an argument is long enough to be split, which is usually the first real tool rather than the test one.

A stream cannot become an error. The status line and headers are sent before the first token, so once you have a 200 there is no way for the service to escalate. A connection dropped mid-stream therefore looks to a naive client exactly like a completed short answer. The only reliable completion signals are the data: [DONE] sentinel and a non-null finish_reason; treat a stream that ends without either as failed, not as finished. This is also why the content_filter finish reason exists as a value rather than as a status code.

Something in the middle will buffer it. Reverse proxies, ingress controllers and serverless platforms frequently accumulate a response body before forwarding it, which converts a stream back into a single slow reply and produces the report that “streaming works locally but not in production”. The fixes are per-hop: disable response buffering on the proxy, send Cache-Control: no-cache and X-Accel-Buffering: no on your own response, and check that the hosting platform supports streamed responses at all before designing around them. Azure’s default filter buffering, described above, makes this harder to diagnose: chunky delivery has two possible causes and you have to rule out the one you control.