DEV Community

Alex
Alex

Posted on

Streaming chat completions from an OpenAI-compatible API: the SSE lines, the usage chunk, tool calls and the usual bugs

Disclosure: I run UberLLM; the examples use its API, but the setup works the same with any compatible provider where noted.

Streaming is the right default for anything with a long prompt, long output or a high token cap: the first words show up quickly, and long answers do not hit request timeouts. It is also where most client bugs live. This post walks through what actually comes over the wire from an OpenAI-compatible /v1/chat/completions endpoint, and the mistakes that show up again and again.

Turn it on

Add "stream": true:

curl -N https://api.uberllm.dev/v1/chat/completions \
  -H "Authorization: Bearer $UBERLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek/deepseek-v4-flash",
    "stream": true,
    "messages": [{"role": "user", "content": "Count to five, one per line."}]
  }'
Enter fullscreen mode Exit fullscreen mode

-N turns off curl's buffering so you see events as they arrive. Any OpenAI-compatible provider takes the same body; change the URL, key and model id.

What arrives

The reply is Server-Sent Events: lines starting with data:, each a chat completion chunk, separated by blank lines, and a last line data: [DONE]. Schematically:

: ping

data: {"choices":[{"delta":{"content":"1"}}], …}

data: {"choices":[{"delta":{"content":"\n2"}}], …}

… more chunks …

data: {… "usage": {"prompt_tokens": …, "completion_tokens": …, "cost_usd": …}}

data: [DONE]
Enter fullscreen mode Exit fullscreen mode

Three kinds of line, and a client has to handle all of them:

  1. Comment lines starting with :. UberLLM sends a : ping comment the moment you connect and keep-alives after that, so a long stream survives proxy idle timeouts. They are not data: skip them.
  2. data: lines with JSON. The text is in choices[0].delta.content.
  3. data: [DONE]. The end. It is not JSON, so do not parse it.

Parsing it by hand

If you are not using an SDK, the loop is short. With Python's requests:

import json, os, requests

resp = requests.post(
    "https://api.uberllm.dev/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['UBERLLM_API_KEY']}"},
    json={
        "model": "deepseek/deepseek-v4-flash",
        "stream": True,
        "messages": [{"role": "user", "content": "Write a short poem."}],
    },
    stream=True,
)
resp.raise_for_status()  # refusals come back as a normal HTTP error before the stream

done, usage = False, None
for raw in resp.iter_lines():               # bytes; decode as UTF-8 yourself
    line = raw.decode("utf-8")
    if not line or line.startswith(":"):    # blank separator or keep-alive comment
        continue
    if not line.startswith("data:"):
        continue
    payload = line[len("data:"):].strip()
    if payload == "[DONE]":
        done = True
        break
    chunk = json.loads(payload)
    if chunk.get("usage"):
        usage = chunk["usage"]
    for choice in chunk.get("choices") or []:  # a chunk may carry no text
        text = (choice.get("delta") or {}).get("content")
        if text:
            print(text, end="", flush=True)

if not done:
    raise RuntimeError("stream ended without [DONE]: treat the answer as incomplete")
print("\n", usage)
Enter fullscreen mode Exit fullscreen mode

Things this gets right that hand-rolled parsers often miss: comments are skipped, a chunk with no text (or no choices) does not crash the loop, [DONE] is never passed to json.loads, and a stream that closes without [DONE] is treated as cut off, not finished.

With the OpenAI SDK you get the parsing for free:

from openai import OpenAI

client = OpenAI(base_url="https://api.uberllm.dev/v1", api_key="ull_...")

with client.chat.completions.create(
    model="deepseek/deepseek-v4-flash",
    stream=True,
    messages=[{"role": "user", "content": "Write a short poem."}],
) as stream:
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
Enter fullscreen mode Exit fullscreen mode

Note the trailing /v1 in base_url; without it every call is a 404.

The usage chunk

Token counts in a stream come at the end, in a chunk with a usage object. In the OpenAI format you normally have to ask for it with stream_options: {"include_usage": true}. On other compatible providers, send that yourself where it is supported; in the Vercel AI SDK's OpenAI-compatible provider it is includeUsage: true.

UberLLM adds stream_options: {"include_usage": true} for you when the upstream supports it, and when an upstream sends no usage chunk it adds one before data: [DONE]. The usage carries the prompt and completion token counts and cost_usd, what that request is billed in US dollars.

The usual bug: a loop that only looks at delta.content never sees the usage chunk, so token logging silently records nothing. Read usage from every chunk, as above.

On UberLLM a non-streamed reply also has an X-UberLLM-Cost-USD header. A stream does not, because its headers go out before the cost is known: read usage.cost_usd from the last chunk.

Reasoning models

Reasoning text comes in the delta too. UberLLM sends it in both reasoning and reasoning_content, with the same text, so clients that read either one get it. The bug to avoid: printing both doubles every word. Pick one. A reply with no reasoning has neither field.

Tool calls

Pass tools as usual; streaming does not change the request. Things to get right:

  • Use a model that takes tools. The model list (GET https://api.uberllm.dev/v1/models) marks supports_tools on each chat model. On UberLLM, tools sent to a model that cannot call them are refused with 400 tools_not_supported before anything is charged.
  • Wait for the end before running the tool. Tool calls arrive in the delta in pieces. Do not parse the arguments as JSON until the stream says the call is complete. SDK helpers and frameworks (LangChain, the Vercel AI SDK) assemble the pieces for you.
  • Check finish_reason. A reply that calls a tool ends with finish_reason: "tool_calls". Some providers have ended tool calls with "stop" instead, and clients that only act on "tool_calls" never ran the tool. UberLLM normalises this so it is always "tool_calls", streamed or not; on other providers, it is worth checking for tool calls whatever the finish reason says.
  • Send tool calls back exactly as you got them. Some carry an extra_content field (Gemini's thought signature) that the model needs on the next turn. Rebuilding the message from only the fields you know drops it.

Errors, aborts and retries

  • Before the stream starts, a refusal is a normal HTTP status with an OpenAI-style error body ({"error": {"code": …, "message": …}}): 402 insufficient_credits, 404 model_not_found, 429 rate_limited, 503 no_available_provider. Check the status before you start reading lines.
  • Once streaming has started, nothing is retried. UberLLM retries a call that fails before the first content byte (up to 4 attempts), but a failure mid-stream ends the stream, and you are billed only for what was produced. Your client should treat a missing [DONE] as a failure and decide whether to retry.
  • Disconnecting is safe. If you close the connection mid-stream, the gateway cancels the upstream request and settles for the tokens actually streamed.
  • Time to first byte. A stream gets 10 seconds to start (more for a long prompt or a usually slow model); a reply without "stream": true gets 85 seconds, because nothing arrives until it is complete. Another reason to stream long answers.

The other two surfaces

  • Anthropic format (/v1/messages with "stream": true): the same idea in Anthropic's event names. max_tokens is required, and on UberLLM the cost is in the usage of the message_delta event.
  • Responses API (/v1/responses): events have a type such as response.output_text.delta, and the stream ends with response.completed (or response.failed, never completed, if it stopped early). The same : ping comment lines appear while a slow model starts.

Checklist

  • Skip lines starting with :.
  • Never json.loads the [DONE] line; treat a stream without it as incomplete.
  • Guard for chunks with no choices or no text.
  • Read usage from the last chunk, and ask for it where your provider needs include_usage.
  • Print reasoning or reasoning_content, not both.
  • Assemble tool calls before parsing them, and send them back unchanged.

Sources

Top comments (0)