DEV Community

shashank ms
shashank ms

Posted on

Streaming with LLM

#aiinfrastructure #oxlo #ai

Streaming is the default expectation for production LLM applications. Users want to see text appear token by token rather than waiting for an entire response to buffer. For developers, implementing streaming correctly means balancing low latency with robust error handling, all while keeping costs predictable. Oxlo.ai supports streaming across its full catalog of chat and reasoning models, and because its pricing is request-based rather than token-based, you can stream long responses without watching the meter run on every single chunk.

What streaming actually does under the hood

When you set stream=True on a chat completion request, the server does not wait to assemble the full response before sending it. Instead, it opens an HTTP connection using Server-Sent Events (SSE) and pushes partial completion objects as they are generated. Each chunk contains a delta, which is a fragment of the assistant's message content, along with metadata such as finish reasons and logprobs if requested.

The client receives these deltas in order and appends them to the UI. From the user's perspective, this creates the illusion that the model is typing in real time. Under the surface, you are simply iterating over an HTTP response body that uses a text/event-stream content type.

Implementing streaming with the OpenAI SDK

Because Oxlo.ai is fully OpenAI SDK compatible, you can enable streaming by changing exactly one parameter in your existing client code. Point your SDK to the Oxlo.ai base URL and set stream=True.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

# Replace MODEL_ID with your chosen Oxlo.ai model,
# such as Llama 3.3 70B, Qwen 3 32B, or DeepSeek V4 Flash
response = client.chat.completions.create(
    model="MODEL_ID",
    messages=[{"role": "user", "content": "Write a Python function for merge sort"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
Enter fullscreen mode Exit fullscreen mode

The iterator yields objects as they arrive. You should check that delta.content exists before rendering it, because the first and last chunks in the stream may contain role assignments or finish reason metadata rather than text.

Why request-based pricing changes the streaming equation

On token-based providers, every chunk that arrives represents additional tokens billed to your account. Streaming a long reasoning trace from a model like DeepSeek R1 671B MoE or Kimi K2.6 can become expensive quickly because you pay for every token in the chain of thought plus the final output.

Oxlo.ai uses flat per-request pricing. One API call costs the same amount regardless of whether the model returns fifty tokens or five thousand. This means streaming is not a luxury feature that inflates your bill. It is a standard UX pattern you can enable for every user interaction without cost anxiety. For agentic workloads that produce long tool-usage traces or for code generation that emits lengthy outputs, this predictability is significant. You can see the exact structure on the Oxlo.ai pricing page.

Handling errors and aborts in production

Streaming connections are long-lived HTTP requests, so they are vulnerable to network hiccups and client-side cancellations. You should wrap your stream iterator in a try/except block that catches APIError and APIConnectionError from the SDK. If a user clicks "stop generating" in your UI, use an abort controller or close the iterator to free the connection on both sides.

Because Oxlo.ai serves popular models with no cold starts, the time-to-first-byte is consistently low. You do not need to build retry logic around waking up idle containers. The connection opens immediately, and the first chunk typically arrives within milliseconds of the request being accepted.

Choosing models for low-latency streaming

Not every model streams at the same speed. Large mixture-of-experts models like GLM 5 or DeepSeek R1 671B MoE produce excellent reasoning, but they may emit tokens more deliberately than smaller generalist models. For interactive use cases where latency matters, Oxlo.ai offers several fast options:

  • DeepSeek V4 Flash: Efficient MoE architecture with a 1M context window and near state-of-the-art open-source reasoning.
  • Qwen 3 32B: Strong multilingual output and agent workflow support at a manageable parameter count.
  • Oxlo.ai Coder Fast: Built specifically for code generation tasks where rapid streaming improves the editing experience.

You can switch between these models by changing the model parameter in the same SDK call. There is no change in integration logic required.

Conclusion

Streaming is not an advanced optimization. It is a baseline requirement for modern LLM applications. Implementing it through Oxlo.ai requires no custom client libraries, no changes to your prompt logic, and no penalty for long outputs. If you are currently using a token-based provider and paying extra for every character that appears on screen, moving to Oxlo.ai's request-based model can simplify both your architecture and your forecasting. Point your OpenAI client to https://api.oxlo.ai/v1, set stream=True, and ship a faster UX today.

Top comments (0)