Building Real-Time Chatbots with LLMs

Real-time chatbots do not send complete replies in a single block. They stream tokens as they are generated, maintain conversational context across turns, and often invoke tools before responding. For developers, this means architecting around three constraints: low latency, persistent session state, and unpredictable output length. This guide walks through a production-ready pattern for building streaming chatbots, and shows how to integrate Oxlo.ai as the inference backend using its OpenAI-compatible API.

Streaming Architecture

Most real-time interfaces rely on server-sent events (SSE) rather than WebSockets. SSE runs over standard HTTP, reconnects automatically, and works through most corporate proxies. Your backend opens an SSE channel to the client, then forwards tokens from the LLM as they arrive.

Oxlo.ai supports streaming responses on all chat models. When you set stream=True in your request, the API returns chunks in the same delta format as OpenAI. Because Oxlo.ai has no cold starts on popular models, the time-to-first-token remains consistent even after periods of low traffic.

SDK Setup and First Streamed Request

Because Oxlo.ai is fully OpenAI SDK compatible, you can reuse existing Python or Node.js clients. Point the base_url to https://api.oxlo.ai/v1 and select a model that matches your latency and reasoning requirements.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Explain recursion in Python"}],
    stream=True,
    max_tokens=512
)

for chunk in response:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

For multilingual or agentic workflows, swap the model identifier to qwen3-32b. If you need deep reasoning before the bot replies, deepseek-r1-671b or kimi-k2.6 provide advanced chain-of-thought capabilities.

Managing Context and Multi-Turn State

A real-time bot must remember prior turns. The simplest approach is to keep an array of messages in memory or in a fast key-value store such as Redis. Append each user message and assistant response, then pass the full array to the next request.

With token-based providers, long histories inflate costs linearly. Oxlo.ai uses request-based pricing, so the cost per turn stays flat regardless of how many tokens are in the prompt. This makes Oxlo.ai significantly cheaper for long-context and agentic workloads where conversation history grows with every turn. Models such as deepseek-v4-flash support a 1M context window, and kimi-k2.6 handles 131K tokens, both available under the same flat request model.

Function Calling for Interactive Workflows

Modern chatbots rarely just chat. They fetch data, update records, or trigger actions. Oxlo.ai supports function calling and tool use across its LLM lineup, so you can define JSON schemas and let the model decide when to invoke them.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"}
            },
            "required": ["city"]
        }
    }
}]

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=messages,
    tools=tools,
    stream=False  # set True if you want to stream tool-call reasoning
)

After receiving the tool call, execute the function locally, append the result to the message array, and send a follow-up request to generate the final natural-language response.

Latency Optimization

Perceived real-timeness depends on time-to-first-token (TTFT) and inter-token latency. You can optimize both without rewriting your application:

• Model selection: Use lightweight models such as qwen3-32b or oxlo.ai-coder-fast for high-frequency, low-latency turns. Reserve heavy reasoning models for explicit deep-analysis commands.
• Prompt hygiene: Remove redundant system text and shorten tool descriptions to the essentials.
• No cold starts: Oxlo.ai keeps popular models warm, so you do not pay a startup penalty on the first request after idle time.

Cost Predictability at Scale

Token-based billing introduces variance. A user who pastes a thousand lines of logs into a support chat can suddenly multiply the cost of that turn. For products with tight margins, this unpredictability complicates forecasting.

Oxlo.ai charges one flat cost per API request regardless of prompt length. For chatbots that accumulate context, include image inputs, or process long documents, flat request pricing removes the penalty for long inputs. It can be 10-100x cheaper than token-based alternatives for long-context workloads. See the exact tiers on the Oxlo.ai pricing page.

Production Checklist

• Streaming fallbacks: If a stream is interrupted, resume from the last complete chunk or restart the turn with a truncated history.
• JSON mode: When the bot must return structured data, use response_format={"type": "json_object"} to avoid parsing fragile markdown.
• Rate-limit awareness: Oxlo.ai Free plans include 60 requests per day, Pro includes 1,000 per day, and Premium includes 5,000 per day with priority queue access. Enterprise plans offer unlimited requests and dedicated GPUs.
• Vision inputs: If your chatbot accepts screenshots, use gemma-3-27b-it or kimi-vl-a3b through the same chat completions endpoint.

Conclusion

Building a real-time chatbot comes down to three implementation details: streaming tokens efficiently, managing state across turns, and controlling costs as usage scales. Oxlo.ai covers all three with OpenAI SDK-compatible streaming, flat per-request pricing that rewards long conversations, and a broad catalog of models ready without cold starts. If you are evaluating inference backends for your next conversational agent, drop Oxlo.ai into your existing client by changing the base_url and compare the results.