Building Chatbots with LLM and Low Latency

#product #oxlo #ai

Latency is the silent killer of conversational AI. Users expect chatbot responses to feel instantaneous, but large language models introduce unavoidable delays during prefill, decoding, and network transit. When time-to-first-token stretches past a few hundred milliseconds, engagement drops and trust erodes. The good news is that low-latency chatbots are achievable without sacrificing model quality, provided you optimize the full stack from model selection to client rendering.

Why Latency Breaks Chatbots

Human perception research suggests that feedback loops start to feel sluggish when a system takes longer than roughly 100 milliseconds to respond. Large language models violate this threshold by design. A chatbot must first transmit a potentially long prompt over the network, wait for the model to process every input token during prefill, and then decode the response one token at a time. Longer prompts, which are typical in retrieval-augmented generation and multi-turn conversations, directly extend prefill time. If your architecture also incurs cold starts or queues requests behind other users, the delay compounds quickly.

Measure What Matters

Before optimizing, instrument three distinct intervals. First, time-to-first-token, or TTFT, measures the delay from request submission to the arrival of the first response chunk. Second, inter-token latency, or ITL, captures the gap between successive output tokens and determines how smoothly the response feels when streaming. Third, end-to-end latency tracks the full round trip from the user pressing enter to the final rendered character. Optimizing only TTFT while ignoring ITL produces jittery streams, and optimizing both while ignoring network overhead leaves room for improvement at the edge.

Architectural Patterns for Sub-Second Response

Several engineering choices move the needle. Prompt compression and careful context pruning reduce prefill work. Quantization and efficient attention implementations lower memory bandwidth pressure. Speculative decoding can accelerate generation by drafting future tokens with a smaller auxiliary model. Perhaps most importantly, model architecture matters. Mixture-of-Experts designs such as DeepSeek V4 Flash on Oxlo.ai activate only a subset of parameters per token, delivering near state-of-the-art reasoning with higher throughput and a 1 million token context window. These efficiencies translate directly into lower TTFT for long inputs.

Streaming and Incremental Rendering

Streaming is non-negotiable for modern chatbots. Rather than buffering the entire completion, your application should parse Server-Sent Events as they arrive and append tokens to the user interface immediately. This improves perceived latency even when total generation time is unchanged. Oxlo.ai supports streaming responses across its chat completions endpoint, so you can pipe tokens directly into a React component, terminal buffer, or voice synthesizer without waiting for a final JSON payload.

Smarter Routing and Model Selection

Not every user query requires your largest model. A productive pattern is to classify intent with a fast generalist model and escalate complex reasoning, coding, or long-context tasks to a specialized counterpart. Oxlo.ai hosts more than 45 models across seven categories, which lets you build a routing layer without integrating multiple providers. For example, you might default to Qwen 3 32B for multilingual agent workflows, promote coding questions to DeepSeek V3.2 or Qwen 3 Coder 30B, and reserve Llama 3.3 70B or DeepSeek R1 671B MoE for deep reasoning. Because Oxlo.ai charges one flat cost per request regardless of prompt length, you can include full conversation history, large codebases, or retrieved documents in every call without the cost scaling typical of token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale.

Caching Context and Conversation State

Repeated work is wasted work. Cache embedding lookups, retrieved documents, and static system prompts in Redis or a similar store. For conversational memory, store prior turns server-side and inject only the recent history required for coherence. Even with aggressive context windows, caching reduces redundant prefill. Oxlo.ai eliminates cold starts on popular models, so once your cache warms up, subsequent requests hit an already active inference path. This consistency is critical for chatbots that must maintain responsive turn-taking across extended sessions.

Code: A Low-Latency Chatbot with Oxlo.ai

The following Python example uses the OpenAI SDK, which is fully compatible with Oxlo.ai. It streams a multi-turn conversation and measures TTFT. Notice that the request includes a system prompt and a detailed user question, illustrating how long context fits naturally within a single flat-priced request.

import os
import time
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def chat_stream(messages, model="qwen3-32b"):
    """Stream a chat completion and measure TTFT."""
    t0 = time.time()
    stream = client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
        max_tokens=1024
    )

    ttft = None
    for chunk in stream:
        delta = chunk.choices[0].delta
        if delta.content:
            if ttft is None:
                ttft = time.time() - t0
                print(f"\n[TTFT: {ttft*1000:.0f} ms]\n")
            print(delta.content, end="", flush=True)
    print()

# Multi-turn conversation with long context
history = [
    {"role": "system", "content": "You are a concise coding assistant."},
    {"role": "user", "content": "How do I implement a connection pool in Python using asyncio?"}
]

# With Oxlo.ai's request-based pricing, long system prompts and large
# message histories do not inflate cost per character.
chat_stream(history, model="deepseek-v4-flash")

Optimizing the Last Mile

Client-side and network optimizations matter as much as model inference. Reuse TCP and TLS connections across requests rather than opening a new socket for every turn. Ensure your HTTP client supports HTTP/2 to multiplex parallel requests if you are prefetching or routing. If you deploy edge functions, place them close to your users to minimize propagation delay. Oxlo.ai serves inference from a centralized API at https://api.oxlo.ai/v1, so pairing it with a globally distributed edge worker keeps the wire latency between your user, your business logic, and the model as short as possible.

Putting It Together

Low-latency chatbots are a system-level achievement, not a single configuration toggle. You need precise measurement, streaming architecture, intelligent model routing, aggressive caching, and client-side efficiency. Oxlo.ai supports this stack natively with no cold starts, full OpenAI SDK compatibility, and a catalog of more than 45 models ranging from efficient MoE architectures to high-capability reasoning engines. Because Oxlo.ai uses request-based pricing, long-context workloads and agentic loops that would be prohibitively expensive on token-based platforms become practical and predictable. For details on flat per-request pricing, see https://oxlo.ai/pricing.

DEV Community