Building Chatbots and Virtual Assistants with LLM: A Step-by-Step Guide

Building a production-grade chatbot or virtual assistant requires more than prompting a large language model. You need to manage conversation state, enforce structured outputs, integrate external tools, and control latency and cost as session length grows. This guide walks through a practical, code-first approach to assembling these components, from selecting a backbone model to deploying an agent that can hold context, call APIs, and respond in real time.

Architecture Overview

A reliable virtual assistant typically splits into four layers. The input layer handles user messages, image uploads, and voice transcripts. The context layer maintains multi-turn history and injects retrieved documents or system instructions. The reasoning layer is the LLM itself, which may also emit tool calls. The output layer streams text back to the user, renders structured JSON to a frontend, or triggers actions in your backend. Keeping these boundaries explicit makes it easier to swap models, add modalities, and optimize cost without rewriting core logic.

Choosing a Backbone Model

Model selection should map to your assistant's primary task. General-purpose support bots need strong instruction following and multilingual fluency. Coding agents need deep reasoning. High-volume prototypes need efficient context windows.

Oxlo.ai hosts 45+ open-source and proprietary models across seven categories, all exposed through a single OpenAI-compatible endpoint. Relevant options include:

• General-purpose chat: Llama 3.3 70B and Qwen 3 32B for broad instruction following.
• Advanced reasoning and coding: DeepSeek R1 671B MoE, Kimi K2.6, and DeepSeek V4 Flash for complex multi-step tasks.
• Long-horizon agents: GLM 5 (744B MoE) and Minimax M2.5 for tool-heavy workflows.
• Cost-efficient prototyping: DeepSeek V3.2, which is available on the free tier.

Because Oxlo.ai charges a flat rate per request rather than per token, long system prompts and extended multi-turn histories do not inflate your bill. This is especially useful when you inject large retrieval contexts or maintain lengthy agent conversations.

Project Setup and Authentication

Oxlo.ai is fully OpenAI SDK compatible. You can use the Python or Node.js client by swapping the base URL and API key. No other client-side changes are required.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Managing Multi-Turn Conversations

A virtual assistant must remember context. Instead of passing raw strings, maintain a message list and append each turn before the next request.

messages = [
    {"role": "system", "content": "You are a helpful assistant for Acme Corp. Be concise."},
    {"role": "user", "content": "What are your support hours?"},
    {"role": "assistant", "content": "Our support team is available 24/7."},
    {"role": "user", "content": "How do I reset my password?"}
]

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=messages
)

# Append the reply before the next user turn
messages.append({
    "role": "assistant",
    "content": response.choices[0].message.content
})

For production, trim or summarize history when you approach the model's context limit. If you are running retrieval-augmented generation, place retrieved documents after the system prompt and before the latest user message.

Adding Tool Use and Function Calling

Real assistants need to query databases, check calendars, or call APIs. Function calling lets the model emit structured tool requests rather than guessing answers.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": "Retrieve the status of a customer order",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"}
                },
                "required": ["order_id"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="qwen3-32b",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

Parse response.choices[0].message.tool_calls, execute the function in your backend, and return the result in a message with role: "tool". Models like Qwen 3 32B, GLM 5, and Minimax M2.5 on Oxlo.ai are specifically strong at agentic tool use.

Enforcing Structured Output with JSON Mode

Chatbots often need to emit machine-readable payloads for frontend rendering or downstream processing. JSON mode guarantees valid JSON from the model.

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[
        {"role": "system", "content": "You are a booking assistant. Respond in JSON with keys: action, restaurant, party_size, time."},
        {"role": "user", "content": "Book a table for 2 at 7pm under the name Chen."}
    ],
    response_format={"type": "json_object"}
)

data = response.choices[0].message.content

Always describe the desired JSON schema in the system prompt so the model knows which keys to produce. Validate the parsed result against your schema server-side before acting on it.

Streaming Responses for Real-Time UX

Waiting for a full completion before displaying text creates a sluggish experience. Streaming lets you render tokens as they arrive.

stream = client.chat.completions.create(
    model="kimi-k2.6",
    messages=messages,
    stream=True
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="")

Oxlo.ai supports streaming on all chat models with no cold starts, so popular assistants backed by Llama 3.3 70B or Kimi K2.6 begin responding immediately.

Handling Vision Inputs

If your assistant must interpret screenshots or user-uploaded images, use a vision-capable model through the same chat/completions endpoint.

response = client.chat.completions.create(
    model="gemma-3-27b-it",
    messages=[
        {"role": "user", "content": [
            {"type": "text", "text": "What is wrong with this error message?"},
            {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
        ]}
    ]
)

Oxlo.ai offers vision models including Gemma 3 27B and Kimi VL A3B, so you can handle image understanding without adopting a separate API.

Cost Engineering for Production Chatbots

As assistants grow more agentic, prompts expand. Retrieval-augmented generation injects pages of documentation, and multi-turn sessions accumulate history. On token-based providers, this linearly increases cost.

Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For long-context and agentic workloads, this can be significantly cheaper than token-based alternatives because your bill does not scale with input tokens. You can send a full 131K context through Kimi K2.6 or a 1M context through DeepSeek V4 Flash and pay the same flat request rate. See https://oxlo.ai/pricing for current plan details.

Deployment Checklist

Before going live, verify the following:

• Conversation history is trimmed or compressed to stay within context limits.
• Function schemas are strict and validated server-side.
• JSON mode is used for any output parsed by code.
• Streaming is enabled for user-facing interfaces.
• Rate limits and retry logic are configured on your side.
• Fallback models are selected in case of overload.

Oxlo.ai simplifies this stack by offering 45+ models, from lightweight coding assistants to 744B parameter MoE agents, behind one OpenAI-compatible API. Whether you are prototyping a support bot or scaling a multi-agent system, the flat per-request pricing removes the penalty for long prompts, letting you focus on behavior rather than token budgets.