Building Chatbots with LLMs and Other AI Models

Modern chatbots are no longer simple retrieval systems wrapped around a prompt. They are long-running, stateful agents that ingest multimodal inputs, maintain conversation history across dozens of turns, and invoke external tools to complete tasks. The cost model of the inference platform underneath these agents directly determines how sophisticated they can be. Token-based billing penalizes long contexts and extended sessions, which forces developers to truncate memory, compress prompts, or limit functionality. Oxlo.ai uses a flat per-request pricing model instead, making it a natural fit for conversational workloads where input length grows with every turn.

Core Architecture

At the center of every chatbot is a loop: receive user input, append it to a conversation history, send the full history to a chat completions endpoint, and render the assistant response. Because Oxlo.ai is fully compatible with the OpenAI SDK, you can adopt this pattern without changing your client code beyond the base URL.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."}
]

def chat(user_input):
    messages.append({"role": "user", "content": user_input})
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages
    )
    assistant_msg = response.choices[0].message.content
    messages.append({"role": "assistant", "content": assistant_msg})
    return assistant_msg

This pattern works for any text-based LLM on Oxlo.ai, including general-purpose models like Llama 3.3 70B, reasoning models like DeepSeek R1 671B MoE, and multilingual agents like Qwen 3 32B.

Memory and Context Management

In production, conversation histories grow. A user might paste a long document, trigger a multi-turn debugging session, or reference earlier parts of the chat after twenty exchanges. Under token-based pricing, every previous message is re-billed on each new request, so costs accelerate as the session deepens.

Oxlo.ai charges one flat cost per API request regardless of prompt length. That means a chatbot can retain full conversation context, include lengthy retrieved documents, or attach few-shot examples without the cost scaling linearly with token count. For agentic workflows that iterate over planning, execution, and reflection steps, this structure removes the penalty on long context.

A simple memory strategy is to keep a rolling buffer of the last N messages and summarize older turns when the buffer fills. Because the underlying cost is per request, you can afford to run a separate summarization call on Oxlo.ai without worrying about the token overhead of the summary itself.

Tool Use and Function Calling

Useful chatbots do more than generate text. They query calendars, run code, or fetch live data. Oxlo.ai supports function calling through the standard OpenAI SDK interface, so you can define tools and let the model decide when to invoke them.

import json

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }
    }
]

def run_tool(name, args):
    if name == "get_weather":
        return {"temperature": 22, "condition": "sunny", "city": args["city"]}

messages = [
    {"role": "user", "content": "What is the weather in Berlin?"}
]

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=messages,
    tools=tools
)

msg = response.choices[0].message

if msg.tool_calls:
    tool_call = msg.tool_calls[0]
    result = run_tool(tool_call.function.name, json.loads(tool_call.function.arguments))
    messages.append({
        "role": "tool",
        "tool_call_id": tool_call.id,
        "name": tool_call.function.name,
        "content": json.dumps(result)
    })
    final = client.chat.completions.create(
        model="qwen-3-32b",
        messages=messages,
        tools=tools
    )
    print(final.choices[0].message.content)

Models such as Qwen 3 32B, GLM 5, and Minimax M2.5 on Oxlo.ai are particularly strong at agentic tool use and multi-step planning.

Multimodal Inputs

Text is only one channel. Users increasingly expect to drop screenshots, diagrams, or voice memos into a chat thread. Oxlo.ai offers vision models including Gemma 3 27B and Kimi VL A3B, as well as audio transcription through Whisper Large v3.

A vision-enabled chatbot can process an image URL or base64 payload using the same chat completions format:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What error does this stack trace show?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/trace.png"}}
        ]
    }
]

response = client.chat.completions.create(
    model="gemma-3-27b",
    messages=messages
)

Because Oxlo.ai bills per request, adding a high-resolution image to the prompt does not spike the inference cost the way a token-based multiplier would. You can also pipe audio through the audio/transcriptions endpoint first, then feed the resulting text into the chat loop.

Structured Outputs

Chatbots often need to emit machine-readable data, such as a JSON object that populates a UI form or triggers a downstream workflow. Oxlo.ai supports JSON mode, allowing you to constrain the model output to valid JSON via the response_format parameter.

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[
        {"role": "system", "content": "Extract the meeting details as JSON."},
        {"role": "user", "content": "Schedule a standup with engineering at 10am tomorrow in Room B."}
    ],
    response_format={"type": "json_object"}
)

data = json.loads(response.choices[0].message.content)

For stricter schemas, you can combine JSON mode with detailed prompts or use the tool-calling pattern to force output that matches a known structure.

Selecting the Right Model

Not every turn in a conversation needs the same capability. Oxlo.ai hosts more than 45 models across seven categories, so you can route queries to the right endpoint.

• General chat and reasoning: Llama 3.3 70B, DeepSeek V4 Flash, GPT-Oss 120B
• Deep reasoning and complex coding: DeepSeek R1 671B MoE, Kimi K2.6, Kimi K2 Thinking
• Agentic tool use: Qwen

Comments

[Marcus Kim]

The Qwen weather example appends the tool result without first adding the assistant message containing its tool_calls to messages; that can break the follow-up request on OpenAI-compatible endpoints. The rolling last-N buffer and separate summary call make sense for a long debugging chat. Flat per-request pricing changes the cost equation, but context limits, latency, and the chance of summarizing away a key decision still make memory quality the harder product problem.