Building a Chatbot using LLM

#product #oxlo #ai

Building a reliable chatbot requires more than wrapping a prompt around a text endpoint. You need state management, context handling, and an inference backend that stays predictable under load. This guide walks through a minimal but production-ready pattern using Python and the OpenAI SDK, with Oxlo.ai as the inference layer.

Architecture Overview

A production chatbot has three concerns: the client interface, conversation memory, and the inference backend. The client collects user input and renders responses. Memory stores message history so the model retains context across turns. The inference backend executes the model and returns completions. Oxlo.ai provides a fully OpenAI-compatible chat/completions endpoint at https://api.oxlo.ai/v1, which means you can keep your existing SDK code and simply change the base URL and API key.

Project Setup

Install the official OpenAI SDK and set your Oxlo.ai API key. If you do not have a key yet, the Oxlo.ai free tier includes 60 requests per day and a 7-day full-access trial across more than 16 models.

pip install openai
export OXLO_API_KEY="your_oxlo_api_key"

Minimal Chat Loop

The following example implements a multi-turn chat loop against Oxlo.ai using Llama 3.3 70B. Because Oxlo.ai supports streaming responses, you can flush tokens to the terminal as they arrive.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

messages = [{"role": "system", "content": "You are a helpful assistant."}]

while True:
    user_input = input("User: ")
    if user_input.lower() in {"exit", "quit"}:
        break

    messages.append({"role": "user", "content": user_input})

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
        stream=True
    )

    print("Assistant: ", end="", flush=True)
    assistant_content = ""
    for chunk in response:
        if chunk.choices[0].delta.content:
            token = chunk.choices[0].delta.content
            print(token, end="", flush=True)
            assistant_content += token
    print()

    messages.append({"role": "assistant", "content": assistant_content})

This loop appends each assistant response back into the messages list, giving the model full conversational context on the next turn.

Managing Conversation Memory

As conversations grow, so does the payload. Token-based providers scale cost linearly with input length, which makes long sessions expensive. Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For chatbots that accumulate messages or ingest long documents, this can be significantly cheaper than token-based alternatives such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale.

Even with flat pricing, you should still manage memory to stay within model context limits. A simple sliding window keeps the last N message pairs and drops the oldest turns.

MAX_HISTORY = 20

def trim_messages(messages):
    # Keep system prompt plus last MAX_HISTORY turns
    if len(messages) > MAX_HISTORY + 1:
        return [messages[0]] + messages[-MAX_HISTORY:]
    return messages

Pass trim_messages(messages) into the API call before each request to avoid silent truncation.

Adding Tool Use

Modern chatbots rarely stop at text generation. Oxlo.ai supports function calling and tool use across many models, including Qwen 3 32B and Kimi K2.6. The snippet below registers a mock weather tool and lets the model decide when to invoke it.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

if response.choices[0].message.tool_calls:
    # Execute tool logic and append results to messages
    ...

Because Oxlo.ai charges per request rather than per token, multi-turn tool workflows that shuttle large JSON schemas back and forth do not inflate your bill the way token-based pricing would.

Selecting a Model

Oxlo.ai hosts more than 45 models across seven categories. For chatbots, these are strong starting points:

General purpose: Llama 3.3 70B
Multilingual reasoning and agents: Qwen 3 32B
Deep reasoning and complex coding: DeepSeek R1 671B MoE
Advanced agentic coding and vision: Kimi K2.6 (131K context)
Efficient long-context reasoning: DeepSeek V4 Flash (1M context)
Coding and reasoning, free tier: DeepSeek V3.2

All models share the same endpoint and SDK compatibility, so you can switch models by changing a single string.

Why Oxlo.ai for Chatbot Workloads

Chatbots are uniquely sensitive to cost structure. A long running session or an agentic loop that appends tool results can quickly accumulate thousands of tokens per turn. Oxlo.ai flattens that curve with per-request pricing, and it removes cold starts on popular models so your users are not waiting for containers to warm up.

The platform is a drop-in replacement for any OpenAI SDK implementation. Change base_url to https://api.oxlo.ai/v1, bring your existing message handling logic, and you are live. For exact plan details, see the Oxlo.ai pricing page.

DEV Community