Chatbots built on large language models have moved from demos to production infrastructure. Whether you are automating support, powering internal research assistants, or orchestrating multi-step agentic workflows, the core challenge is the same: reliably turning user intent into contextually aware responses without letting latency or cost scale out of control. This guide walks through the architectural decisions, code patterns, and infrastructure choices that separate prototype chatbots from production systems.
Architecture of a Production Chatbot
A production chatbot is not just a prompt sent to an API. It is a pipeline that ingests user input, retrieves relevant context, manages conversation state, and generates a response under latency constraints. Most systems separate concerns into four layers:
- Interface layer: Handles user input normalization, rate limiting, and session tracking.
- Memory layer: Stores conversation history and external context. This can be as simple as an in-memory list or as complex as a vector database for RAG.
- Orchestration layer: Decides when to call the LLM, when to execute tools, and how to handle errors.
- Inference layer: The LLM backend that executes generation. This is where infrastructure costs accumulate.
For the inference layer, Oxlo.ai provides a fully OpenAI-compatible API with request-based pricing. Instead of metering tokens, you pay a flat rate per request. This makes cost predictable for long-context sessions and agentic loops where input length grows with every turn.
Selecting a Model for Your Use Case
Not every chatbot needs the largest model. Match capability to task:
- General conversation and support: Llama 3.3 70B or Qwen 3 32B offer strong multilingual reasoning and fast response times.
- Deep reasoning and complex coding: DeepSeek R1 671B MoE or Kimi K2.6 excel at chain-of-thought tasks and agentic coding.
- Long-document analysis: DeepSeek V4 Flash supports a 1 million token context window, and Kimi K2.6 handles 131K contexts, letting you pass entire manuals or codebases in a single request.
- Cost-sensitive prototyping: DeepSeek V3.2 is available on the Oxlo.ai free tier, letting you validate logic before scaling.
Because Oxlo.ai hosts over 45 models across seven categories, you can route different user intents to different models without managing multiple provider accounts.
Building the Core Conversation Loop
The simplest chatbot is a stateful loop that appends user messages to a history array and streams the model output back. Because Oxlo.ai is fully OpenAI SDK compatible, you can switch endpoints by changing two lines of code.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
conversation = [
{"role": "system", "content": "You are a helpful assistant that answers questions about infrastructure."}
]
def chat(user_input):
conversation.append({"role": "user", "content": user_input})
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=conversation,
stream=True
)
reply = ""
for chunk in response:
if chunk.choices[0].delta.content:
reply += chunk.choices[0].delta.content
print(chunk.choices[0].delta.content, end="")
conversation.append({"role": "assistant", "content": reply})
return reply
chat("What is request-based pricing?")
Streaming is supported on all chat models, so you can flush tokens to the user interface as they arrive rather than waiting for the full generation.
Memory and Context Management
As conversations grow, so does the context window. Token-based providers charge for every token in the prompt, which means long sessions become exponentially more expensive. Oxlo.ai charges per request, so adding history does not increase the inference cost on its own.
For sessions that exceed the model context limit, implement a sliding window or summarization strategy:
def trim_history(messages, max_turns=10):
# Keep system prompt and last N turns
if len(messages) <= max_turns * 2 + 1:
return messages
return [messages[0]] + messages[-(max_turns * 2):]
conversation = trim_history(conversation)
If your chatbot needs to reference external documents, use the embeddings endpoint to build a retrieval pipeline. Oxlo.ai offers BGE-Large and E5-Large for vector search, and the resulting chunks can be injected into the system prompt without worrying about per-token cost escalation.
Tool Use and Function Calling
Modern chatbots rarely operate in isolation. They query APIs, run code, or search databases. Oxlo.ai supports function calling across its LLM lineup, letting you define tools as JSON schemas and letting the model decide when to invoke them.
tools = [
{
"type": "function",
"function": {
"name": "get_uptime",
"description": "Get server uptime in seconds",
"parameters": {
"type": "object",
"properties": {
"host": {"type": "string"}
},
"required": ["host"]
}
}
}
]
response = client.chat.completions.create(
model="qwen-3-32b",
messages=conversation,
tools=tools,
tool_choice="auto"
)
if response.choices[0].message.tool_calls:
# Execute tool and append result
tool_call = response.choices[0].message.tool_calls[0]
result = execute_tool(tool_call)
conversation.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result)
})
Agentic workflows that chain multiple tool calls benefit from request-based pricing. Each reasoning step is one request, and the cost remains flat even when tool outputs inject large JSON payloads back into the context.
Cost Engineering for Production
The dominant cost driver for most chatbots is not the model size, but the input length. Support transcripts, code repositories, and multi-turn agent logs can quickly inflate token counts. Token-based providers meter both input and output, so a long context workload that sends 50K tokens and returns 500 tokens can cost nearly as much as the reverse.
Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For long-context and agentic workloads, this can be significantly cheaper than token-based alternatives because your cost scales with the number of interactions, not the size of the conversation history.
Oxlo.ai also eliminates cold starts on popular models, so you do not pay latency penalties for scaling from zero. You can start on the free tier with 60 requests per day and 16+ models, then move to Pro or Premium as traffic grows. See https://oxlo.ai/pricing for current plan details.
Deployment Patterns
Deploy your chatbot as a stateless service. Keep conversation history in Redis or a database, pass the relevant slice to the LLM on each request, and rely on Oxlo.ai for inference. This decouples your application logic from the model lifecycle.
Key production checks:
- Retries with backoff: Wrap the SDK client to handle transient network errors.
- Timeout handling: Set client-side timeouts slightly above your worst-case latency target.
- Guardrails: Validate tool inputs and filter outputs before returning them to users.
- Model fallback: If one model is heavily loaded, route to another Oxlo.ai model with similar capabilities, such as falling back from Llama 3.3 70B to Qwen 3 32B.
Conclusion
Building a chatbot with LLMs and machine learning is fundamentally an exercise in systems design. The model is only one component. Memory management, tool integration, and cost controls determine whether the project survives past the prototype stage.
Oxlo.ai fits naturally into this stack. Its OpenAI-compatible API means zero refactoring to migrate existing chatbot code, its request-based pricing protects budgets as context grows, and its broad model catalog lets you optimize for quality, latency, or cost without managing multiple providers. If you are shipping a chatbot this quarter, start with the free tier and measure how flat per-request pricing changes your cost curve.
Top comments (0)