Building a production-ready chatbot requires more than wrapping a prompt in an HTTP call. You need state management, streaming responses, tool use, and an inference backend that stays predictable under load. In this guide, you will build a conversational agent in Python that handles multi-turn dialogue, optional function calling, and real-time streaming. We will use Oxlo.ai as the inference provider because its OpenAI-compatible API and request-based pricing remove the usual friction of scaling long-context conversations.
What You Will Build
You will create a command-line chatbot that:
- Maintains conversation history across multiple turns
- Streams tokens to the terminal for low-latency feedback
- Optionally delegates tasks to external tools via function calling
- Switches between general-purpose and reasoning models without code changes
Prerequisites
- Python 3.10 or newer
- An Oxlo.ai API key (sign up at https://oxlo.ai/pricing)
- The OpenAI Python SDK (
pip install openai)
Choosing a Model
Oxlo.ai hosts 45+ models across seven categories. For a general chatbot, Llama 3.3 70B works well as a flagship option. If you need multilingual reasoning or agent workflows, Qwen 3 32B is a strong alternative. For deep reasoning or complex coding tasks, DeepSeek R1 671B MoE or Kimi K2.6 (with 131K context and advanced reasoning) fit naturally. Because Oxlo.ai charges one flat cost per request regardless of prompt length, you can send long system prompts or extended conversation histories without the cost scaling typical of token-based providers.
Connecting to the API
Oxlo.ai is fully OpenAI SDK compatible. Change the base URL and API key, and your existing code runs unchanged.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)This drop-in replacement means you can migrate from OpenAI, Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale without rewriting your client logic.
Building the Chat Loop
A robust chatbot stores messages in a list and appends the assistant's reply after each turn. Below is a minimal streaming implementation.
def chat_loop():
messages = [
{"role": "system", "content": "You are a helpful assistant. Be concise."}
]
while True:
user_input = input("User: ")
if user_input.lower() in {"exit", "quit"}:
break
messages.append({"role": "user", "content": user_input})
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
stream=True
)
print("Assistant: ", end="", flush=True)
assistant_content = ""
for chunk in response:
if chunk.choices[0].delta.content:
token = chunk.choices[0].delta.content
print(token, end="", flush=True)
assistant_content += token
print()
messages.append({"role": "assistant", "content": assistant_content})
if __name__ == "__main__":
chat_loop()Streaming responses reduce perceived latency, and Oxlo.ai serves popular models with no cold starts.
Adding Tool Use
Real chatbots need to interact with the outside world. Oxlo.ai supports function calling and tool use through the same OpenAI schema. The following example gives the bot a calculator.
import json
def calculate(expression: str) -> str:
try:
result = eval(expression)
return str(result)
except Exception as e:
return f"Error: {e}"
tools = [
{
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a mathematical expression.",
"parameters": {
"type": "object",
"properties": {
"expression": {"type": "string"}
},
"required": ["expression"]
}
}
}
]
def chat_with_tools():
messages = [{"role": "system", "content": "You can use a calculator when needed."}]
while True:
user_input = input("User: ")
if user_input.lower() in {"exit", "quit"}:
break
messages.append({"role": "user", "content": user_input})
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
tools=tools,
tool_choice="auto"
)
message = response.choices[0].message
messages.append(message)
if message.tool_calls:
for tool_call in message.tool_calls:
if tool_call.function.name == "calculate":
args = json.loads(tool_call.function.arguments)
result = calculate(args["expression"])
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": tool_call.function.name,
"content": result
})
second_response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages
)
print("Assistant:", second_response.choices[0].message.content)
Top comments (0)