Conversational AI is moving from simple Q&A bots to stateful, multi-turn agents that reason across long sessions. Building these systems requires more than calling a chat endpoint. You need to manage context windows, enforce output schemas, integrate external tools, and control latency. This guide walks through the practical engineering decisions involved, with concrete code you can run today.
Architecture and Model Selection
Start by matching the model to the conversation pattern. For open-ended dialogue and reasoning, Llama 3.3 70B and Qwen 3 32B handle multi-turn coherence well. If your application requires deep reasoning or complex coding within the conversation flow, DeepSeek R1 671B MoE or Kimi K2.6 provide advanced chain-of-thought capabilities. For vision-enabled chat, Kimi VL A3B and Gemma 3 27B process image inputs alongside text.
Oxlo.ai hosts 45+ models across these categories with fully OpenAI SDK compatibility, so you can prototype against one model and swap to another without rewriting client code. Because Oxlo.ai charges per request rather than per token, long system prompts and extended multi-turn histories do not inflate costs. This makes it practical to keep richer context in scope for agentic workloads.
Setting Up the SDK
Oxlo.ai is a drop-in replacement for the OpenAI client. Point your base URL to https://api.oxlo.ai/v1 and use your Oxlo.ai API key.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="your-oxlo.ai-api-key"
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a concise technical assistant."},
{"role": "user", "content": "Explain how to manage conversation state."}
]
)
print(response.choices[0].message.content)
No cold starts on popular models means the first request after idle time returns immediately, which matters for synchronous chat interfaces.
Designing System Prompts
The system prompt is your contract with the model. Define the persona, constraints, and output rules in a single message. Keep it explicit. Instead of "be helpful," use "Answer in fewer than 100 words. Ask clarifying questions if the request is ambiguous."
For multilingual agents, Qwen 3 32B responds reliably to system prompts in non-English languages. If you are building a coding assistant, DeepSeek V3.2 or Oxlo.ai Coder Fast interpret instructions about code style and context boundaries with high fidelity.
Test system prompts by holding user messages constant and measuring variance across temperature values. A temperature of 0.1 to 0.3 works best for deterministic conversational agents, while 0.7 to 0.9 suits creative brainstorming modes.
Managing Context and Memory
LLMs do not remember state between API calls. Your application must maintain the message list and truncate or compress it when approaching the model's context limit. A simple sliding window drops the oldest user-assistant pairs, but this loses early context.
A better approach is summarization. When the message count exceeds a threshold, send the existing history to a lightweight model with the instruction: "Summarize the following conversation into a single paragraph, preserving all facts, decisions, and open tasks." Replace the oldest messages with that summary.
With Oxlo.ai, request-based pricing removes the penalty for sending long context windows. You can afford to keep more turns in scope before compressing, which improves coherence in long-horizon agentic tasks.
Adding Tool Use and Function Calling
Conversational AI becomes useful when it can act. Function calling lets the model decide when to invoke external tools. Define your schema in the tools parameter, and handle the tool_calls response in your application logic.
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[{"role": "user", "content": "What is the weather in Berlin?"}],
tools=tools,
tool_choice="auto"
)
message = response.choices[0].message
if message.tool_calls:
for call in message.tool_calls:
print(f"Call {call.function.name} with {call.function.arguments}")
# Execute your local function, then append the result to messages
Models like Kimi K2.6, GLM 5, and Minimax M2.5 are specifically strong at agentic tool use. Oxlo.ai exposes function calling across its chat endpoints, so you can build retrieval-augmented generation pipelines or trigger actions directly from conversation turns.
Enforcing Structured Output with JSON Mode
Chatbots often need to emit structured data, such as appointment slots or form fields, while still conversing naturally. JSON mode forces the model to return valid JSON.
response = client.chat.completions.create(
model="deepseek-v4-flash",
messages=[
{"role": "system", "content": "You are a helpful assistant. Respond in JSON."},
{"role": "user", "content": "Book a table for two at 7 PM under the name Chen."}
],
response_format={"type": "json_object"}
)
data = response.choices[0].message.content
When using JSON mode, always include an explicit JSON instruction in the system or user message. DeepSeek V4 Flash, with its 1M context window, can extract structured entities from very long documents in a single request without truncation.
Streaming Responses for Real-Time Interaction
Perceived latency is critical in conversation. Streaming lets you render tokens as they arrive rather than waiting for the full completion.
stream = client.chat.completions.create(
model="qwen3-32b",
messages=[{"role": "user", "content": "Explain quantum computing simply."}],
stream=True
)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
print(token, end="", flush=True)
Oxlo.ai supports streaming across its chat models. Combine this with server-sent events in your backend to push tokens to a web frontend as they are generated.
Evaluating Conversational Quality
Build an evaluation pipeline early. Track three metrics: relevance (does it answer the question?), groundedness (is it faithful to provided context?), and coherence (does it follow the conversation thread?).
Use a judge model to score outputs. For example, send the conversation history and the candidate response to Llama 3.3 70B with a rubric. Store results and compare across prompt versions. If you see drift in long sessions, your context compression strategy is likely failing.
Cost Considerations at Scale
Token-based billing scales with input length, so long system prompts and agentic loops become expensive quickly. If your conversational AI maintains a detailed persona or iterates through tool calls, per-token costs accumulate on every turn.
Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For long-context and agentic workloads, this can be significantly cheaper than token-based alternatives. You can send full documentation in the system prompt or retain long conversation histories without re-evaluating your budget on every call. See the exact tiers on the Oxlo.ai pricing page.
Putting It All Together
Building conversational AI is an exercise in context management. Choose a model that matches your reasoning requirements, design a rigid system prompt, maintain state carefully, and layer in tools and structured output where needed. Stream responses to keep the interaction feeling alive, and evaluate continuously.
Oxlo.ai provides the infrastructure to do this without rewriting your stack. With OpenAI SDK compatibility, 45+ models, and request-based pricing that favors long-context workloads, it is a strong option for production conversational agents. Point your client to https://api.oxlo.ai/v1, pick a model, and start building.
Top comments (0)