Building conversational AI that retains context across dozens of turns, executes external tools, and returns structured data requires more than prompting a large language model. It demands careful orchestration of memory, state management, and inference infrastructure. In this article, we walk through the architectural patterns that make multi-turn dialogue reliable, and we show how to implement them using standard SDKs against modern inference platforms.
Architecture of a Conversational LLM System
A production conversational stack typically has four layers: the interface, the orchestrator, the inference backend, and the memory store. The orchestrator handles turn history, injects system prompts, and routes tool results back to the model. The inference backend must support high context limits, streaming, and native function calling.
Oxlo.ai provides these primitives across its model catalog. For general dialogue, Llama 3.3 70B and Qwen 3 32B offer strong multilingual reasoning. When you need extended context, Kimi K2.6 supports 131K tokens and DeepSeek V4 Flash supports 1M tokens, letting you keep more history in memory before truncating or summarizing.
Context Management and Memory
Long conversations exhaust context windows quickly. Most teams adopt one or more mitigation strategies:
- Sliding window truncation: Keep the last N turns and discard the rest.
- Summarization: Periodically compress old turns into a condensed memory block.
- External retrieval: Store conversation facts in a vector database and inject only relevant passages.
These techniques increase the number of tokens sent per request. On token-based providers, this directly raises cost. Oxlo.ai uses request-based pricing, so the cost per turn stays flat regardless of prompt length. That predictability makes it practical to send fuller context or use retrieval-augmented generation without surprise billing. See the Oxlo.ai pricing page for plan details.
Function Calling and Tool Use
Conversational agents rarely stop at text generation. They need to query calendars, update databases, or call internal APIs. This requires the model to emit structured tool calls, the orchestrator to execute them, and the results to be re-inserted into the chat history as new messages with the tool role.
Oxlo.ai supports function calling and tool use across its chat models, and the API is fully compatible with the OpenAI SDK. You define tools with JSON Schema, request the model, parse the tool_calls field, run your local functions, and return the outputs. The loop is identical to the standard OpenAI pattern, so existing agent frameworks usually require only a base URL change.
Deploying with Oxlo.ai
Oxlo.ai exposes a single base URL, https://api.oxlo.ai/v1, that acts as a drop-in replacement for the standard OpenAI client. The platform hosts 45+ models across seven categories, including chat, vision, code, audio, and embeddings, with no cold starts on popular workloads.
For conversational AI, you can route simple queries to efficient models like DeepSeek V3.2 and complex reasoning steps to DeepSeek R1 671B MoE or GLM 5, all through the same endpoint and SDK. If your agent needs to read images during a conversation, vision models such as Gemma 3 27B and Kimi VL A3B are available through identical chat completions endpoints.
End-to-End Code Example
The following Python script demonstrates a multi-turn conversation with a calculator tool. Notice that the only vendor-specific configuration is the base_url and model name.
from openai import OpenAI
import json
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
def calculate(expression: str) -> str:
try:
return str(eval(expression))
except Exception as e:
return f"Error: {e}"
tools = [
{
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a mathematical expression",
"parameters": {
"type": "object",
"properties": {
"expression": {"type": "string"}
},
"required": ["expression"]
}
}
}
]
messages = [
{"role": "system", "content": "You are a helpful assistant. Use the calculate tool for math."},
{"role": "user", "content": "What is 136 times 42? Also, what is the square root of 144?"}
]
# First request: model decides to call the tool
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
tools=tools,
stream=False
)
msg = response.choices[0].message
messages.append(msg)
# Execute tool calls locally
if msg.tool_calls:
for tc in msg.tool_calls:
if tc.function.name == "calculate":
result = calculate(json.loads(tc.function.arguments)["expression"])
messages.append({
"role": "tool",
"tool_call_id": tc.id,
"content": result
})
# Second request: model answers with tool results
final = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
tools=tools,
stream=True
)
for chunk in final:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
This pattern works for any Oxlo.ai chat model that supports tool use. You can also set response_format={"type": "json_object"} when you need structured output instead of free-form text.
Cost Optimization for Long Conversations
Token-based billing penalizes long system prompts and lengthy chat histories. Because Oxlo.ai charges per request, not per token, long-context workloads and agentic loops become predictable. You can keep a detailed system prompt, include retrieved documents, and maintain a longer sliding window without linear cost growth.
For teams running customer support bots or internal agents that process hundreds of turns per session, the difference is significant. Rather than estimating token counts for every turn, you know the exact cost per API call. You can compare plans and request allowances on the Oxlo.ai pricing page.
Conclusion
Conversational AI is a systems problem. With the right memory strategy, tool use, and an inference backend that does not tax context length, you can build agents that feel stateful and responsive. Oxlo.ai gives you the models, the OpenAI-compatible API, and the request-based pricing structure to do this without re-architecting your client code.
Top comments (0)