Building a production chatbot requires more than prompting a large language model. Without dialogue management, even the most capable LLM will lose track of user intent across turns, repeat questions, and fail to maintain business logic. Dialogue management provides the structural layer that governs turn-taking, slot filling, state persistence, and contextual memory. When paired with a powerful inference backend, you get a system that is both fluent and reliable.
What Is Dialogue Management
Large language models are stateless. Each API call is an independent function of the messages you send. Dialogue management compensates for this by maintaining a persistent state machine. It tracks which slots have been filled, what the user's current intent is, and whether the conversation has satisfied the conditions needed to trigger an action. In practice, this means your application logic owns the conversation flow, while the LLM owns the language generation.
Architecture Overview
A robust chatbot architecture separates concerns into three layers. The NLU layer extracts intents and entities. The dialogue manager updates an internal state object and decides the next system action. The LLM layer generates natural language responses conditioned on that state. This separation prevents the model from hallucinating business logic, because critical decisions are handled by deterministic code.
Choosing the Inference Backend
Your inference provider determines which models you can run, how you pay for context, and whether you face cold starts. Token-based pricing penalizes the long system prompts and extended conversation histories that dialogue managers naturally produce. Oxlo.ai uses flat per-request pricing, so the cost of a multi-turn agentic request does not scale with input length. This makes Oxlo.ai significantly cheaper for long-context and agentic workloads. The platform hosts 45+ models across seven categories, including Llama 3.3 70B for general-purpose dialogue and Qwen 3 32B for multilingual agent workflows, and it is fully OpenAI SDK compatible with no cold starts on popular models.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY")
)Implementing the Dialogue Manager
The dialogue manager is a Python class that owns the conversation state. It appends user messages to a history buffer, injects a system prompt that describes available tools and constraints, and calls the LLM through the OpenAI SDK. Oxlo.ai supports streaming, JSON mode, and function calling, so the manager can enforce structured outputs or delegate to external APIs.
from typing import List, Dict, Any
class DialogueState:
def __init__(self):
self.slots: Dict[str, Any] = {}
self.history: List[Dict[str, str]] = []
self.turn_count = 0
class DialogueManager:
def __init__(self, client, model: str = "llama-3.3-70b"):
self.client = client
self.model = model
self.state = DialogueState()
self.system_prompt = (
"You are a helpful assistant. You may ask clarifying questions "
"until all required slots are filled. Current slots: {}"
)
def process_turn(self, user_input: str) -> str:
self.state.history.append({"role": "user", "content": user_input})
self.state.turn_count += 1
messages = [
{"role": "system", "content": self.system_prompt.format(self.state.slots)}
] + self.state.history
response = self.client.chat.completions.create(
model=self.model,
messages=messages,
temperature=0.3,
max_tokens=512
)
assistant_msg = response.choices[0].message.content
self.state.history.append({"role": "assistant", "content": assistant_msg})
return assistant_msgTool Use and Function Calling
Real chatbots rarely operate in isolation. They query databases, call CRM APIs, or check inventory. Instead of parsing free text, the dialogue manager should expose tools via the LLM's native function-calling interface. Oxlo.ai supports function calling on compatible models, allowing the dialogue manager to receive structured tool calls, execute them locally, and feed the results back into context.
import json
TOOLS = [
{
"type": "function",
"function": {
"name": "check_order_status",
"description": "Retrieve the status of a customer order",
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string"}
},
"required": ["order_id"]
}
}
}
]
def handle_tool_call(self, tool_call):
name = tool_call.function.name
args = json.loads(tool_call.function.arguments)
if name == "check_order_status":
return {"status": "shipped", "order_id": args["order_id"]}
return {}
def process_turn_with_tools(self, user_input: str) -> str:
self.state.history.append({"role": "user", "content": user_input})
messages = [
{"role": "system", "content": self.system_prompt.format(self.state.slots)}
] + self.state.history
response = self.client.chat.completions.create(
model=self.model,
messages=messages,
tools=TOOLS,
tool_choice="auto"
)
msg = response.choices[0].message
if msg.tool_calls:
self.state.history.append({
"role": "assistant",
"tool_calls": [tc.model_dump() for tc in msg.tool_calls],
"content": msg.content or ""
})
for tc in msg.tool_calls:
result = self.handle_tool_call(tc)
self.state.history.append({
"role": "tool",
"tool_call_id": tc.id,
"content": json.dumps(result)
})
# Follow-up to get natural language response
second = self.client.chat.completions.create(
model=self.model,
messages=messages + self.state.history[-2:]
)
reply = second.choices[0].message.content
self.state.history.append({"role": "assistant", "content": reply})
return reply
self.state.history.append({"role": "assistant", "content": msg.content})
return msg.contentManaging Context and Memory
As conversations grow, unbounded history will eventually exceed model limits or dilute attention. A production dialogue manager should implement summarization, sliding windows, or key-value memory stores. Because Oxlo.ai charges per request rather than per token, you can maintain larger context windows without cost scaling linearly with input length. For extremely long sessions, models such as DeepSeek V4 Flash offer a 1 million token context window, while Kimi K2.6 provides 131K context for advanced reasoning and agentic coding tasks.
Complete Minimal Example
Below is a runnable skeleton that ties together state, tool use, and the Oxlo.ai inference endpoint. It uses the OpenAI SDK and targets a general-purpose model such as Llama 3.3 70B.
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ["OXLO_API_KEY"]
)
class SimpleChatbot:
def __init__(self):
self.history = []
self.tools = TOOLS # define TOOLS as shown above
def chat(self, user_input: str) -> str:
self.history.append({"role": "user", "content": user_input})
resp = client.chat.completions.create(
model="llama-3.3-70b",
messages=self.history,
tools=self.tools,
temperature=0.2
)
msg = resp.choices[0].message
# In production, handle msg.tool_calls here before appending
self.history.append({"role": "assistant", "content": msg.content})
return msg.content
if __name__ == "__main__":
bot = SimpleChatbot()
print(bot.chat("What is the status of order 12345?"))Deployment Considerations
When moving from prototype to production, latency and cost predictability become critical. Oxlo.ai offers no cold starts on popular models, so your dialogue manager receives consistent response times even during low-traffic periods. Streaming responses improve perceived latency for end users. Because pricing is request-based, your monthly bill is tied to conversation volume, not to the length of each system prompt or the depth of your context history. For teams evaluating providers, the flat per-request model can be 10 to 100 times cheaper than token-based alternatives for long-context and agentic workloads. See the Oxlo.ai pricing page for plan details.
Top comments (0)