Chatbots that rely solely on prompt engineering for intent classification waste tokens and drift off-topic in production. A more reliable pattern combines explicit intent recognition with LLM-powered response generation. This guide walks through a hybrid architecture that classifies user intent, routes to the correct handler, and generates contextual replies. You will build a working prototype using Python, the OpenAI SDK, and Oxlo.ai for inference.
Why Intent Recognition Still Matters
Modern LLMs can follow instructions, but treating every message as an open-ended generation task creates unnecessary cost and latency. When a user asks to "reset my password" or "check order status," you want deterministic routing, not a creative guess. Explicit intent recognition gives you guardrails. It also reduces the context window pressure on your generation model, because you only inject relevant documents or tools after the intent is known.
Architecture Overview
The pipeline has four stages:
- Intent classification: An LLM call with JSON mode parses the user message into a structured intent and confidence score.
- Handler routing: A lightweight router maps the intent to a handler function, API call, or retrieval pipeline.
- Context assembly: The handler fetches data and formats it into a system prompt.
- Response generation: A chat model produces the final, grounded reply.
This pattern shines when you run it on a platform optimized for both low-latency classification and long-context generation. Oxlo.ai offers request-based pricing, so your cost per turn stays flat even when you pass long conversation histories or large retrieved documents to the generation model. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale, Oxlo.ai does not scale cost with input length. For chatbots that maintain multi-turn state or retrieve lengthy knowledge bases, that difference is significant. See the Oxlo.ai pricing page for plan details.
SDK Setup
Because Oxlo.ai is fully OpenAI SDK compatible, you only need to change the base URL and API key.
pip install openai
import openai
import json
import os
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
If you are prototyping, the Oxlo.ai Free tier includes 60 requests per day across 16+ models with a 7-day full-access trial. That is enough to validate this pipeline before moving to a production plan.
Step 1: Intent Classification with JSON Mode
We will use a fast, capable model to classify incoming messages. Qwen 3 32B is a strong candidate for multilingual reasoning and agent workflows, but Llama 3.3 70B works well as a general-purpose flagship. The key is to force valid JSON so your router never receives malformed output.
INTENT_SCHEMA = {
"type": "object",
"properties": {
"intent": {
"type": "string",
"enum": ["order_status", "reset_password", "product_question", "talk_to_human", "general"]
},
"confidence": {"type": "number", "minimum": 0, "maximum": 1},
"extracted_entities": {"type": "array", "items": {"type": "string"}}
},
"required": ["intent", "confidence", "extracted_entities"]
}
def classify_intent(user_message: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b", # or qwen3-32b on Oxlo.ai
messages=[
{"role": "system", "content": "You are an intent classifier. Respond only with JSON."},
{"role": "user", "content": user_message}
],
response_format={"type": "json_object"},
max_tokens=256
)
return json.loads(response.choices[0].message.content)
JSON mode guarantees parseable output. On Oxlo.ai, there are no cold starts on popular models, so this classification step stays consistently fast.
Step 2: Handler Routing and Context Assembly
Once you have a structured intent, route it with simple Python logic. Avoid prompting an LLM to make this decision. It adds latency and cost for a deterministic task.
def route(intent_result: dict, user_message: str):
intent = intent_result["intent"]
confidence = intent_result["confidence"]
if confidence < 0.7:
return "I'm not sure I understood. Could you rephrase that?"
if intent == "order_status":
return handle_order_status(intent_result["extracted_entities"])
elif intent == "reset_password":
return handle_reset_password(intent_result["extracted_entities"])
elif intent == "talk_to_human":
return handle_escalation()
else:
return generate_rag_response(user_message, intent)
def handle_order_status(entities):
# Mock database call
return {"order_id": entities[0] if entities else None, "status": "shipped"}
Step 3: Response Generation with Long Context
For the final reply, you often need to inject retrieved documents, conversation history, or tool results. This is where input length grows. On token-based providers, long system prompts and multi-turn histories inflate cost linearly. Oxlo.ai uses request-based pricing, so one flat cost per API request covers the full prompt regardless of length. For chatbots with 131K context windows or heavy retrieval, that model can be 10-100x cheaper than token-based alternatives.
Below is a generation call that includes a long system prompt and prior turns. We will use DeepSeek V4 Flash for efficient MoE inference with a 1M context window, or Kimi K2.6 for advanced reasoning and agentic coding.
def generate_rag_response(user_message: str, intent: str, history: list = None):
system_prompt = f"""You are a support assistant. The user's intent is classified as {intent}.
Use the following retrieved documentation to answer accurately.
---
{retrieved_docs} # Potentially thousands of tokens
"""
messages = [{"role": "system", "content": system_prompt}]
if history:
messages.extend(history)
messages.append({"role": "user", "content": user_message})
response = client.chat.completions.create(
model="deepseek-v4-flash", # or kimi-k2.6 on Oxlo.ai
messages=messages,
stream=True,
max_tokens=1024
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
Streaming responses improve perceived latency. Oxlo.ai supports streaming, function calling, and multi-turn conversations out of the box.
Step 4: Tool Use and Fallback Patterns
For intents that require live data, use function calling instead of hardcoded handlers. Define your tools, let the model request them, and inject the results into the next turn. This is especially useful for agentic chatbots that need to check calendars, query APIs, or modify records.
tools = [
{
"type": "function",
"function": {
"name": "get_order_status",
"description": "Retrieve status for an order ID",
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string"}
},
"required": ["order_id"]
}
}
}
]
def agentic_turn(user_message: str):
response = client.chat.completions.create(
model="qwen3-32b",
messages=[{"role": "user", "content": user_message}],
tools=tools,
tool_choice="auto"
)
return response.choices[0].message
If the classifier returns low confidence or the user explicitly asks for a human, escalate immediately. A clean fallback path is as important as a happy path.
Putting the Full Pipeline Together
def chatbot_turn(user_message: str, conversation_history: list):
# 1. Classify
intent_result = classify_intent(user_message)
# 2. Route
if intent_result["intent"] == "general" and intent_result["confidence"] < 0.6:
return "Could you clarify what you need help with?"
handler_data = route(intent_result, user_message)
# 3. Generate with context
if isinstance(handler_data, str):
return handler_data # Direct response from router
reply = generate_rag_response(
user_message,
intent_result["intent"],
history=conversation_history
)
return reply
Production Considerations
Latency: Splitting the pipeline into a fast classification call and a longer generation call lets you optimize each independently. Models like DeepSeek V3.2 or Oxlo.ai Coder Fast can handle classification if you need even lower latency.
Cost: Chatbots accumulate context. A production thread with 50 turns and retrieved documents can push thousands of tokens per request. Because Oxlo.ai charges per request, not per token, long-context workloads and agentic loops do not trigger surprise bills. Token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale scale cost with every word in your prompt and history. For a flat monthly rate, Oxlo.ai Pro offers 1,000 requests per day across all models, and Premium provides 5,000 requests per day with priority queue access. Enterprise plans add dedicated GPUs and unlimited volume.
Model choice: Oxlo.ai hosts over 45 models across seven categories. For this pipeline you might pair Qwen 3 32B or Llama 3.3 70B for classification, and Kimi K2.6, GLM 5, or DeepSeek R1 671B MoE for complex reasoning steps. You can switch models without changing client code because the API is fully OpenAI compatible.
Conclusion
Building a production chatbot requires more than a single prompt. By separating intent recognition from response generation, you gain reliability, lower latency, and easier maintenance. Running that architecture on Oxlo.ai gives you predictable request-based pricing, no cold starts, and a broad model catalog accessible through the standard OpenAI SDK. Start with the Free tier to prototype the pipeline, then scale as your context windows grow.
Top comments (0)