Modern customer support is no longer limited to text. Customers paste screenshots of error messages, upload photos of damaged goods, and expect agents to resolve issues across languages and channels in a single session. Building an autonomous support agent requires three capabilities working in concert: natural language understanding to parse intent, computer vision to interpret visual evidence, and a large language model to reason over both and execute actions through external tools. Oxlo.ai provides the inference backend for this stack, with request-based pricing and a model catalog that covers fast classification, deep reasoning, multimodal understanding, and agentic tool use.
Architecture Overview
A production support agent typically runs as an event-driven service. When a ticket arrives, the system first extracts structured intent and entities. If the ticket contains an image, a vision model generates a textual description or structured damage report. The core agent LLM then consumes the original message, the image analysis, and any retrieved context, such as order history, to decide whether to answer directly or invoke a tool. The result is streamed back to the user.
Oxlo.ai hosts models for every stage of this pipeline. You can route classification tasks to lightweight endpoints, route image understanding to vision models like Gemma 3 27B or Kimi VL A3B, and route complex reasoning to agentic models such as Qwen 3 32B, DeepSeek R1 671B MoE, or Kimi K2.6. Because the platform is fully OpenAI SDK compatible, you can orchestrate all three stages with a single client instance pointed at https://api.oxlo.ai/v1.
Intent Understanding with Structured Outputs
Traditional NLU pipelines rely on separate intent classifiers and entity extractors. A simpler modern approach is to use an LLM with JSON mode to produce structured intent and slots in one call. This reduces latency and eliminates the need to maintain bespoke classification models.
On Oxlo.ai, models such as Llama 3.3 70B and Qwen 3 32B support JSON mode and function calling. You can define a schema that captures intents like refund_request, technical_issue, or order_status, along with entities such as order_id and product_name. The model returns a parseable object that drives downstream routing.
import openai
import json
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
intent_schema = {
"type": "object",
"properties": {
"intent": {"enum": ["refund_request", "technical_issue", "order_status", "other"]},
"order_id": {"type": "string"},
"urgency": {"enum": ["low", "medium", "high"]}
},
"required": ["intent", "urgency"]
}
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "Extract intent and urgency from the support message. Respond in JSON."},
{"role": "user", "content": "My package arrived damaged. Order #48291. I need a refund immediately."}
],
response_format={"type": "json_object"},
max_tokens=256
)
result = json.loads(response.choices[0].message.content)
print(result)
# {'intent': 'refund_request', 'order_id': '48291', 'urgency': 'high'}
Interpreting Visual Evidence
When a customer attaches an image, the agent must read text in the image, identify UI elements, or assess physical damage. Oxlo.ai offers several vision models for this workload. Gemma 3 27B and Kimi VL A3B handle image comprehension, while Kimi K2.6 adds advanced reasoning and a 131K context window for long troubleshooting threads that include multiple screenshots.
The OpenAI SDK accepts base64-encoded images or image URLs in the message payload, so integrating vision is a matter of constructing the correct content list.
import base64
def encode_image(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
b64_image = encode_image("damaged_box.jpg")
response = client.chat.completions.create(
model="gemma-3-27b",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the damage visible in this image and estimate if the contents are likely affected."},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_image}"}}
]
}
],
max_tokens=512
)
vision_report = response.choices[0].message.content
Agentic Tool Use and Action
Understanding the problem is only half the task. The agent must act: query an order database, initiate a return label, or escalate to a human. Function calling lets the LLM declare which tool to invoke and with what arguments. Oxlo.ai supports function calling across its chat and reasoning models, including agent-optimized options like Qwen 3 32B, DeepSeek V3.2, and Kimi K2.6.
The following example defines a get_order_status tool. The model returns a tool call instead of plain text when it needs external data.
tools = [
{
"type": "function",
"function": {
"name": "get_order_status",
"description": "Retrieve the current status of an order by ID.",
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string", "description": "The order identifier."}
},
"required": ["order_id"]
}
}
}
]
messages = [
{"role": "system", "content": "You are a support agent. Use the available tool to answer order questions."},
{"role": "user", "content": "Where is my order #48291? Also, here is a photo of the damage."}
]
# Append the vision analysis from the previous step
messages.append({"role": "user", "content": f"Vision analysis: {vision_report}"})
response = client.chat.completions.create(
model="qwen3-32b",
messages=messages,
tools=tools,
tool_choice="auto"
)
if response.choices[0].message.tool_calls:
tool_call = response.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)
# Execute local function
order_data = get_order_status(args["order_id"])
# Append result and generate final answer
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": tool_call.function.name,
"content": json.dumps(order_data)
})
final = client.chat.completions.create(model="qwen3-32b", messages=messages)
print(final.choices[0].message.content)
Managing Context Length and Cost
Customer support threads accumulate history, policy documents, and previous ticket context. With token-based providers, long inputs inflate costs linearly and make multi-turn agent loops expensive. Oxlo.ai uses flat per-request pricing, so the cost of an API call does not increase when you add more tokens to the prompt. For long-context workloads, request-based pricing can be 10-100x cheaper than token-based alternatives, which matters when every turn includes a full system prompt, retrieved documentation, and prior conversation history.
Oxlo.ai also offers models with extended context windows for this exact use case. DeepSeek V4 Flash supports a 1 million token context, and Kimi K2.6 provides 131K tokens with advanced reasoning and vision. You can keep extensive ticket history in context without worrying about per-token metered billing. For details on request-based plans, see the Oxlo.ai pricing page.
Putting It All Together
A complete support agent pipeline on Oxlo.ai looks like this:
- Ingest: Receive the ticket text and optional image.
- Classify: Call a fast model with JSON mode to extract intent and urgency.
- Perceive: If an image is present, call a vision model to generate a structured report.
- Reason: Feed the intent, image report, and relevant history into an agentic LLM with tool definitions.
- Act: Execute any tool calls, append results, and stream the final response back to the customer.
Because every model is accessible through the same OpenAI-compatible endpoint, you can switch between a lightweight classifier and a heavy reasoning model without changing client libraries or authentication logic. There are no cold starts on popular models, so the agent remains responsive even under variable load.
Selecting Models for Each Subtask
Not every stage requires the same capacity. Oxlo.ai lets you optimize for latency and capability by selecting different models for each step.
- Fast classification and routing: Llama 3.3 70B or DeepSeek V3.2 for low-latency intent parsing.
- Vision analysis: Gemma 3 27B for general image understanding, or Kimi K2.6 when you need vision combined with deep reasoning and long context.
- Complex reasoning and tool use: Qwen 3 32B for multilingual agent workflows, DeepSeek R1 671B MoE for deep troubleshooting, or GLM 5 for long-horizon agentic tasks.
This mix-and-match approach ensures you are not over-provisioning compute for simple queries while still having the headroom for difficult cases.
Conclusion
Building a multimodal customer support agent requires more than a single large model. It demands a pipeline of specialized capabilities: structured NLU, computer vision, and stateful tool use. Oxlo.ai provides the models and the pricing structure to run this pipeline economically. With flat per-request pricing, OpenAI SDK compatibility, and a broad catalog that includes vision, reasoning, and agentic models, Oxlo.ai is a strong backend choice for teams shipping production support agents.
Top comments (0)