We are building a multimodal support agent that reads user messages, inspects device photos, and drafts structured repair tickets. It routes tasks across four specialized models hosted on Oxlo.ai. If you handle technical support or field service automation, this pattern cuts response time and keeps costs predictable.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Set up the Oxlo.ai client
I start every project with a single client instance. Oxlo.ai exposes an OpenAI-compatible endpoint, so the SDK works without changes.
from openai import OpenAI
import json
import base64
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
Step 2: Define the agent system prompt
The system prompt anchors tone and constraints. I keep it editable because product teams always tweak it later.
SYSTEM_PROMPT = """You are a smart home technical support agent.
Your job is to help users diagnose device issues and decide whether to send a technician.
Be concise, ask clarifying questions when needed, and never guess about safety-critical wiring.
If a user provides a photo, base your assessment on the visual evidence.
"""
Step 3: Classify the issue with Qwen 3 32B
Before we touch any image, we route the request. Qwen 3 32B is strong at multilingual reasoning and agent workflows, so I use it to classify intent and decide if we need vision analysis.
def classify_issue(user_message: str) -> str:
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": "Classify the user issue into one of: hardware_damage, connectivity, firmware_bug, or general_inquiry. Reply with only the category name."},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.content.strip()
Step 4: Analyze device photos with Kimi K2.6
If the user attached an image, we pass it to Kimi K2.6. It handles advanced reasoning and vision, and because Oxlo.ai uses flat per-request pricing, sending a high-resolution photo does not balloon the cost the way token-based providers charge for image tokens. See https://oxlo.ai/pricing for details.
def encode_image(image_path: str) -> str:
with open(image_path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
def analyze_image(image_path: str, user_message: str) -> str:
base64_image = encode_image(image_path)
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": "Describe any visible damage, model numbers, or LED status in the photo. Be factual."},
{"role": "user", "content": [
{"type": "text", "text": user_message},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}},
]},
],
)
return response.choices[0].message.content.strip()
Step 5: Draft the response with Llama 3.3 70B
Now we combine the classification, the image description, and the original question into a final answer. Llama 3.3 70B is the general-purpose workhorse, and on Oxlo.ai we can switch models mid-pipeline without juggling multiple API keys or base URLs.
def draft_response(user_message: str, category: str, image_description: str | None) -> str:
context_parts = [f"Issue category: {category}"]
if image_description:
context_parts.append(f"Image analysis: {image_description}")
context = "\n".join(context_parts)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"{user_message}\n\n{context}"},
],
)
return response.choices[0].message.content.strip()
Step 6: Emit a structured ticket with DeepSeek V3.2
For the CRM, we need a structured repair ticket. DeepSeek V3.2 handles coding and reasoning tasks well, and Oxlo.ai supports JSON mode, so we can enforce valid output.
def generate_ticket(user_message: str, category: str, image_description: str | None, agent_response: str) -> dict:
prompt = f"""Create a repair ticket JSON with keys: summary, category, visual_evidence, recommended_action, parts_needed (list).
User message: {user_message}
Category: {category}
Image description: {image_description or 'None'}
Agent response: {agent_response}
"""
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[
{"role": "system", "content": "You emit only valid JSON. No markdown fences."},
{"role": "user", "content": prompt},
],
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
Step 7: Wire the pipeline
The pipeline checks for an image path, branches accordingly, and returns both the human reply and the machine-readable ticket.
def support_agent(user_message: str, image_path: str | None = None) -> dict:
category = classify_issue(user_message)
image_description = analyze_image(image_path, user_message) if image_path else None
reply = draft_response(user_message, category, image_description)
ticket = generate_ticket(user_message, category, image_description, reply)
return {"reply": reply, "ticket": ticket}
Run it
Here is how I test the agent locally with a simulated camera offline issue and a photo of the device.
if __name__ == "__main__":
msg = "My front door camera went offline after last night's storm. The LED is blinking red."
# Replace with a local file path to test vision
photo = "doorbell_camera.jpg"
result = support_agent(msg, image_path=photo)
print("=== AGENT REPLY ===")
print(result["reply"])
print("\n=== TICKET ===")
print(json.dumps(result["ticket"], indent=2))
Example output:
=== AGENT REPLY ===
The blinking red LED after a storm usually indicates a power or network issue. Check that the transformer is dry and the Wi-Fi access point is online. If the LED does not turn solid blue within 10 minutes, we will dispatch a technician.
=== TICKET ===
{
"summary": "Doorbell camera offline post-storm with red LED blink",
"category": "connectivity",
"visual_evidence": "LED blinking red, no visible water ingress on casing",
"recommended_action": "Verify transformer and network connectivity before dispatch",
"parts_needed": []
}
Wrapping up
This pipeline is easy to extend. Two concrete next steps: wire the support_agent function into a FastAPI endpoint so your CRM can call it directly, and add the Oxlo.ai audio/transcriptions endpoint to accept voice memos from field technicians instead of text.
Top comments (0)