shashank ms

Posted on Aug 5

Multimodal Reasoning Deployment Guide: Overcoming Common Challenges

#engineering #oxlo #ai

Intro

We are building a visual defect triage agent that consumes screenshots and user descriptions to generate structured incident reports. This is for platform teams that want to cut down on manual Level-0 ticket sorting. We will wire it to Oxlo.ai using the OpenAI-compatible SDK and a vision-capable model so you can drop it into an existing pipeline without vendor lock-in.

What you'll need

Before starting, grab the following:

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A sample PNG or JPG screenshot for testing

Oxlo.ai uses request-based pricing, so your cost stays flat per call regardless of how large the image or prompt is. See https://oxlo.ai/pricing for details.

Step 1: Configure the Oxlo.ai client

I always start by verifying the connection with a lightweight text call. This confirms the API key and base URL are correct before we add image bytes to the mix.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY"),
)

# Quick connectivity check
response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[{"role": "user", "content": "Say OK"}],
    max_tokens=10,
)
print("API status:", response.choices[0].message.content)

Step 2: Encode the image for the multimodal payload

Vision models on Oxlo.ai accept base64-encoded images inside the standard OpenAI chat format. I keep this in a helper so I can swap filenames without touching the inference logic.

import base64

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def build_vision_content(image_path: str, user_text: str) -> list:
    b64 = encode_image(image_path)
    uri = f"data:image/png;base64,{b64}"
    return [
        {
            "type": "image_url",
            "image_url": {"url": uri, "detail": "high"},
        },
        {"type": "text", "text": user_text},
    ]

Step 3: Define the system prompt for structured reasoning

The system prompt forces the model to reason explicitly before emitting JSON. This reduces hallucinated component names and gives us an audit trail.

SYSTEM_PROMPT = """You are a visual defect triage agent.
Analyze the provided screenshot and the user's description.
Reason step by step inside the reasoning field, then output a JSON object with exactly these keys:
- summary: a one-sentence description of the issue
- severity: one of Critical, High, Medium, Low
- component: the UI or backend component most likely involved
- confidence: a float between 0 and 1
- suggested_fix: a concise remediation step for engineering

Be conservative. If the image does not match the user's description, set confidence below 0.5 and explain why."""

Step 4: Assemble the user message with vision and text

Now we combine the image and text into a single user message. I use the high detail setting so small UI elements remain readable.

def build_messages(image_path: str, user_text: str):
    content = build_vision_content(image_path, user_text)
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": content},
    ]

# Example inputs
IMAGE_PATH = "checkout_error.png"
USER_TEXT = (
    "The checkout button is unresponsive after entering shipping info. "
    "No spinner appears and the page does not redirect."
)

Step 5: Run inference with JSON mode on Oxlo.ai

We call kimi-k2.6 because it handles vision, long context, and structured reasoning well. Setting response_format to json_object forces valid JSON, which removes a whole class of parsing errors in production.

import json

def triage(image_path: str, user_text: str) -> dict:
    messages = build_messages(image_path, user_text)
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=messages,
        response_format={"type": "json_object"},
        temperature=0.2,
        max_tokens=1024,
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

report = triage(IMAGE_PATH, USER_TEXT)
print(json.dumps(report, indent=2))

Step 6: Validate the response and handle low-confidence predictions

Parsing JSON is not enough. I add a validation layer that flags low-confidence results for human review so the agent never silently ships a bad classification.

REQUIRED_KEYS = {"summary", "severity", "component", "confidence", "suggested_fix"}

def validate(report: dict) -> dict:
    missing = REQUIRED_KEYS - report.keys()
    if missing:
        raise ValueError(f"Missing keys: {missing}")

    confidence = float(report.get("confidence", 0))
    if confidence < 0.7:
        report["escalate"] = True
        report["escalation_reason"] = "Low confidence score"
    else:
        report["escalate"] = False

    if report.get("severity") == "Critical" and confidence < 0.8:
        report["escalate"] = True
        report["escalation_reason"] = "Critical severity requires high confidence"

    return report

validated = validate(report)
print("Escalate?", validated["escalate"])
print("Final report:", json.dumps(validated, indent=2))

Run it

Putting it all together, here is the full script entrypoint. When I run this against a screenshot of a frozen checkout page, the agent returns a structured report in under a second with no cold start.

if __name__ == "__main__":
    if not os.environ.get("OXLO_API_KEY"):
        raise RuntimeError("Set OXLO_API_KEY first.")

    report = triage("checkout_error.png", USER_TEXT)
    validated = validate(report)

    if validated["escalate"]:
        print("Human review required:", validated.get("escalation_reason"))
    else:
        print("Auto-routed to team:", validated["component"])

    print(json.dumps(validated, indent=2))

Example output:

{
  "summary": "Checkout button remains unresponsive after shipping form submission",
  "severity": "High",
  "component": "Payment Frontend",
  "confidence": 0.85,
  "suggested_fix": "Check frontend event listener on checkout button and verify payment gateway script is loading without CORS errors",
  "escalate": false,
  "escalation_reason": null
}

Wrap-up and next steps

This agent is already useful as a ticket pre-processor. The next concrete step is to wire the validate() output into a Slack webhook or PagerDuty integration so Critical items wake up an on-call engineer immediately. After that, you can add a function calling tool that queries your component registry to auto-link the suspected component to the owning team.

Because Oxlo.ai charges per request rather than per token, adding larger screenshots or longer system prompts does not change the unit cost. That makes it straightforward to iterate on prompt length and image detail without surprises. See https://oxlo.ai/pricing to pick a plan that matches your volume.

DEV Community