What is Multimodal Reasoning?

#learnai #oxlo #ai

What we are building

We are building a support ticket triage agent that reasons across text descriptions and screenshots to classify severity and suggest fixes. This saves engineering teams from manually sorting visual bug reports. I shipped a version of this for my own project last quarter, and it cut initial response time in half.

What you will need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

Oxlo.ai exposes an OpenAI-compatible endpoint, so we only need to swap the base URL and model name.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

Step 2: Write the system prompt

The prompt tells the model it is a triage engineer and defines the JSON schema we expect back.

SYSTEM_PROMPT = """You are a senior support engineer. A user will submit a screenshot of an error and a short text description.

Analyze both modalities together. Use the image to confirm what the text claims. Then output a JSON object with exactly these keys:
- severity: one of "critical", "high", "low"
- component: the likely affected subsystem
- next_action: a concrete one-sentence fix or investigation step

Be concise. Do not hallucinate details not present in either the image or the text."""

Step 3: Encode the screenshot

The Oxlo.ai vision endpoint accepts base64-encoded PNG images embedded directly in the message content list.

import base64

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("bug_screenshot.png")

Step 4: Build the multimodal message

We pack the text description and the image into a single user message using the standard OpenAI vision format.

def build_ticket_message(description: str, base64_image: str):
    return {
        "role": "user",
        "content": [
            {"type": "text", "text": description},
            {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/png;base64,{base64_image}"
                },
            },
        ],
    }

user_message = build_ticket_message(
    "Checkout button is unresponsive after adding a discount code. See attached screenshot.",
    image_b64
)

Step 5: Call the vision model

I use Kimi K2.6 on Oxlo.ai because it handles advanced reasoning across vision and long context inputs without cold starts.

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        user_message,
    ],
    max_tokens=512,
)

output = response.choices[0].message.content
print(output)

Run it

Here is the complete script and an example of the agent's output when I fed it a real screenshot of a 500 error modal.

import os
import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"],
)

SYSTEM_PROMPT = """You are a senior support engineer. A user will submit a screenshot of an error and a short text description.

Analyze both modalities together. Use the image to confirm what the text claims. Then output a JSON object with exactly these keys:
- severity: one of "critical", "high", "low"
- component: the likely affected subsystem
- next_action: a concrete one-sentence fix or investigation step

Be concise. Do not hallucinate details not present in either the image or the text."""

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def build_ticket_message(description: str, base64_image: str):
    return {
        "role": "user",
        "content": [
            {"type": "text", "text": description},
            {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/png;base64,{base64_image}"
                },
            },
        ],
    }

if __name__ == "__main__":
    image_b64 = encode_image("bug_screenshot.png")
    user_message = build_ticket_message(
        "Checkout button is unresponsive after adding a discount code. See attached screenshot.",
        image_b64
    )

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            user_message,
        ],
        max_tokens=512,
    )

    print(response.choices[0].message.content)

Example output:

{
  "severity": "high",
  "component": "checkout/promotions",
  "next_action": "Check the browser console for JavaScript errors on the discount application handler and verify the POST /api/v1/discounts response."
}

Wrap-up and next steps

Wire this agent into your Slack bug channel. Because Oxlo.ai uses request-based pricing, the cost does not balloon when you attach high-resolution screenshots. See https://oxlo.ai/pricing for details. You can also extend the pipeline to audio by adding Whisper transcription on the same API.

DEV Community