Introduction to Multimodal Reasoning

We are building a visual diagnosis agent that consumes screenshots and user descriptions to reason about UI bugs and system errors. It combines vision and language understanding to generate structured incident reports without sending your data to closed-source platforms. Frontend teams and SREs can use it to triage issues faster by automating the first pass of root-cause analysis.

What you'll need

• Python 3.10 or newer.
• An Oxlo.ai API key from https://portal.oxlo.ai.
• The OpenAI SDK: pip install openai.
• A sample image file named error_screenshot.png in your working directory.

Step 1: Instantiate the Oxlo.ai client

I keep the client in a dedicated module so I do not repeat boilerplate. The base URL points to Oxlo.ai, and I use kimi-k2.6 because it handles both vision and long-context reasoning in a single request.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Encode the image

Vision models on Oxlo.ai accept base64-encoded PNG and JPEG data inline. This helper reads a local file and returns the data URI string.

import base64

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

Step 3: Define the reasoning system prompt

I want the model to separate observation from inference. The prompt forces it to reason aloud before concluding and to output strict JSON.

SYSTEM_PROMPT = """You are a senior site-reliability engineer with expertise in frontend systems.
When given a screenshot and a user report, reason step by step:
1. Observe what is visibly wrong in the image (errors, blank states, layout issues, console messages).
2. Correlate those observations with the user's text description.
3. Hypothesize the most likely root cause.
4. Propose an immediate fix and a prevention step.

Respond in valid JSON with keys: observation, correlation, root_cause, immediate_fix, prevention."""

Step 4: Build the multimodal request

The user message is an array of content blocks. One block carries the base64 image, the other carries the text query. Because Oxlo.ai uses request-based pricing, sending a large screenshot and a verbose system prompt does not inflate cost the way token-based inference does. You can compare plans at https://oxlo.ai/pricing.

def diagnose_issue(image_path, user_description):
    b64_image = encode_image(image_path)

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{b64_image}"
                        }
                    },
                    {
                        "type": "text",
                        "text": f"User report: {user_description}"
                    }
                ]
            }
        ],
        response_format={"type": "json_object"}
    )

    return response.choices[0].message.content

Step 5: Parse and display the result

The raw response is a JSON string. I load it and print each reasoning stage so the output is readable in a terminal.

import json

def run_diagnosis(image_path, user_description):
    raw = diagnose_issue(image_path, user_description)
    result = json.loads(raw)

    print("=== Visual Diagnosis Report ===")
    for key, value in result.items():
        print(f"{key.replace('_', ' ').title()}: {value}")

    return result

Run it

Call the agent with a sample screenshot and a vague user complaint. The model reasons across both modalities to fill the gaps.

if __name__ == "__main__":
    report = run_diagnosis(
        image_path="error_screenshot.png",
        user_description="The checkout page is broken and customers are complaining."
    )
    print("\nRaw JSON:")
    print(json.dumps(report, indent=2))

Example output:

=== Visual Diagnosis Report ===
Observation: The payment form iframe is rendering a 404 error from the payment processor subdomain. The submit button is present but non-functional.
Correlation: User reports checkout is broken, which matches the visible 404 in the embedded payment frame.
Root Cause: The merchant's payment processor endpoint URL changed during a recent gateway migration, but the frontend iframe src was not updated.
Immediate Fix: Update the iframe src attribute in the checkout component to point to the new live endpoint.
Prevention: Add a daily health-check probe that loads the checkout iframe URL and asserts HTTP 200.

Raw JSON:
{
  "observation": "The payment form iframe is rendering a 404 error...",
  "correlation": "User reports checkout is broken...",
  "root_cause": "The merchant's payment processor endpoint URL changed...",
  "immediate_fix": "Update the iframe src attribute...",
  "prevention": "Add a daily health-check probe..."
}

Next steps

Swap the image helper to fetch from an S3 bucket if you want to process tickets at scale. You can also add a second call to deepseek-v3.2 on Oxlo.ai to generate the actual code diff for the proposed fix, turning the agent into a full repair pipeline.