Multimodal Reasoning Tutorial for Developers

#learnai #oxlo #ai

We are building a command-line visual auditor that takes a screenshot and a short user complaint, then reasons over both image and text to produce a structured JSON report. This helps QA engineers and frontend developers triage UI issues without maintaining brittle computer-vision pipelines. Because Oxlo.ai charges a flat rate per request, adding high-resolution screenshots and long system prompts does not inflate inference costs.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai
A test image such as a screenshot or mobile photo saved as checkout_bug.png

Step 1: Configure the client

Import the SDK and point it at Oxlo.ai. Reading the API key from the environment keeps credentials out of source control.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

Step 2: Encode the image

Vision endpoints on Oxlo.ai accept standard base64 data URLs. This helper reads any local PNG or JPEG and returns the encoded string.

import base64

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("checkout_bug.png")

Step 3: Define the system prompt

The prompt forces chain-of-thought reasoning and locks the output to a strict JSON schema so downstream automation can consume it reliably.

SYSTEM_PROMPT = """You are a senior QA engineer performing visual root-cause analysis.
Study the screenshot and the user's description. Reason step by step:
1. Identify every visible UI element relevant to the complaint.
2. Spot visual bugs, layout shifts, or missing elements.
3. Assign a severity: critical, major, minor, or cosmetic.
4. Recommend a precise fix.

Respond in this exact JSON structure:
{
  "elements_observed": ["..."],
  "likely_root_cause": "...",
  "severity": "...",
  "recommended_fix": "..."
}
"""

Step 4: Send the multimodal request

We use the OpenAI-compatible chat completions endpoint, passing the image as an image_url payload alongside the user's text. I am using kimi-k2.6 on Oxlo.ai because it handles vision and advanced reasoning under a single request-based price.

user_text = "The total price is overlapping with the checkout button on mobile."

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": user_text},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_b64}"
                    }
                }
            ]
        }
    ],
    response_format={"type": "json_object"}
)

raw_output = response.choices[0].message.content
print(raw_output)

Step 5: Parse and validate

Always verify structure before feeding the result into a CI pipeline or ticketing system. This guard raises early if the model omits a required field.

import json

def parse_report(raw_json: str) -> dict:
    result = json.loads(raw_json)
    required = {"elements_observed", "likely_root_cause", "severity", "recommended_fix"}
    missing = required - result.keys()
    if missing:
        raise ValueError(f"Missing keys: {missing}")
    return result

report = parse_report(raw_output)

Run it

Wrap the pipeline in a main block so you can call it from the terminal. Here is the complete script followed by example output.

if __name__ == "__main__":
    image_b64 = encode_image("checkout_bug.png")

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": user_text},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{image_b64}"
                        }
                    }
                ]
            }
        ],
        response_format={"type": "json_object"}
    )

    report = parse_report(response.choices[0].message.content)
    print(json.dumps(report, indent=2))

Example output:

{
  "elements_observed": [
    "Checkout button (class btn-primary)",
    "Order total label ($49.99)",
    "Mobile viewport 375px wide"
  ],
  "likely_root_cause": "The price container uses fixed positioning without z-index or responsive margin, causing overlap on viewports under 400px.",
  "severity": "major",
  "recommended_fix": "Replace fixed pixel margins with flexbox spacing and add a @media query for viewports below 400px that stacks the total above the button."
}

Next steps

Wire this script into a Playwright test suite so screenshots are captured automatically on failure and fed straight into the analyzer. If you need to process hundreds of images in a batch, keep in mind that Oxlo.ai bills per request rather than per token, so long system prompts and high-resolution images do not increase the cost of each inference call. You can review request-based plans at https://oxlo.ai/pricing.