Building Multimodal Models with LLMs: A Step-by-Step Guide

We are building a multimodal diagnostic agent that ingests screenshots or technical diagrams and returns structured bug reports. This gives QA teams and frontend engineers a fast way to triage visual issues without maintaining custom computer vision pipelines. The entire agent runs against Oxlo.ai's vision-capable API using the standard OpenAI SDK.

What you'll need

• Python 3.10 or newer
• The OpenAI SDK: pip install openai
• An Oxlo.ai API key from https://portal.oxlo.ai
• A sample image file such as a PNG screenshot

Step 1: Configure the Oxlo.ai client

Create a client instance pointed at Oxlo.ai. We will use kimi-k2.6 because it handles vision, reasoning, and long context in a single request.

from openai import OpenAI
import base64

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"  # from https://portal.oxlo.ai
)

Step 2: Prepare image input for the API

Oxlo.ai accepts images as base64 data URLs inside the chat messages payload. This helper reads a local file and returns the encoded string.

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

image_b64 = encode_image("dashboard_bug.png")
data_url = f"data:image/png;base64,{image_b64}"

Step 3: Define the system prompt

A strong system prompt keeps the model's output structured and actionable. Store it as a constant so you can iterate without touching the request logic.

SYSTEM_PROMPT = """You are a visual QA engineer. Analyze the provided screenshot and identify UI bugs, layout issues, or accessibility problems.

Return your findings as a JSON object with this exact structure:
{
  "issues": [
    {
      "severity": "critical|major|minor",
      "component": "name of UI element",
      "description": "what is wrong",
      "recommendation": "how to fix it"
    }
  ],
  "summary": "one sentence overall assessment"
}

Be concise. If no issues are found, return an empty issues array."""

Step 4: Build the analyzer function

Combine the text prompt and the image into a single user message, then send it to Oxlo.ai. The OpenAI-compatible format lets you mix text and image_url content blocks seamlessly.

def analyze_screenshot(image_path: str, question: str = "Analyze this screenshot for UI issues.") -> str:
    b64 = encode_image(image_path)
    url = f"data:image/png;base64,{b64}"
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {"type": "image_url", "image_url": {"url": url}},
                ],
            },
        ],
        max_tokens=1024,
    )
    return response.choices[0].message.content

Run it

Call the analyzer on a real image and print the result. If you do not have a dashboard bug handy, any screenshot with text and buttons will produce useful output.

if __name__ == "__main__":
    report = analyze_screenshot("dashboard_bug.png")
    print(report)

Example output from a malformed navigation bar:

{
  "issues": [
    {
      "severity": "major",
      "component": "navigation sidebar",
      "description": "The sidebar overlaps the main content area at viewport widths below 768px.",
      "recommendation": "Add a media query to collapse the sidebar into a hamburger menu on mobile breakpoints."
    },
    {
      "severity": "minor",
      "component": "submit button",
      "description": "Color contrast ratio is 2.9:1, below WCAG AA standards.",
      "recommendation": "Darken the button background to #005fcc to reach a 4.5:1 ratio."
    }
  ],
  "summary": "Two layout and accessibility issues detected that should be fixed before release."
}

Next steps

Batch process an entire directory of screenshots by wrapping analyze_screenshot in a loop and writing the JSON results to a log file. You can also add Oxlo.ai function calling to the system prompt so the agent opens Jira tickets automatically when it detects critical severity issues.