A Comprehensive Guide to Multimodal LLMs

#learnai #oxlo #ai

We are building a multimodal bug report agent that ingests a screenshot and a short user description, then returns structured JSON with severity, component, and recommended fix. This helps support and QA teams triage visual issues without manual template filling. Because Oxlo.ai charges per request rather than per token, attaching a high resolution screenshot does not inflate the cost.

What you'll need

Before starting, grab an Oxlo.ai API key from https://portal.oxlo.ai. You will also need Python 3.10 or newer and the OpenAI SDK installed.

pip install openai

You will also need a sample PNG or JPEG screenshot to test with. Save it as bug_screenshot.png in your working directory.

1. Set up the Oxlo.ai client

Import the OpenAI SDK and point it at Oxlo.ai. This is a drop-in replacement. I keep my key in an environment variable, but you can paste it directly for local testing.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

2. Prepare image input

Multimodal LLMs on Oxlo.ai accept base64-encoded images inside the chat messages. I use a small helper to read a local file and wrap it in the required image_url payload.

import base64

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def make_multimodal_message(text, image_path):
    b64 = encode_image(image_path)
    return {
        "role": "user",
        "content": [
            {"type": "text", "text": text},
            {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/png;base64,{b64}"
                },
            },
        ],
    }

3. Define the system prompt

The system prompt constrains the model to act as a strict JSON generator. I ask it to identify severity, component, and a recommended fix based on the visual and textual input.

SYSTEM_PROMPT = """You are a precise bug triage assistant. 
Analyze the user description and the attached screenshot. 
Respond with a single JSON object containing exactly these keys:
- severity: one of "critical", "high", "medium", "low"
- component: the UI area or backend module affected
- summary: a one-sentence description of the bug
- fix: a concrete recommended next step for the developer

Do not include markdown formatting, explanations, or line breaks outside the JSON."""

4. Build the bug report function

Now I wire the pieces together. The function accepts a user message and an image path, constructs the multimodal payload, and sends it to Oxlo.ai using the vision-capable kimi-k2.6 model.

def analyze_bug(user_text, image_path):
    message = make_multimodal_message(user_text, image_path)
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            message,
        ],
    )
    
    return response.choices[0].message.content

5. Enforce structured output with JSON mode

To guarantee valid JSON, I enable JSON mode in the completion request. This locks the model to output only a parseable object, which is useful when piping results into a dashboard or database.

import json

def analyze_bug_structured(user_text, image_path):
    message = make_multimodal_message(user_text, image_path)
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            message,
        ],
        response_format={"type": "json_object"},
    )
    
    raw = response.choices[0].message.content
    return json.loads(raw)

Run it

Call the agent with a real screenshot and a short description. The example below assumes you have bug_screenshot.png in the same folder.

if __name__ == "__main__":
    result = analyze_bug_structured(
        user_text="The checkout button is missing on the mobile view after the latest deploy.",
        image_path="bug_screenshot.png"
    )
    print(json.dumps(result, indent=2))

Example output:

{
  "severity": "high",
  "component": "mobile-checkout-ui",
  "summary": "Checkout button not rendered in mobile viewport due to CSS regression.",
  "fix": "Inspect the media query for max-width 768px and restore the display:block rule for .checkout-btn."
}

Wrap-up and next steps

The agent now turns unstructured screenshots into structured tickets. A concrete next step is to wrap this function in a FastAPI endpoint and wire it to a Slack slash command so anyone can submit bugs directly from a channel. If you plan to process large batches of images, Oxlo.ai's request-based pricing keeps the cost flat regardless of image resolution or prompt length, which you can verify on the pricing page.

DEV Community