Multimodal Reasoning Tutorial: Unlocking the Power of AI

We are going to build a visual reasoning agent that accepts an image and a question, then returns a structured analysis with separate reasoning and answer blocks. This is useful for automated debugging, content moderation, or any workflow where you need a model to think through what it sees before acting. I chose Oxlo.ai for this because request-based pricing does not scale with input length, which matters when we are shipping large base64 image strings back and forth.

What you'll need

• Python 3.10 or newer
• The OpenAI SDK: pip install openai
• An Oxlo.ai API key from https://portal.oxlo.ai
• A sample PNG or JPEG to test with

Step 1: Configure the Oxlo.ai client

Oxlo.ai exposes a fully OpenAI-compatible API. We only need to swap the base URL and plug in our API key.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Define the system prompt

I force the model to separate its chain-of-thought from its final answer so downstream code can log reasoning independently.

SYSTEM_PROMPT = """You are a visual reasoning assistant.
When given an image and a question, first analyze what you see in detail.
Describe relevant objects, text, layout, and any anomalies.
Then provide your reasoning step by step.
Finally, give a concise answer.

Format your response exactly like this:

Your step-by-step visual analysis and reasoning.


Your final, concise answer.
"""

Step 3: Encode the image

The chat completions endpoint expects a base64 data URL for inline images. This helper reads a local file and returns the encoded string.

import base64

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        encoded = base64.b64encode(image_file.read()).decode("utf-8")
    return f"data:image/png;base64,{encoded}"

Step 4: Build the agent call

We assemble a multimodal message payload and send it to Kimi K2.6, which handles vision, long context, and reasoning in one pass.

def analyze_image(image_path: str, question: str) -> str:
    data_url = encode_image(image_path)

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": data_url}},
                    {"type": "text", "text": question},
                ],
            },
        ],
        max_tokens=2048,
    )

    return response.choices[0].message.content

Step 5: Parse the structured response

The model returns reasoning inside custom XML tags. This parser extracts both sections so the rest of your pipeline can use them separately.

import re

def parse_response(text: str) -> dict:
    thinking_match = re.search(r"(.*?)", text, re.DOTALL)
    answer_match = re.search(r"(.*?)", text, re.DOTALL)

    return {
        "thinking": thinking_match.group(1).strip() if thinking_match else "",
        "answer": answer_match.group(1).strip() if answer_match else text,
    }

Run it

Here is a complete script that wires everything together. Save it as visual_agent.py, put a screenshot named dashboard.png in the same folder, and run it.

if __name__ == "__main__":
    raw_output = analyze_image("dashboard.png", "Why is the error rate spike happening?")
    parsed = parse_response(raw_output)

    print("=== REASONING ===")
    print(parsed["thinking"])
    print("\n=== ANSWER ===")
    print(parsed["answer"])

Example output:

=== REASONING ===
The screenshot shows a Grafana dashboard for the checkout service.
Between 14:00 and 14:45, the error-rate panel jumps from 0.2% to 12%.
The logs panel indicates a sharp increase in 503 responses from the payment-gateway upstream.
No deployment markers appear during that window, so the root cause is likely an external dependency, not a code change.

=== ANSWER ===
The spike is caused by elevated 503 errors from the payment-gateway upstream between 14:00 and 14:45. Check the gateway status page and retry logic.

Wrap-up and next steps

You now have a working multimodal reasoning agent on top of Oxlo.ai. Because Oxlo.ai charges per request rather than per token, adding more images or asking for longer reasoning traces does not inflate your bill.

Two concrete next steps: wire this into a CI pipeline to auto-explain failed UI tests, or swap Kimi K2.6 for Qwen 3 32B if you need stronger multilingual reasoning on non-English screenshots. See the Oxlo.ai pricing page to pick a plan that fits your volume.