What is Multimodal Reasoning? An Introduction

We are building a lightweight multimodal reasoning agent that accepts an image and a text question, then returns a structured analysis by combining visual perception with step-by-step logic. This helps developers automate visual inspection, content moderation, or documentation validation without managing separate vision and language pipelines. Because Oxlo.ai charges a flat rate per request, adding large images or long reasoning chains does not inflate cost the way token-based pricing does.

What you'll need

• Python 3.10 or newer
• The OpenAI SDK: pip install openai
• An Oxlo.ai API key from https://portal.oxlo.ai
• A sample image file named sample.jpg in your working directory

Step 1: Define the system prompt

The system prompt tells the model to act as a visual reasoning engine and to return its work as structured JSON. Keeping this prompt versioned in code makes the agent behavior reproducible.

SYSTEM_PROMPT = """You are a visual reasoning agent. Your job is to inspect an image, describe what you see, and answer the user's question using step-by-step logic.

Rules:
1. First, list the key visual elements you observe.
2. Then, reason through how those elements relate to the user's question.
3. Finally, provide a concise answer.

Output strictly valid JSON with these keys:
- "observations": a list of strings
- "reasoning": a short paragraph
- "answer": a string
"""

Step 2: Initialize the Oxlo.ai client

We point the OpenAI SDK at Oxlo.ai's endpoint. This is a drop-in replacement, so the same code works for any Oxlo.ai model. I am using Kimi K2.6 because it handles both vision and advanced reasoning in a single request.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 3: Encode the image

Vision models on Oxlo.ai accept base64 data URLs inline. This helper reads a local JPEG and returns the formatted string.

import base64

def encode_image(path):
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/jpeg;base64,{b64}"

image_data_url = encode_image("sample.jpg")

Step 4: Build the multimodal message

Instead of a plain text string, the user message content is a list. The first part is the question, and the second part is the image URL. This is the standard OpenAI format, and Oxlo.ai supports it for vision models like Kimi K2.6.

QUESTION = "What safety hazards are visible in this image, and what is the most urgent one?"

user_message = [
    {"type": "text", "text": QUESTION},
    {"type": "image_url", "image_url": {"url": image_data_url}}
]

Step 5: Run the inference

We call the chat completions endpoint with JSON mode enabled. On Oxlo.ai, this counts as a single request regardless of image size or output length, which makes iterating on visual agents predictable.

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    response_format={"type": "json_object"},
    max_tokens=1024,
)

result = response.choices[0].message.content
print(result)

Run it

Here is the complete script. Save it as inspect.py, replace YOUR_OXLO_API_KEY, place sample.jpg next to it, and run python inspect.py.

import base64
from openai import OpenAI

SYSTEM_PROMPT = """You are a visual reasoning agent. Your job is to inspect an image, describe what you see, and answer the user's question using step-by-step logic.

Rules:
1. First, list the key visual elements you observe.
2. Then, reason through how those elements relate to the user's question.
3. Finally, provide a concise answer.

Output strictly valid JSON with these keys:
- "observations": a list of strings
- "reasoning": a short paragraph
- "answer": a string
"""

def encode_image(path):
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/jpeg;base64,{b64}"

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

image_data_url = encode_image("sample.jpg")

user_message = [
    {"type": "text", "text": "What safety hazards are visible, and what is the most urgent one?"},
    {"type": "image_url", "image_url": {"url": image_data_url}}
]

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    response_format={"type": "json_object"},
    max_tokens=1024,
)

print(response.choices[0].message.content)

Example output:

{
  "observations": [
    "A frayed power cable is lying in a puddle of water.",
    "A metal ladder is leaning against an exposed electrical panel.",
    "No wet floor signage is present."
  ],
  "reasoning": "Water and electricity create a lethal combination, so the submerged cable is the most immediate threat to life. The ladder contacting the panel is secondary but still dangerous. The missing signage is a compliance issue rather than an urgent physical hazard.",
  "answer": "The most urgent hazard is the frayed power cable in the water because it poses an immediate electrocution risk."
}

Next steps

Try adding tool use so the agent can query an external database after it identifies a hazard, or batch-process a directory of images and write structured results to SQLite. You can also extend the system prompt to request specific confidence scores or cited bounding boxes if your application needs finer granularity.