Introduction to Multimodal Reasoning: Concepts, Applications, and Benefits

#learnai #oxlo #ai

We're going to build a lightweight multimodal reasoning agent on Oxlo.ai that ingests both images and text, then returns structured JSON describing what it sees and why. This is useful for automating visual inspection tasks, content moderation, or any workflow where you need an LLM to reason over pixels and language together.

What you'll need

Before starting, make sure you have the following ready:

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK installed: pip install openai

Step 1: Configure the Oxlo.ai client

I always start by verifying the connection with a simple text call. This confirms my key and the Oxlo.ai endpoint are working before I add image payloads.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "user", "content": "Say 'Connection OK'"},
    ],
)
print(response.choices[0].message.content)

Step 2: Define the system prompt

I want the model to act like an inspector that outputs raw JSON. I keep the prompt in a module-level constant so I can tweak it without touching the logic.

SYSTEM_PROMPT = """You are a multimodal reasoning agent. Analyze the user's image and question carefully.
Respond with a JSON object containing exactly these keys:
- summary: a one-sentence description of the image
- reasoning: your step-by-step thought process
- answer: the direct answer to the user's question
Do not output markdown fences or commentary outside the JSON."""

Step 3: Send a vision-enabled request

Oxlo.ai's API is fully OpenAI-compatible, so I can pass an image URL inside a content array. I use kimi-k2.6 because it handles vision and long context well.

user_message = [
    {
        "type": "text",
        "text": "What manufacturing defect is visible on this circuit board, if any?"
    },
    {
        "type": "image_url",
        "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3f/Arduino_Uno_Rev3.jpg/1200px-Arduino_Uno_Rev3.jpg"
        }
    }
]

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)
print(response.choices[0].message.content)

Step 4: Build a reusable agent class

Hardcoding messages gets messy. I wrote a small class that accepts either a public URL or a local file path, converts local files to base64 data URLs, and returns parsed JSON.

import base64
import json
from pathlib import Path
from openai import OpenAI

class VisualReasoningAgent:
    def __init__(self, api_key: str, model: str = "kimi-k2.6"):
        self.client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=api_key)
        self.model = model
        self.system_prompt = SYSTEM_PROMPT

    def _build_image_content(self, image_source: str):
        if image_source.startswith("http"):
            return {"type": "image_url", "image_url": {"url": image_source}}
        path = Path(image_source)
        if not path.exists():
            raise FileNotFoundError(f"Image not found: {image_source}")
        ext = path.suffix.lstrip(".")
        if ext == "jpg":
            ext = "jpeg"
        b64 = base64.b64encode(path.read_bytes()).decode()
        data_url = f"data:image/{ext};base64,{b64}"
        return {"type": "image_url", "image_url": {"url": data_url}}

    def ask(self, question: str, image_source: str):
        content = [
            {"type": "text", "text": question},
            self._build_image_content(image_source),
        ]
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": self.system_prompt},
                {"role": "user", "content": content},
            ],
            temperature=0.2,
        )
        raw = response.choices[0].message.content
        cleaned = raw.strip().removeprefix("

```json").removeprefix("```

").removesuffix("

```

").strip()
        return json.loads(cleaned)

Run it

Now I can point the agent at any image and get structured reasoning back. Here is a small script that exercises the class with a public image and prints the parsed result.

agent = VisualReasoningAgent(api_key="YOUR_OXLO_API_KEY")
result = agent.ask(
    question="List all visible components and their likely functions.",
    image_source="https://upload.wikimedia.org/wikipedia/commons/thumb/3/3f/Arduino_Uno_Rev3.jpg/1200px-Arduino_Uno_Rev3.jpg"
)
print(json.dumps(result, indent=2))

When I ran this, the output looked like this:

{
  "summary": "An Arduino Uno Rev3 board with an ATmega328P microcontroller, USB connector, and pin headers.",
  "reasoning": "The board has the distinctive blue PCB and 'Arduino Uno' silkscreen. The large rectangular chip is the ATmega328P. The USB-B port is for power and programming. Rows of female headers provide digital and analog I/O.",
  "answer": "Visible components include the ATmega328P microcontroller, USB-B port, barrel jack, reset button, and female pin headers for shields."
}

Wrap-up and next steps

That is the core of a multimodal reasoning agent on Oxlo.ai. Because Oxlo.ai uses request-based pricing, adding high-resolution images does not inflate your bill the way token-based providers do. You can explore the exact plans at https://oxlo.ai/pricing.

Two concrete next steps: wire the agent into a FastAPI endpoint so you can POST images and questions from a frontend, or swap kimi-k2.6 for qwen-3-32b if you need stronger multilingual reasoning on non-English diagrams.