Introduction to Multimodal LLM Learning: Concepts and Applications

#learnai #oxlo #ai

We are building a multimodal learning tutor that takes an image, such as a textbook diagram or whiteboard photo, and answers targeted questions about it. This is useful for developers prototyping AI study tools or internal documentation assistants. Because Oxlo.ai uses flat per-request pricing, sending high-resolution vision payloads for experimentation does not inflate costs the way token-based metering would.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An image file to analyze, such as diagram.png
An Oxlo.ai API key from https://portal.oxlo.ai

1. Initialize the client and encode the image

We point the OpenAI SDK at Oxlo.ai and convert our local image to a base64 string. The model expects a standard data URL, so we do that conversion upfront.

import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("diagram.png")

2. Define the tutor's system prompt

The system prompt constrains the model to behave like a technical tutor, not a generic chatbot. It forces structured observations and a concrete follow-up suggestion.

SYSTEM_PROMPT = """You are a multimodal learning assistant. When given an image and a question:
1. Describe what you see in the image accurately.
2. Answer the user's question using evidence from the image.
3. Suggest one follow-up concept the user could explore next.
Be concise, technical, and avoid hallucinating details not present in the image."""

3. Build the multimodal message payload

OpenAI's chat format accepts a list of content blocks. We mix a text question with a base64-encoded image URL so the model receives both modalities in a single user turn.

user_question = "Explain the main feedback loop shown in this diagram."

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {"type": "text", "text": user_question},
            {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/png;base64,{image_b64}"
                },
            },
        ],
    },
]

4. Send the request and stream the response

We call Oxlo.ai using kimi-k2.6, a model that handles vision, reasoning, and long context. Streaming lets us watch the explanation arrive token by token without waiting for the full generation.

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=messages,
    stream=True,
)

for chunk in response:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)
print()

Run it

Save the complete script below as tutor.py, place any PNG named diagram.png in the same directory, and run it from your terminal. The example output shows the kind of structured response you should expect.

import argparse
import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = """You are a multimodal learning assistant. When given an image and a question:
1. Describe what you see in the image accurately.
2. Answer the user's question using evidence from the image.
3. Suggest one follow-up concept the user could explore next.
Be concise, technical, and avoid hallucinating details not present in the image."""

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def explain_image(image_path, question):
    b64 = encode_image(image_path)
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{b64}"},
                    },
                ],
            },
        ],
        stream=True,
    )

    for chunk in response:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)
    print()

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("image", help="Path to image file")
    parser.add_argument("question", help="Question about the image")
    args = parser.parse_args()
    explain_image(args.image, args.question)

Terminal usage and example output:

$ python tutor.py diagram.png "Explain the main feedback loop shown in this diagram."
The image shows a closed-loop control system containing a plant, sensor, controller, and actuator. The main feedback loop routes the sensor output back to the controller input, allowing the system to correct deviations from the setpoint in real time. A follow-up concept you could explore is the difference between negative feedback and feedforward control architectures.

Wrap-up and next steps

You now have a working multimodal tutor on top of Oxlo.ai. Two concrete ways to extend it: first, pass multiple images in the same content list to compare diagrams side by side, which is still a single flat request. Second, add function calling so the agent can query a documentation index when the image alone is not enough, leveraging Oxlo.ai's tool use support.

DEV Community