Introduction to Multimodal LLM Learning

#learnai #oxlo #ai

We are building a command-line multimodal tutor that ingests a textbook diagram and answers questions about it. This is useful for students and developers who need to reason over visual material without copy-pasting descriptions into a chat interface. We will run the whole thing on Oxlo.ai so one flat request covers both the image bytes and the text prompt.

What you'll need

Python 3.10 or newer
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A local image file named diagram.png

Step 1: Set up the Oxlo.ai client

I keep secrets out of source control, so I read the key from the environment. Initialize the OpenAI SDK with Oxlo.ai's base URL and you are ready to call any model in the catalog.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Encode the image

The OpenAI-compatible format expects images as base64 data URLs. This helper handles the encoding so we can inline the image directly in the JSON payload.

import base64

def encode_image(path):
    with open(path, "rb") as f:
        return f"data:image/png;base64,{base64.b64encode(f.read()).decode('utf-8')}"

image_url = encode_image("diagram.png")

Step 3: Write the system prompt

I define the tutor personality up front. Keeping it in a constant makes A/B testing easy without touching request logic.

SYSTEM_PROMPT = (
    "You are a concise multimodal tutor. When a user sends an image and a question:\n"
    "1. Identify the key concepts shown in the image.\n"
    "2. Answer the question using evidence from the image.\n"
    "3. Suggest one follow-up concept the user should study next.\n"
    "Keep responses under 150 words."
)

Step 4: Send the multimodal request

We assemble a user message whose content is a list: one text item and one image_url item. I use kimi-k2.6 because it handles vision, long context, and reasoning in a single pass. Because Oxlo.ai charges per request rather than per token, adding the full base64 image does not inflate the cost.

user_message = [
    {"type": "text", "text": "Explain what this diagram is showing and why the central node matters."},
    {"type": "image_url", "image_url": {"url": image_url}},
]

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

print(response.choices[0].message.content)

Step 5: Keep conversation context

Real learning is multi-turn. I append each exchange to a messages list so the tutor retains context. On Oxlo.ai, long context histories still cost the same flat per-request rate, which makes back-and-forth tutoring sessions predictable.

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": user_message},
    {"role": "assistant", "content": response.choices[0].message.content},
]

follow_up = "Can you give me a real-world analogy for that central node?"
messages.append({"role": "user", "content": follow_up})

response2 = client.chat.completions.create(
    model="kimi-k2.6",
    messages=messages,
)

print(response2.choices[0].message.content)

Run it

Save a diagram as diagram.png, export your key, and run python tutor.py. Here is what my session looked like with a transformer architecture diagram.

The diagram shows a standard transformer encoder stack. The central node is the multi-head self-attention block; it lets the model weigh every other token in the sequence when encoding the current one. Without it, the network would process tokens in isolation and miss contextual relationships.

Follow-up concept: study positional encoding next, because attention itself is order-agnostic.

Real-world analogy: think of the central node like a moderator in a panel discussion. Instead of listening to only the person speaking right now, the moderator constantly gauges the relevance of every panelist's comments to decide how much weight each opinion should carry in the final summary.

Wrap up

You now have a working multimodal tutor backed by Oxlo.ai. Two ways to extend it: add function calling so the agent can generate interactive quizzes on the fly, or switch to llama-3.3-70b for text-only review sessions after the image has been parsed. Either way, the flat request pricing means your cost stays predictable even as you pass long conversation histories back and forth. See https://oxlo.ai/pricing for the latest plan details.

DEV Community