shashank ms

Posted on Sep 14

Multimodal Learning and Human-Computer Interaction with LLM: Opportunities and Challenges

#learnai #oxlo #ai

We are building a hands-free multimodal agent that looks at your screen, listens to a spoken question, and replies with both text and speech. It is useful for debugging a UI, walking through a diagram, or controlling a workstation when your hands are busy.

What you'll need

Grab an Oxlo.ai API key from https://portal.oxlo.ai. You will need Python 3.10 or newer and the OpenAI SDK.

pip install openai

You also need a sample image and an audio file. A screenshot saved as screen.png and a 16 kHz mono WAV question saved as question.wav are enough to test.

Step 1: Initialize the Oxlo.ai client

Every request goes through the OpenAI-compatible endpoint. I keep the client at the top of the script so the rest of the pipeline can reuse it.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Transcribe the voice command

If the user sends audio, pipe it through Whisper Large v3 on Oxlo.ai first. The transcription becomes the text prompt for the vision model.

def transcribe(audio_path: str) -> str:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-large-v3",
            file=f,
            language="en"
        )
    return response.text

# Example usage
user_question = transcribe("question.wav")
print("Transcript:", user_question)

Step 3: See the screen with a vision model

I use Gemma 3 27B on Oxlo.ai because it handles screenshots and diagrams without bloated context windows. The image is base64-encoded and passed inside the chat messages.

import base64

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def analyze_screen(image_path: str, question: str) -> str:
    b64 = encode_image(image_path)
    data_url = f"data:image/png;base64,{b64}"

    response = client.chat.completions.create(
        model="gemma-3-27b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {"type": "image_url", "image_url": {"url": data_url}},
                ],
            },
        ],
    )
    return response.choices[0].message.content

The system prompt grounds the model so it describes what it sees and then answers the question.

SYSTEM_PROMPT = """You are a multimodal HCI assistant. Look at the image carefully.
Describe the layout or diagram in one sentence, then answer the user's question.
Be concise. If you see a user interface, name the visible elements."""

Step 4: Speak the answer with Kokoro TTS

Text responses are useful, but speech closes the loop for hands-free interaction. Oxlo.ai hosts Kokoro 82M, which streams fast enough for real-time replies.

def speak(text: str, out_path: str = "response.mp3"):
    response = client.audio.speech.create(
        model="kokoro-82m",
        voice="default",
        input=text,
    )
    with open(out_path, "wb") as f:
        f.write(response.content)
    return out_path

Step 5: Wire the multimodal agent together

Now I bundle the three stages into a single callable agent. It accepts an image path and an audio path, runs the pipeline, prints the transcript and answer, and writes the spoken reply to disk.

class MultimodalAgent:
    def __init__(self, client: OpenAI):
        self.client = client

    def run(self, image_path: str, audio_path: str):
        print("1. Transcribing audio...")
        question = transcribe(audio_path)

        print("2. Analyzing image...")
        answer = analyze_screen(image_path, question)

        print("3. Synthesizing speech...")
        audio_out = speak(answer)

        print("\n--- Result ---")
        print(f"Q: {question}")
        print(f"A: {answer}")
        print(f"Audio saved to: {audio_out}")
        return answer

# Instantiate and run
agent = MultimodalAgent(client)
agent.run("screen.png", "question.wav")

Run it

Export your key and execute the script.

export OXLO_API_KEY="sk-oxlo.ai-..."
python agent.py

With a screenshot of a Python traceback and a spoken question, "What is causing this error?", the output looks like this:

1. Transcribing audio...
2. Analyzing image...
3. Synthesizing speech...

--- Result ---
Q: What is causing this error?
A: The screenshot shows a Python KeyError on line 42 of config.py. The dictionary 'settings' does not contain the key 'debug_mode'. Add a default value or check membership before accessing it.
Audio saved to: response.mp3

Wrap-up and next steps

The full pipeline runs on Oxlo.ai's request-based pricing, so a long screenshot with a lengthy voice prompt does not inflate the cost the way token-based providers do. For details, see https://oxlo.ai/pricing.

Two concrete next steps. First, replace the static audio file with a microphone stream and a voice-activity detector so the agent triggers on a wake word. Second, swap Gemma 3 27B for Kimi K2.6 when you need deeper reasoning over multi-page document screenshots, since Kimi K2.6 handles 131K context and advanced agentic coding.

DEV Community