LLM Models for Multimodal Learning and Human-Computer Interaction for Accessibility

We're building a multimodal accessibility bridge. It takes an image and a user question, analyzes the visual content with a vision-capable LLM, and returns both a structured text description and a spoken audio file. This helps users with visual impairments interact with graphical interfaces, diagrams, or physical environments through natural language.

What you'll need

You need Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key. Install the SDK with pip and grab your key from the Oxlo.ai portal.

pip install openai

You will also need a sample image. Save one as sample.png in your working directory.

Step 1: Configure the Oxlo.ai client

First, import the SDK and point it at Oxlo.ai's OpenAI-compatible endpoint. I read my key from the environment to keep it out of source control.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

Step 2: Prepare image input

The chat completions endpoint accepts base64-encoded images. This helper reads any local image file and returns a data URL string.

import base64

def image_to_data_url(path):
    with open(path, "rb") as image_file:
        b64 = base64.b64encode(image_file.read()).decode("utf-8")
    ext = path.split(".")[-1].lower()
    mime = f"image/{ext}" if ext != "jpg" else "image/jpeg"
    return f"data:{mime};base64,{b64}"

image_url = image_to_data_url("sample.png")

Step 3: Define the system prompt

The system prompt tells the model how to describe images for accessibility. I keep it rigid so the output is predictable and easy to parse.

SYSTEM_PROMPT = (
    "You are an accessibility assistant. When shown an image, describe it for a "
    "user with low vision. Respond in exactly three sections:\n\n"
    "1. Summary: one sentence stating what the image shows.\n"
    "2. Elements: a bulleted list of visible text, buttons, icons, or objects.\n"
    "3. Layout: a spatial guide using clock positions, e.g., 'Submit button is at bottom right'.\n\n"
    "Be concise. Avoid visual-only terms like 'as you can see'."
)

Step 4: Analyze the image with Kimi K2.6

Kimi K2.6 handles vision, reasoning, and long context. We pass the image as an image_url content part and ask for a navigation description.

def describe_image(image_data_url, user_question):
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": user_question},
                    {"type": "image_url", "image_url": {"url": image_data_url}},
                ],
            },
        ],
        temperature=0.2,
    )
    return response.choices[0].message.content

text_description = describe_image(image_url, "Describe this interface so I can navigate it.")
print(text_description)

Step 5: Generate audio with Kokoro

Text output is useful, but speech is often faster for accessibility workflows. Oxlo.ai hosts Kokoro 82M, a lightweight text-to-speech model. We pipe the description into the audio speech endpoint and stream the result to a file.

def speak_text(text, output_path="output.mp3"):
    response = client.audio.speech.create(
        model="kokoro-82m",
        voice="af_bella",
        input=text,
    )
    response.stream_to_file(output_path)
    return output_path

audio_file = speak_text(text_description)
print(f"Audio saved to {audio_file}")

Step 6: Wrap it in a CLI

I tie the pieces together in a small script that accepts an image path and a question, then prints the description and writes the audio file.

import sys

def main(image_path, question):
    data_url = image_to_data_url(image_path)
    description = describe_image(data_url, question)
    print("\n--- Description ---\n")
    print(description)
    out = speak_text(description, "accessibility_description.mp3")
    print(f"\nAudio written to: {out}")

if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: python accessibility_bridge.py <image> <question>")
        sys.exit(1)
    main(sys.argv[1], sys.argv[2])

Run it

Save the full script as accessibility_bridge.py and invoke it with a screenshot and a question.

export OXLO_API_KEY="YOUR_OXLO_API_KEY"
python accessibility_bridge.py sample.png "What is on this screen?"

Example output:

--- Description ---

1. Summary: A web login form with email and password fields, a forgot-password link, and a blue sign-in button.
2. Elements:
   - "Email address" text input near top center.
   - "Password" masked input below email.
   - "Forgot password?" link under password field.
   - "Sign In" blue button below link.
3. Layout: Email at 12 o'clock, password at 1 o'clock, forgot link at 2 o'clock, sign-in button at 5 o'clock.

Audio written to: accessibility_description.mp3

Because Oxlo.ai charges per request rather than per token, you can send large high-resolution images and long prompts without the cost scaling with input size. For teams running accessibility tools at volume, that pricing model removes the guesswork from image-heavy workloads. See the details at https://oxlo.ai/pricing.

Next steps

Wire this into a browser extension so users can right-click any image and hear a description instantly. Or swap Kimi K2.6 for Qwen 3 32B to add multilingual support for non-English accessibility workflows.