Building Multimodal Models with LLM

#aiinfrastructure #oxlo #ai

Multimodal large language models process and reason across text, images, audio, and structured data in a single forward pass. For developers, this means applications can now accept a screenshot, a voice memo, or a PDF diagram and return actionable text, generated images, or API calls without maintaining separate pipelines for each modality. Building production systems on top of these capabilities requires more than selecting a model. You need a composable inference stack that handles vision encoders, audio transcription, embedding retrieval, and image generation under one API contract with predictable latency and cost.

Unified Inference Architecture

Most production multimodal systems are not trained from scratch. They are orchestrated. A typical pipeline routes user input through modality-specific encoders, feeds serialized tokens into a central reasoning model, and dispatches structured outputs to downstream tools. This pattern demands an inference backend that exposes chat, vision, audio, embeddings, and image generation through a single SDK.

Oxlo.ai provides exactly this topology. With 45+ open-source and proprietary models across seven categories, you can route text to Llama 3.3 70B, vision to Gemma 3 27B or Kimi VL A3B, audio to Whisper Large v3, and image generation to Flux.1 or Oxlo.ai Image Pro, all behind one base URL. Because the platform is fully OpenAI SDK compatible, you can point your existing client to https://api.oxlo.ai/v1 and begin multimodal orchestration without rewriting request logic.

Vision Understanding with LLMs

Vision-capable LLMs accept base64-encoded images or HTTP URLs alongside text prompts. The model projects image patches into the language model's embedding space, then performs standard autoregressive reasoning. In practice, a single high-resolution screenshot can add thousands of tokens to your context window.

This is where token-based billing becomes unpredictable. On Oxlo.ai, vision requests are billed per API call, not per image token, so the cost of analyzing a detailed UI mockup or a multi-page scan remains flat regardless of resolution. You can experiment with Gemma 3 27B for on-device-quality vision tasks, or Kimi VL A3B for more complex document understanding.

The following Python example sends an image and a text prompt through Oxlo.ai using the OpenAI SDK:

from openai import OpenAI
import base64

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

b64_image = encode_image("diagram.png")

response = client.chat.completions.create(
    model="gemma-3-27b-it",  # or kimi-vl-a3b
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract all labeled values from this diagram and return them as a JSON object."},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_image}"}}
            ]
        }
    ],
    response_format={"type": "json_object"}
)

print(response.choices[0].message.content)

Audio Transcription and Speech Synthesis

Audio is another high-entropy modality. A five-minute voice memo can produce several thousand tokens of transcript, and voice agents often require multiple model passes: speech-to-text, reasoning, then text-to-speech.

Oxlo.ai exposes dedicated endpoints for both directions. You can transcribe audio with Whisper Large v3, Whisper Turbo, or Whisper Medium through the audio/transcriptions endpoint, then feed the resulting text into a reasoning model such as DeepSeek V3.2 or Qwen 3 32B. For outbound voice, the audio/speech endpoint serves Kokoro 82M for low-latency text-to-speech.

Because Oxlo.ai charges per request, transcribing a long meeting recording costs the same flat rate as a short command, a structural advantage over token-based providers for speech-heavy workflows.

Image Generation as a Modal Response

Multimodal agents do not only consume media; they produce it. A common pattern is to let an LLM decide when to invoke an image generation tool based on user intent, then return the generated asset URL in a structured response.

Oxlo.ai supports this through the images/generations endpoint, with models including Flux.1, Stable Diffusion 3.5, SDXL, and Oxlo.ai Image Pro. Combined with function calling, you can build agents that reason about visual content and then generate new images without leaving the Oxlo.ai ecosystem.

Here is a minimal agent loop that classifies intent and generates an image when requested: