LLM Models for Multimodal Analysis

We are building a multimodal document extractor on Oxlo.ai that reads images of invoices or forms and returns structured JSON. This helps finance and operations teams automate data entry without maintaining separate OCR pipelines.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK: pip install openai
• A sample image such as a receipt, invoice, or form saved as PNG or JPEG

Step 1: Configure the Oxlo.ai client

I start every project by instantiating the client. Oxlo.ai exposes a fully OpenAI-compatible endpoint, so the only difference is the base URL.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 2: Write the system prompt

The system prompt is the contract between me and the model. I keep it strict because I want deterministic JSON back, not conversational filler.

SYSTEM_PROMPT = """You are a precise document extraction engine. Analyze the provided image and extract all visible structured data.

Rules:
- Identify the document type (invoice, receipt, form, or other).
- Extract fields such as vendor, date, total amount, line items, tax, and currency.
- Return ONLY a JSON object. No markdown, no explanations.
- If a field is missing, use null.
- Use camelCase for keys.

Example structure:
{
  "documentType": "invoice",
  "vendor": "Acme Corp",
  "date": "2024-01-15",
  "currency": "USD",
  "total": 123.45,
  "tax": 8.50,
  "lineItems": [
    {"description": "Widget A", "quantity": 2, "unitPrice": 50.00}
  ]
}
"""

Step 3: Encode the image

Oxlo.ai accepts base64 data URIs in the same format as the OpenAI vision API. I wrote a small helper to read a local file and build the message payload.

import base64

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def build_messages(image_path, text_query):
    b64 = encode_image(image_path)
    data_uri = f"data:image/jpeg;base64,{b64}"
    
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": text_query},
                {"type": "image_url", "image_url": {"url": data_uri}}
            ]
        }
    ]

Step 4: Call the vision model

I use Kimi K2.6 on Oxlo.ai because it handles vision, reasoning, and long context in a single call. I also enable JSON mode to guarantee parseable output.

def extract_document(image_path, query="Extract all data from this document."):
    messages = build_messages(image_path, query)
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=messages,
        response_format={"type": "json_object"},
        max_tokens=2048
    )
    
    return response.choices[0].message.content

# Example usage
result = extract_document("invoice_sample.jpg")
print(result)

Run it

Here is the complete script. I keep the API key in an environment variable and accept the image path from the command line.

import os
import sys
import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

SYSTEM_PROMPT = """You are a precise document extraction engine. Analyze the provided image and extract all visible structured data.

Rules:
- Identify the document type (invoice, receipt, form, or other).
- Extract fields such as vendor, date, total amount, line items, tax, and currency.
- Return ONLY a JSON object. No markdown, no explanations.
- If a field is missing, use null.
- Use camelCase for keys.
"""

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def extract(image_path, query="Extract all data from this document."):
    b64 = encode_image(image_path)
    data_uri = f"data:image/jpeg;base64,{b64}"
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": query},
                    {"type": "image_url", "image_url": {"url": data_uri}}
                ]
            }
        ],
        response_format={"type": "json_object"},
        max_tokens=2048
    )
    
    return response.choices[0].message.content

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python extractor.py ")
        sys.exit(1)
    
    print(extract(sys.argv[1]))

Running it against a typical SaaS invoice gives me output like this:

{
  "documentType": "invoice",
  "vendor": "CloudHosting Inc",
  "date": "2025-03-12",
  "currency": "USD",
  "total": 249.00,
  "tax": 20.75,
  "lineItems": [
    {"description": "Dedicated Server - Tier 2", "quantity": 1, "unitPrice": 199.00},
    {"description": "Bandwidth Overage", "quantity": 50, "unitPrice": 1.00}
  ]
}

Wrap up

Two concrete next steps. First, wrap this in a FastAPI endpoint so other services can POST images and receive JSON. Second, batch-process an entire directory. Because Oxlo.ai uses request-based pricing (see https://oxlo.ai/pricing), the cost stays flat even when you add detailed system prompts and high-resolution images to each call.