Introduction to Multimodal Reasoning Tutorial

#learnai #oxlo #ai

We are building a receipt auditor that ingests a photo, extracts line items, and validates the math with a vision-language model. It helps finance ops teams eliminate manual data entry without worrying about per-token costs scaling with image resolution. We will run the whole pipeline on Oxlo.ai using its flat per-request pricing and a vision-capable model.

What you'll need

An Oxlo.ai API key from https://portal.oxlo.ai. The Free plan includes 60 requests per day, which is enough to prototype.
Python 3.10 or newer.
The OpenAI SDK: pip install openai
A sample receipt image saved as receipt.jpg in your working directory.

Step 1: Set up the client and load the image

I start by importing the standard library modules and configuring the OpenAI SDK to point to Oxlo.ai. I also add a small helper that base64-encodes the receipt so we can pass it inline.

import base64
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("receipt.jpg")

Step 2: Write the system prompt

The system prompt tells the model to act as a meticulous auditor and to return strict JSON. Keeping this prompt in its own variable makes it easy to tweak later.

SYSTEM_PROMPT = '''You are a meticulous receipt auditor.
1. Read the attached receipt image carefully.
2. Extract the merchant name, date, and every line item with its price.
3. Compute the sum of all line items.
4. Compare your computed sum to the total printed on the receipt.
5. Return ONLY a JSON object with these exact keys:
   - merchant (string)
   - date (string)
   - line_items (list of objects, each with "item" and "price")
   - stated_total (number)
   - computed_total (number)
   - discrepancy_flag (boolean, true if computed_total != stated_total)
   - reasoning (string, one sentence explaining your confidence)
Do not wrap the JSON in markdown fences.'''

Step 3: Build the multimodal user message

OpenAI-compatible chat completions accept a list of content parts. We combine a text instruction with a base64 data URI so the model can see the image.

user_message = [
    {
        "type": "text",
        "text": "Audit the attached receipt and return the required JSON."
    },
    {
        "type": "image_url",
        "image_url": {
            "url": f"data:image/jpeg;base64,{image_b64}"
        }
    }
]

Step 4: Query the vision model

We call Oxlo.ai with Kimi K2.6, a model that supports vision, advanced reasoning, and a 131K context window. Because Oxlo.ai charges a flat rate per request, the size of the image does not change the cost.

import json

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    response_format={"type": "json_object"}
)

raw = response.choices[0].message.content
result = json.loads(raw)
print(json.dumps(result, indent=2))

Step 5: Validate the math locally

The model does the reading, but we still verify the arithmetic in code. This hybrid approach, multimodal perception plus deterministic validation, is what makes the agent reliable.

def validate_receipt(data):
    items = data.get("line_items", [])
    computed = sum(i["price"] for i in items)
    stated = data.get("stated_total", 0.0)
    data["computed_total"] = round(computed, 2)
    data["discrepancy_flag"] = abs(computed - stated) > 0.01
    return data

final = validate_receipt(result)
print("Validation passed:", not final["discrepancy_flag"])
print(json.dumps(final, indent=2))

Run it

Putting it all together inside a if __name__ == "__main__": block gives us a runnable script. Below is the full flow and an example output from a real receipt.

if __name__ == "__main__":
    image_b64 = encode_image("receipt.jpg")

    user_message = [
        {
            "type": "text",
            "text": "Audit the attached receipt and return the required JSON."
        },
        {
            "type": "image_url",
            "image_url": {
                "url": f"data:image/jpeg;base64,{image_b64}"
            }
        }
    ]

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"}
    )

    result = json.loads(response.choices[0].message.content)
    final = validate_receipt(result)

    print(json.dumps(final, indent=2))

Example output:

{
  "merchant": "Corner Coffee",
  "date": "2025-06-12",
  "line_items": [
    {"item": "Latte", "price": 4.50},
    {"item": "Bagel", "price": 3.25},
    {"item": "OJ", "price": 2.75}
  ],
  "stated_total": 10.50,
  "computed_total": 10.50,
  "discrepancy_flag": false,
  "reasoning": "All line items are clearly legible and the math checks out."
}

Next steps

Swap the image input for a PDF by converting pages to PNG, or wire this script into an email handler so finance teams can forward receipts directly. If you are processing hundreds of receipts, Oxlo.ai request-based pricing keeps the cost predictable regardless of image size or page count. See the details at https://oxlo.ai/pricing.

DEV Community