Vision and LLM Integration: A New Frontier

#learnai #oxlo #ai

We're going to build a Visual Expense Auditor that ingests receipt images and returns structured JSON with line items, totals, and flagged anomalies. This cuts manual data entry for finance teams and personal budgeting apps. By the end you will have a working agent powered by Oxlo.ai's vision-capable models.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai
A sample receipt image in JPEG or PNG format

Step 1: Configure the Oxlo.ai client

I start by importing the OpenAI SDK and pointing it at Oxlo.ai. The base URL and API key are the only changes needed to use Oxlo.ai as a drop-in replacement.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

# Verify connectivity with a simple non-vision ping
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Say OK"}],
)
print(response.choices[0].message.content)

Step 2: Write the system prompt

The system prompt tells the model to act as a structured extractor and anomaly detector. I require pure JSON so downstream code can parse without regex.

SYSTEM_PROMPT = """You are a Visual Expense Auditor. Analyze the provided receipt image and extract the following fields in JSON format:
- merchant: string
- date: string in ISO 8601 format
- line_items: array of objects with keys description, quantity, unit_price, total
- pre_tax_total: number
- tax_amount: number
- grand_total: number
- currency: string ISO code

After extraction, audit for anomalies:
- math errors where sum of line_items does not equal pre_tax_total
- duplicate line items
- suspiciously round or inflated totals

Return a JSON object with keys: receipt, anomalies (array of strings), and audit_passed (boolean).
Do not return markdown or explanations outside the JSON."""

Step 3: Prepare the image payload

Oxlo.ai accepts base64-encoded images inline using the standard OpenAI image_url format. I load the file, encode it, and build the user message.

import base64

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def build_messages(image_path):
    b64 = encode_image(image_path)
    data_url = f"data:image/jpeg;base64,{b64}"
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Audit this receipt and return structured JSON."},
                {"type": "image_url", "image_url": {"url": data_url}},
            ],
        },
    ]

Step 4: Call the vision model

I use kimi-k2.6 on Oxlo.ai because it handles both vision and long-context reasoning. I set response_format to JSON mode to enforce valid output.

import json

def audit_receipt(image_path):
    messages = build_messages(image_path)
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=messages,
        response_format={"type": "json_object"},
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 5: Parse and audit the result

The parsed JSON contains the extracted receipt data and any anomalies. I print a formatted summary so the user can see what the model found without reading raw JSON.

def print_audit(report):
    receipt = report.get("receipt", {})
    anomalies = report.get("anomalies", [])
    passed = report.get("audit_passed", False)

    print(f"Merchant: {receipt.get('merchant')}")
    print(f"Date: {receipt.get('date')}")
    print(f"Grand Total: {receipt.get('grand_total')} {receipt.get('currency')}")
    print(f"Line Items: {len(receipt.get('line_items', []))}")
    print(f"Audit Passed: {passed}")
    if anomalies:
        print("Flagged Anomalies:")
        for a in anomalies:
            print(f"  - {a}")
    else:
        print("No anomalies detected.")

if __name__ == "__main__":
    result = audit_receipt("receipt.jpg")
    print_audit(result)

Run it

Save the script as auditor.py, set your API key, and run it against a sample receipt. Here is what the output looks like for a typical restaurant receipt.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python auditor.py

Merchant: Blue Bottle Coffee
Date: 2024-05-17
Grand Total: 24.75 USD
Line Items: 3
Audit Passed: False
Flagged Anomalies:
  - Sum of line items (22.50) does not match pre_tax_total (21.50)
  - Duplicate entry detected for Croissant

Wrap-up

This agent replaces a tedious manual workflow with a single API call to Oxlo.ai. Because Oxlo.ai uses flat per-request pricing, the cost stays predictable even when you feed it long receipts or multi-page invoice scans.

Next, wire this into a FastAPI endpoint so users can POST images from a mobile app. Alternatively, batch-process a folder of receipts and dump the results into a CSV for accounting.