Using LLMs for Vision Tasks: A Guide

We are building a product photo analyzer that looks at an image and generates a structured e-commerce listing. It extracts title, category, condition, and visible defects from a single photo. This is useful for automating inventory intake, marketplace listings, or archival work.

What you'll need

• Python 3.10 or higher
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK: pip install openai
• A few product photos in JPEG or PNG format

Step 1: Set up the Oxlo.ai client

I keep my API key in an environment variable, but you can paste it directly for local testing. The Oxlo.ai client is a drop-in replacement for the OpenAI SDK.

from openai import OpenAI
import base64
import json
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Define the vision system prompt

The system prompt constrains the model to output strict JSON. This keeps downstream parsing reliable and consistent across different products.

SYSTEM_PROMPT = """You are a product photography analyst. Examine the provided image and generate a structured listing.

Return valid JSON with these exact keys:
- title: A concise, SEO-friendly product title (10 words max)
- description: Two sentences describing visible features and style
- category: One of Electronics, Clothing, Home, Sports, or Other
- condition: New, Like New, Good, or Fair based on visible wear
- defects: A JSON array of visible defects, or ["None detected"] if clean
- confidence: A float from 0.0 to 1.0 representing certainty

Do not wrap the JSON in markdown code blocks. Output raw JSON only."""

Step 3: Add base64 image encoding

Vision models need images as base64 data URLs. This helper reads a local file and returns the encoded string.

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        encoded = base64.b64encode(image_file.read()).decode("utf-8")
    return encoded

Step 4: Build the analysis function

I use kimi-k2.6 because it handles vision and structured reasoning well. The message payload mixes text and an image_url block. I also enable JSON mode to enforce valid output.

def analyze_product(image_path):
    b64_image = encode_image(image_path)
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Generate a structured listing for this product photo."
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{b64_image}"
                        }
                    }
                ]
            }
        ],
        response_format={"type": "json_object"}
    )
    
    return json.loads(response.choices[0].message.content)

Step 5: Batch process multiple photos

Running one photo is useful, but most workflows involve folders. This loop handles multiple files and writes everything to a single JSON report. Because Oxlo.ai charges a flat rate per request, your cost per image stays predictable even if you add detailed instructions to the prompt.

if __name__ == "__main__":
    photos = ["watch.jpg", "sneakers.jpg", "backpack.jpg"]
    catalog = []
    
    for photo in photos:
        if not os.path.exists(photo):
            print(f"Skipping {photo}: file not found")
            continue
            
        try:
            listing = analyze_product(photo)
            catalog.append({"file": photo, "data": listing})
            print(f"OK: {photo} -> {listing['title']}")
        except Exception as e:
            print(f"Error on {photo}: {e}")
    
    with open("catalog.json", "w") as f:
        json.dump(catalog, f, indent=2)
    
    print(f"\nWrote {len(catalog)} listings to catalog.json")

Run it

Save the script as analyze.py, place a few JPEGs in the same folder, and run:

export OXLO_API_KEY="your-key-here"
python analyze.py

You should see output like this:

OK: watch.jpg -> Men's Analog Stainless Steel Dress Watch
OK: sneakers.jpg -> White Leather Low-Top Basketball Sneakers
OK: backpack.jpg -> 28L Water-Resistant Hiking Backpack with Rain Cover

Wrote 3 listings to catalog.json

The resulting catalog.json will contain structured data:

[
  {
    "file": "watch.jpg",
    "data": {
      "title": "Men's Analog Stainless Steel Dress Watch",
      "description": "Classic round dial with date window and stainless steel bracelet. Minimalist design suitable for formal or casual wear.",
      "category": "Electronics",
      "condition": "Good",
      "defects": ["Minor scratch on crystal"],
      "confidence": 0.91
    }
  }
]

Wrap-up

This pipeline runs on Oxlo.ai with flat per-request pricing, so batch processing 100 photos does not get more expensive if your prompts grow. See the Oxlo.ai pricing page for plan details.

Two concrete next steps: wire the JSON output directly into a Shopify or eBay listing API, or extend the prompt to accept multiple angles of the same product in a single request for richer detail.