Using LLM for Vision Tasks: A Comprehensive Guide

#learnai #oxlo #ai

We will build a vision analysis agent that ingests an image and returns structured JSON describing the scene, objects, visible text, and overall mood. This pattern replaces separate OCR and object-detection services with a single inference call, and on Oxlo.ai the cost stays flat regardless of image resolution or prompt length.

What you'll need

Python 3.10 or newer
The OpenAI SDK installed with pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A sample image file or a public image URL for testing

Step 1: Configure the client and encode images

I start by initializing the OpenAI-compatible client against Oxlo.ai and writing a helper that converts local files to base64 data URLs. This matches the OpenAI vision payload format exactly.

import base64
import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

Step 2: Define the system prompt

The system prompt is the schema contract. It locks the model to valid JSON and eliminates markdown wrappers, so downstream code only needs json.loads.

SYSTEM_PROMPT = """You are a vision analysis API.
Given an image, emit exactly one JSON object. No markdown, no preamble.
Schema:
{
  "scene": "one-sentence scene description",
  "objects": ["list of visible objects"],
  "has_text": true or false,
  "transcribed_text": "string, empty if none",
  "mood": "neutral|busy|calm|technical",
  "confidence": "high|medium|low"
}
Be literal. Only state what you can see."""

Step 3: Build the analysis function

This function accepts either a local file path or a public URL. I use kimi-k2.6 because it handles vision, advanced reasoning, and a 131K context window in one model. Because Oxlo.ai uses request-based pricing, sending a large base64 payload costs the same flat rate as a short text prompt.

def analyze_image(source: str, is_url: bool = False) -> dict:
    if is_url:
        image_url = source
    else:
        b64 = encode_image(source)
        image_url = f"data:image/jpeg;base64,{b64}"

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze this image."},
                    {"type": "image_url", "image_url": {"url": image_url}},
                ],
            },
        ],
        max_tokens=1024,
    )

    raw = response.choices[0].message.content.strip()
    # Strip accidental markdown fences
    if raw.startswith("

```"):
        raw = raw.split("\n", 1)[1].rsplit("```

", 1)[0].strip()
    return json.loads(raw)

Step 4: Add defensive validation

Production code should never trust raw LLM output. I wrap the parser in a small validator that checks for required keys and returns a safe error dict if anything drifts.

REQUIRED_KEYS = {"scene", "objects", "has_text", "transcribed_text", "mood", "confidence"}

def safe_analyze(source: str, is_url: bool = False) -> dict:
    try:
        result = analyze_image(source, is_url=is_url)
        missing = REQUIRED_KEYS - set(result.keys())
        if missing:
            return {"error": f"missing keys: {missing}", "raw": result}
        return result
    except Exception as e:
        return {"error": str(e), "source": source}

Run it

Test the agent against a public image URL so you do not need a local file to verify the pipeline.

if __name__ == "__main__":
    url = "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/300px-PNG_transparency_demonstration_1.png"
    output = safe_analyze(url, is_url=True)
    print(json.dumps(output, indent=2))

Example output:

{
  "scene": "A red apple sits in front of a checkered background demonstrating image transparency.",
  "objects": ["red apple", "checkered background"],
  "has_text": false,
  "transcribed_text": "",
  "mood": "technical",
  "confidence": "high"
}

Next steps

Swap the manual JSON scrubbing for a Pydantic model to get typed autocomplete across your codebase. If you are processing large image sets, remember that Oxlo.ai's flat per-request pricing means high-resolution base64 payloads and long system prompts do not inflate your bill the way token-based providers do. See https://oxlo.ai/pricing for plan details.

DEV Community