LLMs for Image Recognition: A Deep Dive

#learnai #oxlo #ai

We are building a CLI image tagger that sends photos to a vision-capable LLM and returns structured JSON metadata. It is useful for automating asset catalogs, content moderation, or alt-text generation without maintaining a separate computer-vision pipeline.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Set up the Oxlo.ai client

Oxlo.ai exposes an OpenAI-compatible endpoint, so the only changes from a standard OpenAI script are the base URL and the model name. I keep my key in an environment variable.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY"),
)

Step 2: Encode the image for the vision API

The chat completions endpoint expects images as base64 data URIs inside a multimodal message. This helper reads any local file and returns the correct payload block.

import base64
from pathlib import Path

def encode_image(image_path: str) -> str:
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"Image not found: {image_path}")
    ext = path.suffix.lower().replace(".", "")
    if ext == "jpg":
        ext = "jpeg"
    with open(path, "rb") as f:
        data = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/{ext};base64,{data}"

def build_user_message(image_path: str, question: str = "Describe this image in detail.") -> dict:
    b64_url = encode_image(image_path)
    return {
        "role": "user",
        "content": [
            {"type": "text", "text": question},
            {"type": "image_url", "image_url": {"url": b64_url}},
        ],
    }

Step 3: Lock the output with a system prompt

To make the result machine-readable, I force the model to reply with valid JSON and define the exact schema I want back. This removes the need for regex parsing.

SYSTEM_PROMPT = """You are a precise image recognition engine.
Analyze the provided image and respond with a single JSON object.
Do not include markdown fences, explanations, or line breaks outside the JSON.

Required schema:
{
  "scene_description": "string",
  "objects": ["list of visible objects"],
  "visible_text": "any text detected in the image, or empty string",
  "confidence_score": 0.0 to 1.0,
  "tags": ["relevant keyword tags"]
}
"""

Step 4: Call the vision model

I use Kimi K2.6 on Oxlo.ai because it handles vision, reasoning, and long context well. The request is standard OpenAI SDK, so the code works with any other Oxlo.ai vision model by changing the model string.

import json

def analyze_image(image_path: str) -> dict:
    user_msg = build_user_message(image_path)

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            user_msg,
        ],
        temperature=0.2,
        max_tokens=1024,
    )

    raw = response.choices[0].message.content.strip()
    # Some models may wrap JSON in markdown; strip it if present
    if raw.startswith("

```"):
        raw = raw.split("```

")[1]
        if raw.startswith("json"):
            raw = raw[4:]
    return json.loads(raw.strip())

Step 5: Wrap it in a CLI

Adding a small argument parser lets me point the tool at any image on disk and pretty-print the result.

if __name__ == "__main__":
    import sys

    if len(sys.argv) < 2:
        print("Usage: python image_tagger.py <path-to-image>")
        sys.exit(1)

    image_path = sys.argv[1]
    result = analyze_image(image_path)
    print(json.dumps(result, indent=2))

Run it

Save the full script as image_tagger.py, set your key, and pass a local image file.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python image_tagger.py warehouse_pallet.jpg
{
  "scene_description": "A plastic shipping pallet stacked with cardboard boxes in a warehouse aisle.",
  "objects": ["pallet", "cardboard boxes", "concrete floor", "fluorescent light"],
  "visible_text": "FRAGILE",
  "confidence_score": 0.92,
  "tags": ["warehouse", "logistics", "packaging", "indoor"]
}

Next steps

Feed a directory of images through the script in a loop and write the aggregated JSON to a CSV or database. Because Oxlo.ai charges a flat rate per request rather than per token, a detailed 4K image analysis costs the same as a short thumbnail description, which keeps pricing predictable at scale. See https://oxlo.ai/pricing for current plan details.

Alternatively, wrap analyze_image in a FastAPI endpoint so mobile or web clients can POST images and receive structured metadata in real time.

DEV Community