LLM for Image Recognition: A Comprehensive Guide

#learnai #oxlo #ai

We are going to build a lightweight image recognition pipeline that feeds a photo to a multimodal LLM and returns structured JSON describing objects, text, and scene context. It is useful for automated cataloging, content moderation, or accessibility alt-text generation. I am running this on Oxlo.ai because its flat per-request pricing does not punish me for sending large base64-encoded images, and its OpenAI-compatible API drops into existing code without friction.

What you'll need

Python 3.10 or newer
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A sample image file named sample.jpg in your working directory

Step 1: Set up the Oxlo.ai client

First I initialize the OpenAI SDK pointing at Oxlo.ai. I keep the key in an environment variable so it does not leak into source control.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY"),
)

Step 2: Prepare the image

The chat completions endpoint accepts an image as a base64 data URI. I wrote a small helper that reads any local file and returns the properly formatted string.

import base64

def encode_image(path):
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    ext = path.split(".")[-1].lower()
    mime = "image/jpeg" if ext in ("jpg", "jpeg") else f"image/{ext}"
    return f"data:{mime};base64,{b64}"

Step 3: Write the system prompt

To get predictable output, I lock the model into a strict JSON schema with a system prompt. I also tell it to skip markdown so I do not have to strip filler later.

SYSTEM_PROMPT = """You are a precise image recognition engine.
Analyze the image and return ONLY a JSON object with no markdown formatting.
Use this exact schema:
{
  "scene": "brief description of the overall scene",
  "objects": ["list", "of", "detected", "objects"],
  "text_in_image": "any visible text, or null",
  "mood": "overall mood or lighting",
  "confidence": "high|medium|low"
}
Be concise. Do not add commentary outside the JSON."""

Step 4: Build the recognition function

Now I wire the pieces together. I call Oxlo.ai's vision-capable kimi-k2.6 model, which handles the image and the system prompt in a single request. Because Oxlo.ai charges per request rather than per token, a high-resolution photo costs the same as a thumbnail. See https://oxlo.ai/pricing for current plan details.

import json

def recognize_image(image_path):
    b64_url = encode_image(image_path)

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": [
                {"type": "text", "text": "Analyze this image and return the JSON."},
                {"type": "image_url", "image_url": {"url": b64_url}}
            ]},
        ],
        max_tokens=1024,
    )

    raw = response.choices[0].message.content.strip()

    # Drop markdown fences if the model added them
    if raw.startswith("

```"):
        raw = raw.split("```

", 1)[1].rsplit("

```

", 1)[0].strip()

    return json.loads(raw)

Step 5: Parse and run

Finally I parse the response and print the result. If the JSON is valid, I can feed it straight into a database or downstream service.

if __name__ == "__main__":
    result = recognize_image("sample.jpg")
    print(json.dumps(result, indent=2))

Run it

With sample.jpg in the same folder, I run the script. The model returns structured fields I can use immediately.

$ python recognize.py
{
  "scene": "A cluttered desk with a laptop, coffee mug, and notebook near a window",
  "objects": [
    "laptop",
    "coffee mug",
    "notebook",
    "pen",
    "window",
    "desk"
  ],
  "text_in_image": "TODO - fix login bug",
  "mood": "warm natural light, casual workspace",
  "confidence": "high"
}

Next steps

That is the core pipeline. To take it further, wrap the recognize_image function in a FastAPI endpoint so you can POST images from a mobile app and get JSON back. Or batch-process a directory of photos and dump the results into a CSV to bootstrap a labeled dataset.

DEV Community