Using LLM for Image Captioning: Challenges and Applications

Image captioning powers accessibility tools, asset management pipelines, and visual search indexes. In this tutorial we will build a small Python utility that reads a local image and returns a structured caption using a vision-capable LLM hosted on Oxlo.ai. The final script is what I use to catalog screenshots and product photos without any manual tagging.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK: pip install openai
• A sample image file named sample.jpg in your working directory

Step 1: Configure the Oxlo.ai client

Set up the OpenAI SDK to point at Oxlo.ai. In production I read the key from an environment variable, but for clarity the snippet below shows the direct assignment.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 2: Load and encode an image

Oxlo.ai vision models accept base64-encoded images inside the chat messages payload. This helper reads a local file and returns the data URI string.

import base64

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_path = "sample.jpg"
base64_image = encode_image(image_path)

Step 3: Write the captioning prompt

A strong system prompt keeps descriptions consistent and prevents hallucinated details. I treat it as a constant I can tune without touching the rest of the script.

SYSTEM_PROMPT = """You are an image captioning assistant. Describe the image in one concise paragraph. Mention the main subject, setting, colors, and any visible text. Do not guess at information that is not visible. Keep the tone neutral and factual."""

Step 4: Generate the first caption

We call kimi-k2.6 because it handles vision and reasoning well on Oxlo.ai. The user message carries both the instruction and the base64 image block.

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Generate a caption for this image."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    },
                },
            ],
        },
    ],
)

caption = response.choices[0].message.content
print(caption)

Step 5: Enforce structured JSON output

Raw text is useful, but downstream pipelines need machine-readable fields. Oxlo.ai supports JSON mode, so we can request an object with keys like subject, setting, and keywords.

import json

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Generate a caption and return valid JSON with keys: subject, setting, colors, visible_text, keywords."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    },
                },
            ],
        },
    ],
    response_format={"type": "json_object"},
)

structured = json.loads(response.choices[0].message.content)
print(json.dumps(structured, indent=2))

Step 6: Batch process a folder

The real value comes from running this over many files. This loop processes every .jpg in ./images/ and writes a single catalog file.

import glob

catalog = []

for path in glob.glob("images/*.jpg"):
    b64 = encode_image(path)
    resp = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Return JSON with keys: filename, subject, setting, colors, visible_text, keywords."},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{b64}"
                        },
                    },
                ],
            },
        ],
        response_format={"type": "json_object"},
    )
    entry = json.loads(resp.choices[0].message.content)
    entry["filename"] = path
    catalog.append(entry)

with open("catalog.json", "w") as f:
    json.dump(catalog, f, indent=2)

print(f"Processed {len(catalog)} images. Output written to catalog.json")

Run it

Save the script as caption.py, place a few images in ./images/, set your key, and run:

export OXLO_API_KEY="sk-..."
python caption.py

For a photo of a city street at dusk, the output looks something like this:

{
  "subject": "A row of brick storefronts with outdoor seating",
  "setting": "Urban street corner during blue hour",
  "colors": "Warm amber from windows, cool slate sky, red brick",
  "visible_text": "Open sign in neon, street number 442",
  "keywords": ["street photography", "evening", "cafes", "city life"]
}

Wrap-up

You now have a working image captioning pipeline that runs entirely on Oxlo.ai. Because the platform uses flat per-request pricing, batching hundreds of long-context vision requests costs the same per call regardless of prompt length. Check the details at https://oxlo.ai/pricing. Two obvious next steps are wiring this into a CMS upload webhook so assets are tagged on arrival, or switching to gemma-3-27b if you need faster throughput for simpler catalog images.