Using LLMs for Image Captioning: A Step-by-Step Guide

#learnai #oxlo #ai

In this guide we will build a lightweight image captioning tool that takes any local image and returns a plain-text description. It is useful for automating alt-text generation, content tagging, or building searchable media libraries. We will run everything against Oxlo.ai using the OpenAI-compatible SDK so you can plug it into an existing pipeline without changes.

What you'll need

Python 3.10 or newer
The OpenAI Python SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A local image file to test with, such as a .jpg or .png

Step 1: Configure the Oxlo.ai client

Because Oxlo.ai is fully OpenAI API compatible, the setup is a drop-in replacement. Point the base URL to Oxlo.ai and load your key.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

print("Oxlo.ai client initialized")

Step 2: Encode the image

Vision models expect a base64 data URL. This helper reads a local file and returns the encoded string with the proper prefix.

import base64

def encode_image(image_path):
    with open(image_path, "rb") as f:
        encoded = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/jpeg;base64,{encoded}"

image_path = "sample.jpg"
base64_image = encode_image(image_path)

Step 3: Write the system prompt

A strict system prompt keeps captions consistent and prevents the model from adding markdown or commentary.

SYSTEM_PROMPT = (
    "You are an image captioning assistant. "
    "Describe the image in one concise sentence. "
    "Focus on the main subject, setting, and any relevant details. "
    "Do not add formatting, bullet points, or preamble."
)

Step 4: Build the captioning function

Oxlo.ai carries several vision models, including Gemma 3 27B and Kimi VL A3B. I use Kimi K2.6 here because it handles vision and reasoning in a single call. The user message contains both the text instruction and the base64 image.

import os
import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

SYSTEM_PROMPT = (
    "You are an image captioning assistant. "
    "Describe the image in one concise sentence. "
    "Focus on the main subject, setting, and any relevant details. "
    "Do not add formatting, bullet points, or preamble."
)

def encode_image(image_path):
    with open(image_path, "rb") as f:
        encoded = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/jpeg;base64,{encoded}"

def caption_image(image_path):
    base64_image = encode_image(image_path)
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Write a caption for this image."},
                    {"type": "image_url", "image_url": {"url": base64_image}},
                ],
            },
        ],
    )
    return response.choices[0].message.content.strip()

Run it

Save the script as caption.py, set your API key, and call the function from the command line.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python -c "from caption import caption_image; print(caption_image('sample.jpg'))"

A golden retriever sits on a sunny park bench next to a red frisbee.

Wrap-up

You now have a working image captioner backed by Oxlo.ai. Two concrete next steps: extend the script to walk an entire directory and write captions to a JSONL file, or switch the response format to JSON mode so the model returns structured tags alongside the description. If you are processing large volumes, Oxlo.ai request-based pricing keeps costs flat per call regardless of image size or prompt length. See https://oxlo.ai/pricing for details.

DEV Community