Unlocking Image Captioning with LLMs

Image captioning sits at the intersection of computer vision and language understanding. Feeding an image to a large language model and receiving a coherent, context-aware description is now a production requirement for content platforms, accessibility tools, and asset management pipelines. The challenge is no longer whether an LLM can describe an image, but how to do so at scale without costs that balloon with every pixel or token.

Why Image Captioning Needs More Than a Vanilla LLM

A text-only LLM cannot see. Production pipelines need a vision-language model that ingests images alongside prompts, handles high-resolution inputs, and returns structured data. Reliability matters: low-latency streaming, JSON mode for schema enforcement, and multi-turn conversation support let you build captioning systems that integrate cleanly into existing workflows. Cold starts break batch pipelines, so consistent inference latency is critical.

Vision Models Available on Oxlo.ai

Oxlo.ai offers vision-capable models through a single OpenAI-compatible endpoint. Gemma 3 27B delivers strong performance for general image captioning, while Kimi VL A3B provides efficient multimodal reasoning. If your pipeline combines vision with long-document context or agentic steps, Kimi K2.6 supports 131K tokens, advanced reasoning, and vision inputs. Switching between these models requires only changing the model parameter in your request.

Implementation with the OpenAI SDK

Because Oxlo.ai is fully OpenAI SDK compatible, you can use the same Python, Node.js, or cURL patterns you already know. The only change is the base URL.

import os
import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("photo.jpg")

response = client.chat.completions.create(
    model="gemma-3-27b",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Generate a concise, neutral caption describing the main subject and setting of this image."
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}
                }
            ]
        }
    ],
    max_tokens=256
)

caption = response.choices[0].message.content
print(caption)

Scaling Batch Workloads with Request-Based Pricing

On token-based platforms, images are converted into large token sequences before the prompt text is even counted. Captioning a high-resolution archive or a video frame sequence causes costs to grow unpredictably with input size. Oxlo.ai uses flat, request-based pricing: one fixed cost per API call regardless of image resolution or prompt length. For long-context and agentic workloads, this model can be significantly cheaper than token-based billing from providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale. You can verify current plans on the Oxlo.ai pricing page.

Enforcing Structure with JSON Mode

Raw text captions are hard to parse. Oxlo.ai supports JSON mode and function calling, so you can enforce schemas directly in the chat completions endpoint.

import os
import base64
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

image_b64 = base64.b64encode(open("photo.jpg", "rb").read()).decode("utf-8")

response = client.chat.completions.create(
    model="gemma-3-27b",
    messages=[
        {
            "role": "system",
            "content": "You are a captioning assistant. Respond with valid JSON containing keys: caption, tags, setting."
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}}
            ]
        }
    ],
    response_format={"type": "json_object"},
    max_tokens=512
)

structured = json.loads(response.choices[0].message.content)
print(structured)

From Prototype to Production

Oxlo.ai removes friction from the prototype stage through its free tier, which includes 60 requests per day and access to 16+ models, plus a 7-day full-access trial. Popular models have no cold starts, so batch jobs and user-facing apps get predictable latency. When volume grows, the Pro and Premium plans offer dedicated request pools and priority queue access. Because the platform is a drop-in replacement for the OpenAI SDK, migrating an existing captioning service takes minutes, not days.