Object Detection with LLMs: A Comprehensive Guide

We are building a vision agent that uses a large language model to detect objects in images and return normalized bounding boxes. Instead of training a custom model, we will prompt a vision-capable LLM from Oxlo.ai to describe what it sees in strict JSON format, then overlay the results onto the original image. This is useful for rapid prototyping, unusual object categories, or any scenario where setting up a dedicated CV pipeline is overkill.

What you'll need

• Python 3.10 or newer.
• An Oxlo.ai API key from https://portal.oxlo.ai. We will use the OpenAI-compatible endpoint at https://api.oxlo.ai/v1.
• The OpenAI SDK and Pillow: pip install openai pillow.
• A sample image file named sample.jpg in your working directory.

Step 1: Encode the image

Vision models need the image sent as a base64 data URL. The helper below reads any local JPEG or PNG and returns the base64 string.

import base64

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

Step 2: Lock in the system prompt

The model needs explicit instructions to act like a deterministic API. We force JSON output, normalized coordinates, and no extra chatter.

SYSTEM_PROMPT = """You are an object detection engine. Analyze the image and return a single JSON object with no markdown, no code fences, and no explanatory text.

Use this exact schema:
{
  "objects": [
    {
      "label": "descriptive name",
      "x1": 0.0,
      "y1": 0.0,
      "x2": 1.0,
      "y2": 1.0
    }
  ]
}

Coordinates must be normalized floats between 0.0 and 1.0 relative to image width and height. Only include clearly visible objects."""

Step 3: Call the vision model

We point the OpenAI client at Oxlo.ai and use kimi-k2.6 because it handles vision and structured reasoning well. The message payload mixes the image URL and a short text instruction.

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def detect_objects(image_path):
    b64 = encode_image(image_path)
    data_url = f"data:image/jpeg;base64,{b64}"
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": [
                {"type": "image_url", "image_url": {"url": data_url}},
                {"type": "text", "text": "Detect all objects in this image."}
            ]}
        ],
        max_tokens=4096,
    )
    
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Draw the boxes

We convert the normalized coordinates back to pixels and render rectangles with Pillow.

from PIL import Image, ImageDraw

def draw_boxes(image_path, detections, out_path="out.jpg"):
    img = Image.open(image_path)
    draw = ImageDraw.Draw(img)
    w, h = img.size
    
    for obj in detections.get("objects", []):
        x1 = int(obj["x1"] * w)
        y1 = int(obj["y1"] * h)
        x2 = int(obj["x2"] * w)
        y2 = int(obj["y2"] * h)
        draw.rectangle([x1, y1, x2, y2], outline="#FF0000", width=3)
        draw.text((x1, y1 - 10), obj["label"], fill="#FF0000")
    
    img.save(out_path)
    print(f"Saved annotated image to {out_path}")

Run it

Tie everything together in a short main block. The first time you run this, inspect the JSON in the terminal to verify the structure before trusting the drawing logic.

if __name__ == "__main__":
    IMAGE = "sample.jpg"
    
    results = detect_objects(IMAGE)
    print(json.dumps(results, indent=2))
    
    draw_boxes(IMAGE, results)

Example output:

{
  "objects": [
    {
      "label": "sedan",
      "x1": 0.12,
      "y1": 0.48,
      "x2": 0.44,
      "y2": 0.71
    },
    {
      "label": "pedestrian",
      "x1": 0.62,
      "y1": 0.41,
      "x2": 0.70,
      "y2": 0.83
    }
  ]
}

Next steps

Add a second pass that asks the model to return object attributes or confidence scores by expanding the JSON schema in the system prompt. When you outgrow LLM-based detection and need deterministic real-time inference, migrate to Oxlo.ai's native YOLOv9 and YOLOv11 models.