Integrating LLM with Computer Vision Tasks: A Step-by-Step Guide

We're building a two-stage visual analysis pipeline. It takes an image, extracts raw observations with a vision-language model, then structures those observations into strict JSON using a reasoning LLM. This is useful for anyone automating inventory checks, safety audits, or content metadata extraction.

What you'll need

You need Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key. Grab your key from https://portal.oxlo.ai. Install the SDK with pip install openai. You also need a test image, such as a photo of a desk or warehouse shelf.

Step 1: Set up the client and encode the image

First, I initialize the Oxlo.ai client and write a small helper to base64-encode a local image so we can send it through the chat completions API.

import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_path = "warehouse.jpg"
base64_image = encode_image(image_path)

Step 2: Extract visual observations

Now I send the image to Kimi K2.6, which accepts vision input. I ask for a plain text list of every visible object and its approximate location so the second model has clean material to work with.

vision_response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "List every distinct object you see in this image. For each item, note its approximate location (left, right, center, top, bottom) and estimated quantity. Be exhaustive and concise."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
            ]
        }
    ]
)

raw_observations = vision_response.choices[0].message.content
print(raw_observations)

Step 3: Define the system prompt

The raw text needs to be converted into structured JSON. I use a system prompt that defines the schema and instructs the model to emit only valid JSON. This keeps the output predictable and easy to parse downstream.

SYSTEM_PROMPT = """You are a structured data extractor. Convert the user's visual observations into a JSON object with this exact schema:

{
  "scene_type": "brief description of the overall setting",
  "items": [
    {
      "name": "object name",
      "count": 1,
      "location": "general area in the image"
    }
  ],
  "total_distinct_items": 0
}

Rules:
- Return only the JSON object, with no markdown fences or explanation.
- If the count is uncertain, use 1 and set a boolean field "estimated": true.
- Normalize object names to lowercase snake_case."""

Step 4: Structure the findings into JSON

I pass the raw observations to Llama 3.3 70B with JSON mode enabled. This gives me a parseable inventory object without regex hacks or post-processing.

structure_response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": raw_observations},
    ],
    response_format={"type": "json_object"}
)

structured_json = structure_response.choices[0].message.content
print(structured_json)

Step 5: Wrap both stages into one function

To make this reusable, I package both API calls into a single function. This hides the implementation details and returns a Python dict ready for a database or webhook.

import json

def analyze_image(image_path: str) -> dict:
    b64 = encode_image(image_path)
    
    # Stage 1: Vision
    v = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "List every distinct object you see in this image. For each item, note its approximate location and estimated quantity. Be exhaustive and concise."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
            ]
        }]
    )
    observations = v.choices[0].message.content
    
    # Stage 2: Structure
    s = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": observations},
        ],
        response_format={"type": "json_object"}
    )
    
    return json.loads(s.choices[0].message.content)

Run it

Call the function with any local image. The pipeline returns structured inventory data in under a few seconds because Oxlo.ai serves these models with no cold starts.

result = analyze_image("warehouse.jpg")
print(json.dumps(result, indent=2))

Example output:

{
  "scene_type": "warehouse storage shelf",
  "items": [
    {"name": "cardboard_box", "count": 12, "location": "center shelf"},
    {"name": "hard_hat", "count": 3, "location": "top left"},
    {"name": "fire_extinguisher", "count": 1, "location": "right side"},
    {"name": "pallet_jack", "count": 1, "location": "bottom center"}
  ],
  "total_distinct_items": 4
}

Next steps

Extend this by swapping in qwen-3-32b for multilingual observations, or route high-confidence counts through a second pass with deepseek-v3.2 to catch edge cases. If you are processing large batches, remember that Oxlo.ai charges a flat rate per request, so long image descriptions do not inflate your bill the way token-based providers do. See https://oxlo.ai/pricing for details.