Integrating LLM with Computer Vision and Robotics

We are going to build a warehouse robot vision planner that ingests a camera frame, reasons about obstacles and targets, and emits a structured JSON command. This is useful for any team prototyping autonomous mobile robots or pick-and-place arms without maintaining their own vision stack.

What you'll need

• Python 3.10 or newer
• The OpenAI SDK: pip install openai
• An Oxlo.ai API key from https://portal.oxlo.ai
• A test image file named warehouse.jpg in your working directory

Because Oxlo.ai uses request-based pricing, the cost stays flat even when you send high-resolution images with heavy context. See https://oxlo.ai/pricing for details.

Step 1: Configure the Oxlo.ai client

Create a client pointing to Oxlo.ai. I keep my key in an environment variable.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

Step 2: Encode the camera frame

The OpenAI-compatible vision API expects a base64 data URL. This helper wraps a local JPEG so you can test without hardware.

import base64
from pathlib import Path

def encode_image(image_path: str) -> str:
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"{image_path} not found. Add any JPEG from your camera.")
    with open(path, "rb") as f:
        encoded = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/jpeg;base64,{encoded}"

image_data_url = encode_image("warehouse.jpg")

Step 3: Write the robotics system prompt

I treat the LLM as a deterministic planner. The prompt locks the output to a strict JSON schema so downstream code never has to guess.

SYSTEM_PROMPT = """You are the vision planner for a mobile warehouse robot.
Analyze the camera image and decide the single next action.
Respond ONLY with a JSON object in this exact format:
{
  "action": "move_forward|turn_left|turn_right|stop|pick_object|report",
  "target": "string describing the object or location",
  "reason": "one sentence explaining your decision"
}
If you see an obstacle, choose stop or turn. If you see a target object, choose pick_object. If the path is clear, choose move_forward."""

Step 4: Send the frame to Kimi K2.6

Oxlo.ai hosts several vision-capable models. I use kimi-k2.6 here because it handles spatial reasoning and agentic coding. I also enable JSON mode so the model is constrained to valid output.

import json

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Current camera frame from front-facing sensor:"},
                {"type": "image_url", "image_url": {"url": image_data_url}}
            ]
        }
    ],
    response_format={"type": "json_object"}
)

plan = json.loads(response.choices[0].message.content)
print(plan)

Step 5: Parse and execute the command

In a real deployment this would publish to ROS2 or a motor controller. For now, I print the command so you can verify the logic before attaching hardware.

def execute_command(plan: dict):
    action = plan.get("action")
    target = plan.get("target", "none")
    reason = plan.get("reason", "no reason provided")

    if action == "stop":
        print(f"HALT: {reason}")
    elif action in ("move_forward", "turn_left", "turn_right"):
        print(f"ACTUATOR: {action} | Target: {target} | Reason: {reason}")
    elif action == "pick_object":
        print(f"ARM: grasping {target} | Reason: {reason}")
    elif action == "report":
        print(f"LOG: {reason} | Object: {target}")
    else:
        raise ValueError(f"Unknown action: {action}")

execute_command(plan)

Run it

Put the pieces together in a single script and point it at any JPEG from your robot or phone.

if __name__ == "__main__":
    image_data_url = encode_image("warehouse.jpg")
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Warehouse camera feed frame 042:"},
                    {"type": "image_url", "image_url": {"url": image_data_url}}
                ]
            }
        ],
        response_format={"type": "json_object"}
    )
    
    robot_plan = json.loads(response.choices[0].message.content)
    execute_command(robot_plan)

Example output with a cluttered aisle image:

ACTUATOR: turn_right | Target: aisle B shelving | Reason: The path ahead is blocked by a pallet, so I will turn toward the clear aisle.

Next steps

Wire this logic into a FastAPI endpoint that ingests an MJPEG stream from a physical camera, or wrap it in a ROS2 node so the planner publishes geometry_msgs/Twist commands directly to your robot base.

Because Oxlo.ai pricing is per request, you can send full-resolution frames with verbose system prompts and still predict your monthly bill. That makes it a strong fit for iterative robotics prototyping where image context changes but budget should not.