Introduction to LLM Robotics

#learnai #oxlo #ai

We are going to build a natural language robot command agent that converts plain English instructions into structured motion primitives. This is useful for robotics researchers and hardware hackers who want to prototype high-level task planning without writing brittle rule-based parsers. I run the LLM inference through Oxlo.ai because its flat per-request pricing stays predictable even when I stuff long sensor context into the prompt, and the OpenAI-compatible SDK means zero client code changes.

What you'll need

Python 3.10 or newer.
The OpenAI SDK installed with pip install openai.
An Oxlo.ai API key from https://portal.oxlo.ai. If you are building agentic loops that replay long state logs, Oxlo.ai request-based pricing removes the token-counting anxiety you get elsewhere. See https://oxlo.ai/pricing for details.

Step 1: Define the robot control schema

I started by locking down a strict JSON schema so the LLM cannot hallucinate invalid commands. My toy warehouse arm understands three primitives: move, pick, and place.

ROBOT_SCHEMA = {
    "type": "object",
    "properties": {
        "commands": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "action": {"enum": ["move", "pick", "place"]},
                    "target": {"type": "string"},
                    "x": {"type": "number"},
                    "y": {"type": "number"},
                    "z": {"type": "number"},
                    "description": {"type": "string"}
                },
                "required": ["action", "target", "x", "y", "z"]
            }
        }
    },
    "required": ["commands"]
}

Step 2: Write the system prompt

The system prompt is the only place the robot personality lives. I keep it concise and mechanical because creativity here leads to motion planning failures.

SYSTEM_PROMPT = """You are the high-level planner for a warehouse robot arm.
The user will describe a task in plain language.
You must break it down into a JSON object containing a "commands" array.
Each command must use one of these actions: move, pick, place.
Coordinates are in centimeters relative to the robot base.
Respond with valid JSON only. No markdown, no explanations outside the JSON."""

Step 3: Build the LLM client

I use the OpenAI SDK pointed at Oxlo.ai because it is a drop-in replacement. I also enable JSON mode so the model is constrained to valid output. Oxlo.ai serves Llama 3.3 70B with no cold starts, which keeps the planning loop snappy.

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def plan_task(instruction: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": instruction},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    return json.loads(response.choices[0].message.content)

Step 4: Parse and validate commands

Before touching any hardware, I validate the LLM output against the schema using basic Python. This catches structure drift early.

def validate_plan(plan: dict):
    if "commands" not in plan:
        raise ValueError("Missing 'commands' key")
    for cmd in plan["commands"]:
        assert cmd["action"] in ("move", "pick", "place"), f"Invalid action {cmd['action']}"
        assert all(k in cmd for k in ("target", "x", "y", "z")), "Missing coordinate or target"
    return plan

def generate_plan(instruction: str):
    raw = plan_task(instruction)
    return validate_plan(raw)

Step 5: Wrap it in a command interface

I added a thin CLI so I can iterate quickly in the terminal. In production this loop would run on the robot's edge computer.

if __name__ == "__main__":
    import sys

    if len(sys.argv) < 2:
        print("Usage: python robot_agent.py 'Pick up the red box and place it on shelf B'")
        sys.exit(1)

    user_instruction = sys.argv[1]
    plan = generate_plan(user_instruction)

    print("Generated plan:")
    for i, cmd in enumerate(plan["commands"], 1):
        print(f"  {i}. {cmd['action'].upper()} {cmd['target']} at ({cmd['x']}, {cmd['y']}, {cmd['z']})")

Run it

Here is a real run against Oxlo.ai with an instruction to move inventory between stations.

$ python robot_agent.py "Pick up the battery pack and place it on the charging dock"

Generated plan:
  1. MOVE battery pack at (45.0, 12.0, 5.0)
  2. PICK battery pack at (45.0, 12.0, 5.0)
  3. MOVE charging dock at (10.0, 80.0, 5.0)
  4. PLACE battery pack at (10.0, 80.0, 5.0)

Next steps

Wire the validated JSON into a ROS2 topic or a micro-controller serial interface so the commands become real motion. Or add a vision model such as Kimi K2.6 on Oxlo.ai to convert camera frames into scene descriptions, then feed that description into the planner prompt so the robot understands what it is looking at.

DEV Community