We are going to build a natural-language robot controller that plans pick-and-place actions in a simulated 2D warehouse. This is for engineers who want to see how an LLM can replace hand-coded state machines for simple embodied tasks without managing physical hardware. We will run the planner against Oxlo.ai so you can iterate on long state descriptions without token costs creeping up, and you can explore the exact pricing at https://oxlo.ai/pricing.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Because Oxlo.ai is fully OpenAI SDK compatible, we only need to change the base_url and plug in your Oxlo.ai key.
Step 1: Set up the environment and Oxlo.ai client
I like to keep the world model minimal but structured enough that the LLM can reason about coordinates without extra tooling. We start with the OpenAI-compatible client and a small grid world.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
class Warehouse:
def __init__(self):
self.robot_pos = (0, 0)
self.objects = {"red_box": (2, 2), "blue_box": (1, 3)}
self.zones = {"green_zone": (3, 3), "yellow_zone": (0, 3)}
self.held = None
self.grid_size = 4
def serialize(self):
lines = [f"Robot at {self.robot_pos}"]
for name, pos in self.objects.items():
lines.append(f"{name} at {pos}")
for name, pos in self.zones.items():
lines.append(f"{name} at {pos}")
lines.append(f"Holding: {self.held}")
return "\n".join(lines)
def is_valid(self, pos):
x, y = pos
return 0 <= x < self.grid_size and 0 <= y < self.grid_sizeStep 2: Define the action space and state serializer
Next we lock down the vocabulary the LLM is allowed to use. I expose a single format_state helper so the prompt stays consistent no matter how much world detail I add later.
VALID_ACTIONS = [
"move_north", "move_south", "move_east", "move_west",
"pick", "place"
]
def format_state(wh):
return wh.serialize()Step 3: Write the system prompt
I keep the prompt strict. Robotics controllers fail silently if the LLM drifts into markdown or adds commentary, so I explicitly forbid it. Here is the exact system prompt we will use.
SYSTEM_PROMPT = """
You are a warehouse robot controller. You receive the current world state and a user command.
Respond with a JSON object containing a single key "plan", which is a list of actions.
Valid actions: move_north, move_south, move_east, move_west, pick, place.
Rules:
- You may only pick an object if the robot is at the same coordinate.
- You may only place if holding an object.
- Do not invent objects or zones not present in the state.
- Output only the JSON. No markdown, no explanation.
"""Step 4: Build the planner loop
Now we ask the LLM to generate the plan. I use Llama 3.3 70B on Oxlo.ai because it follows structured instructions reliably for agent workflows like this.
import json
def plan_task(command, state_text):
user_message = f"Command: {command}\nCurrent state:\n{state_text}"
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
raw = response.choices[0].message.content
parsed = json.loads(raw)
return parsed["plan"]Step 5: Add safety validation
Before we trust the LLM to move our simulated robot, we validate every action against the current world state. This catches hallucinations like moving through walls or picking objects that are across the room.
def is_safe(wh, action):
x, y = wh.robot_pos
if action == "move_north":
return wh.is_valid((x, y + 1))
if action == "move_south":
return wh.is_valid((x, y - 1))
if action == "move_east":
return wh.is_valid((x + 1, y))
if action == "move_west":
return wh.is_valid((x - 1, y))
if action == "pick":
return any(pos == wh.robot_pos for pos in wh.objects.values())
if action == "place":
return wh.held is not None
return False
def execute_action(wh, action):
if action.startswith("move_"):
x, y = wh.robot_pos
if action == "move_north":
wh.robot_pos = (x, y + 1)
elif action == "move_south":
wh.robot_pos = (x, y - 1)
elif action == "move_east":
wh.robot_pos = (x + 1, y)
elif action == "move_west":
wh.robot_pos = (x - 1, y)
return f"Moved to {wh.robot_pos}"
if action == "pick":
for name, pos in list(wh.objects.items()):
if pos == wh.robot_pos:
wh.held = name
del wh.objects[name]
return f"Picked {name}"
if action == "place":
wh.objects[wh.held] = wh.robot_pos
name = wh.held
wh.held = None
return f"Placed {name}"
return "Unknown action"Run it
We can now give the robot a high-level command and watch it plan and execute. Because Oxlo.ai does not charge by the token, you can freely expand the world description or add sensor logs without rethinking the budget on every run.
if __name__ == "__main__":
wh = Warehouse()
print("Initial state:")
print(format_state(wh))
command = "move the red box to the green zone"
plan = plan_task(command, format_state(wh))
print(f"\nGenerated plan: {plan}\n")
for action in plan:
if not is_safe(wh, action):
print(f"Blocked unsafe action: {action}")
continue
result = execute_action(wh, action)
print(f"Action: {action} -> {result}")
print("\nFinal state:")
print(format_state(wh))Example output:
Initial state:
Robot at (0, 0)
red_box at (2, 2)
blue_box at (1, 3)
green_zone at (3, 3)
yellow_zone at (0, 3)
Holding: None
Generated plan: ['move_east', 'move_east', 'move_north', 'move_north', 'pick', 'move_east', 'move_north', 'place']
Action: move_east -> Moved to (1, 0)
Action: move_east -> Moved to (2, 0)
Action: move_north -> Moved to (2, 1)
Action: move_north -> Moved to (2, 2)
Action: pick -> Picked red_box
Action: move_east -> Moved to (3, 2)
Action: move_north -> Moved to (3, 3)
Action: place -> Placed red_box
Final state:
Robot at (3, 3)
blue_box at (1, 3)
red_box at (3, 3)
green_zone at (3, 3)
yellow_zone at (0, 3)
Holding: NoneWrap-up and next steps
That is a working LLM robotics controller. The agent translates natural language into a structured action plan, and our validator keeps the simulation from drifting into impossible states.
Two concrete next steps. First, swap Llama 3.3 70B for Qwen 3 32B on Oxlo.ai if you want stronger multilingual reasoning for voice or mixed-language commands. Second, replace the text state with a vision pipeline using Kimi VL A3B so the agent reasons over raw camera frames instead of hand-written coordinates.
Top comments (0)