Robotics stacks have historically relied on deterministic sense-plan-act pipelines. Adding an LLM layer lets operators issue natural language commands while preserving existing ROS2 or custom middleware investments. The challenge is not choosing between autonomy and legacy systems, but bridging them safely through strict interfaces.
Architecture: The LLM as a Cognitive Layer
Most production robots run ROS2, DDS, or proprietary middleware. The LLM should act as a cognitive layer above existing planners and controllers, not a replacement. A typical integration places the LLM behind a gateway node that translates natural language into structured commands your existing stack already understands.
Oxlo.ai supports this pattern through full OpenAI SDK compatibility. You can point your robotics Python client to https://api.oxlo.ai/v1 and use function calling to emit schemas that your motion planner or manipulation pipeline consumes.
Step 1. Select a Model for Embodied Reasoning
Not every robotics task needs the largest model. For high-frequency intent parsing, a fast model keeps latency low. For long-horizon task planning, reasoning models with tool-use support are more appropriate.
- Agentic planning: Qwen 3 32B on Oxlo.ai handles multilingual reasoning and agent workflows, which is useful when robots operate across languages or need multi-step tool chains.
- Deep reasoning: DeepSeek R1 671B MoE or Kimi K2.6 excel at complex coding and reasoning, ideal for manipulation primitives generated from natural language.
- General orchestration: Llama 3.3 70B provides a balanced flagship for mixed workloads.
- Vision grounding: Gemma 3 27B or Kimi VL A3B process image inputs from robot cameras to inform the LLM scene description.
Because Oxlo.ai uses request-based pricing, long-context prompts that include full sensor logs or lengthy tool histories do not inflate cost the way token-based billing would. This matters for agentic robotics, where context windows grow quickly. See Oxlo.ai pricing for plan details.
Step 2. Configure the OpenAI SDK for Oxlo.ai
Since Oxlo.ai is a drop-in replacement for the OpenAI API, integration requires only a base URL and API key change.
import openai
import os
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ["OXLO_API_KEY"]
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Navigate to the kitchen and list obstacles."}],
tools=[...] # defined in Step 3
)Step 3. Define Robot Tools with Function Calling
The safest way to connect an LLM to hardware is through strict function schemas. Your robot exposes capabilities as tools, and the LLM decides which to invoke. Oxlo.ai supports function calling on its chat completions endpoint, so you can define motion primitives, sensor queries, and manipulation routines as JSON schemas.
tools = [
{
"type": "function",
"function": {
"name": "navigate_to",
"description": "Send a navigation goal to the ROS2 nav2 stack.",
"parameters": {
"type": "object",
"properties": {
"x": {"type": "number"},
"y": {"type": "number"},
"frame_id": {"type": "string", "default": "map"}
},
"required": ["x", "y"]
}
}
},
{
"type": "function",
"function": {
"name": "get_camera_frame",
"description": "Capture the latest RGB image from the front camera.",
"parameters": {"type": "object", "properties": {}}
}
}
]When the model returns a tool call, your gateway node validates arguments against physical safety bounds, then executes the command on the real hardware.
Step 4. Build the ROS2 Gateway Node
The gateway node sits between the LLM client and your existing ROS2 graph. It subscribes to human commands, calls Oxlo.ai, parses tool calls, and publishes to your existing topics.
import rclpy
from rclpy.node import Node
from std_msgs.msg import String
from geometry_msgs.msg import PoseStamped
import openai
import json
class LLMGateway(Node):
def __init__(self):
super().__init__("llm_gateway")
self.cmd_sub = self.create_subscription(String, "/human_command", self.on_cmd, 10)
self.nav_pub = self.create_publisher(PoseStamped, "/goal_pose", 10)
self.client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ["OXLO_API_KEY"]
)
def on_cmd(self, msg: String):
completion = self.client.chat.completions.create(
model="qwen3-32b",
messages=[{"role": "user", "content": msg.data}],
tools=tools,
tool_choice="auto"
)
choice = completion.choices[0]
if choice.message.tool_calls:
for tc in choice.message.tool_calls:
if tc.function.name == "navigate_to":
args = json.loads(tc.function.arguments)
self.publish_nav_goal(args)
def publish_nav_goal(self, args):
pose = PoseStamped()
pose.header.frame_id = args.get("frame_id", "map")
pose.pose.position.x = args["x"]
pose.pose.position.y = args["y"]
self.nav_pub.publish(pose)
def main():
rclpy.init()
node = LLMGateway()
rclpy.spin(node)Step 5. Add Vision for Perceptive Agents
Robots need visual grounding as well as language. Oxlo.ai hosts vision models such as Gemma 3 27B and Kimi VL A3B that accept image inputs through the same chat completions endpoint. You can base64-encode a camera frame and include it in the messages array.
import base64
def encode_image(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "What objects on this table are safe to grasp?"},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encode_image('frame.jpg')}"}}
]
}
]
response = client.chat.completions.create(
model="gemma-3-27b-it",
messages=messages
)The LLM returns structured descriptions that downstream nodes convert into grasp poses or collision objects.
Step 6. Manage Multi-Turn State and Context
Robotics tasks unfold over time. Maintain a conversation history so the LLM retains spatial context, previous actions, and corrections from human supervisors. Oxlo.ai supports multi-turn conversations natively, and because pricing is per request rather than per token, you can keep rich histories without worrying about hidden costs as context length grows.
For long-horizon autonomy, consider models like GLM 5 or Minimax M2.5, which are designed for agentic tool use and extended reasoning sequences.
Step 7. Deploy for Production
In production, run the gateway node on your robot's compute or an edge server with low-latency access to sensors. Oxlo.ai provides no cold starts on popular models, so your robot receives consistent response times even after idle periods. If you need guaranteed throughput, the Enterprise tier offers dedicated GPU capacity and custom request limits.
Monitor tool call accuracy, hallucination rates, and safety boundary violations. Keep the human-in-the-loop path open via the same messaging interface, so operators can override ambiguous commands before they reach hardware.
Conclusion
Integrating an LLM into an existing robotics stack does not require replacing your motion planners or sensor drivers. By using Oxlo.ai as the inference backend, you gain OpenAI SDK compatibility, a broad model catalog spanning reasoning, vision, and code, and request-based pricing that remains predictable as your agents accumulate context. Start with a single ROS2 gateway node, define your hardware capabilities as function schemas, and expand incrementally.
Top comments (0)