Deploying LLM Models on Edge Devices for Robotics

#aiinfrastructure #oxlo #ai

Robotics stacks increasingly rely on large language models for high-level planning, code generation, and human-robot interaction. Running these models directly on edge hardware is rarely practical. Jetson Orin modules and Intel NUCs have hard memory and thermal limits, and a 70B parameter model will not fit into 16 GB of shared GPU memory without aggressive quantization that destroys reasoning quality. A more robust pattern is to run perception and low-level control on the device while offloading LLM inference to a remote API. Oxlo.ai provides a developer-first inference platform with request-based pricing, OpenAI SDK compatibility, and no cold starts, making it a natural backend for edge robotics workloads that need predictable latency and cost.

The Edge Robotics Stack

A typical robotics deployment runs ROS 2 or a lightweight MQTT bus on an edge computer. This layer handles time-critical tasks: motor control, LiDAR processing, and safety interrupts. These loops must complete in milliseconds, so they cannot wait for a billion-parameter model to generate a response. Instead, the LLM acts as a task planner or failure-recovery advisor. It receives a structured state snapshot, optionally with recent camera frames, and returns a JSON action plan or generated Python script that the local controller executes.

Because edge hardware is constrained, even small vision-language models can consume all available resources. Oxlo.ai hosts larger vision and reasoning models, including Gemma 3 27B and Kimi VL A3B, so the edge device only needs to capture and transmit images rather than run inference locally.

Model Selection and Quantization

If you do need an on-device fallback for offline operation, you will usually quantize a small LLM down to Q4_K_M or INT8. Models like Qwen 3 4B or Llama 3.2 3B can run on a Jetson Orin Nano at acceptable throughput for simple intent classification. However, for anything involving multi-step reasoning, tool use, or code generation, the accuracy loss from quantization is noticeable. Oxlo.ai offers unquantized flagship models such as DeepSeek R1 671B MoE, GLM 5, and Qwen 3 32B, which retain full weights and therefore produce more reliable plans for manipulation and navigation tasks.

Oxlo.ai also supports function calling and JSON mode across its chat models. This lets a robot parse sensor data into a prompt, receive a strictly formatted tool call, and execute the corresponding ROS 2 action without fragile regex parsing on the edge side.

Hybrid Cloud-Edge Deployment

The most reliable production pattern is a split architecture. The edge agent maintains a local state machine and buffers telemetry. When a high-level decision is required, it assembles a prompt containing the current goal, recent logs, and base64-encoded images, then sends a single request to the cloud API. Streaming responses from Oxlo.ai let the robot begin parsing the plan before generation is complete, which reduces effective latency. Because Oxlo.ai has no cold starts on popular models, the first request after a quiet period still returns within a predictable window, a critical property for autonomous systems that cannot hang on model warmup.

Integrating Oxlo.ai from the Edge

The following Python snippet runs on an edge computer inside a ROS 2 node. It captures a workspace image, packs it into a multimodal prompt, and asks Qwen 3 32B on Oxlo.ai to return a JSON grasp plan. The code uses the standard OpenAI SDK, so integration requires only a base URL change.

import base64
import json
from openai import OpenAI

# Configure the client for Oxlo.ai
client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def plan_grasp(image_path: str, goal: str) -> dict:
    b64 = encode_image(image_path)

    response = client.chat.completions.create(
        model="qwen3-32b",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            f"Goal: {goal}\n\n"
                            "Analyze the workspace image and return a JSON object "
                            "with keys: 'objects' (list), 'grasp_order' (list), "
                            "and 'safety_notes' (string)."
                        )
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{b64}"}
                    }
                ]
            }
        ],
        response_format={"type": "json_object"},
        stream=False
    )

    return json.loads(response.choices[0].message.content)

# Example usage inside a ROS 2 callback
# plan = plan_grasp("/tmp/camera_frame.jpg", "Pick up the red cube and place it in the bin")

Because Oxlo.ai is fully OpenAI SDK compatible, you can reuse existing prompt engineering, retry logic, and telemetry middleware without rewriting your client. The request-based pricing model means this call costs the same whether the prompt contains 500 tokens or 50,000 tokens, because the price is tied to the API request, not the input length.

Cost Predictability with Request-Based Pricing

Robotics prompts are often token-heavy. A single planning request might include a system prompt, a 4K token state history, a 2K token error log, and a high-resolution image. On token-based providers, this long context drives up cost nonlinearly. For a fleet of robots making intermittent planning calls, that variability makes budgeting difficult.

Oxlo.ai uses flat per-request pricing. Each planning decision costs one request, regardless of how much sensor context you include. For long-context and agentic robotics workloads, this can be significantly cheaper than token-based billing. You can see the exact tiers on the Oxlo.ai pricing page. The free tier offers 60 requests per day, which is enough to prototype a single-robot pipeline before moving to a production plan.

Conclusion

Deploying LLMs on edge devices for robotics does not mean running everything locally. The winning architecture keeps control loops and perception on the device, and offloads heavy reasoning to a fast, compatible API. Oxlo.ai fits this pattern with its OpenAI SDK compatibility, no cold starts, and request-based pricing that stays predictable even when prompts grow. Whether you are prototyping with the free tier or running a fleet on the Premium plan, Oxlo.ai gives you access to 45+ models across reasoning, vision, and code without forcing you to manage infrastructure or count tokens.

DEV Community