shashank ms

Posted on Sep 24

Deploying LLMs for Edge Devices: A Step-by-Step Guide

#learnai #oxlo #ai

I recently shipped a telemetry assistant on a Raspberry Pi 4 at a remote pump station. The device cannot run a large model locally, so it sends sensor logs to Oxlo.ai and returns plain-text maintenance instructions to the operator. In this guide, I will walk through the exact code I used.

What you'll need

An Oxlo.ai API key from https://portal.oxlo.ai
Python 3.10 or newer
The OpenAI SDK: pip install openai
A Linux edge device (Raspberry Pi, Jetson, or x86 gateway)
A local telemetry file. I use a simple text file named sensors.log

Step 1: Scaffold the edge agent

Create a directory and a single Python file. I keep the footprint small because the Pi has limited RAM.

mkdir ~/edge_agent && cd ~/edge_agent
touch agent.py

Step 2: Configure the Oxlo.ai client

Open agent.py and initialize the client. I use deepseek-v3.2 because it handles structured reasoning well and is available on Oxlo.ai's free tier, which keeps prototyping costs at zero. Because Oxlo.ai charges per request rather than per token, I can stuff an entire shift's worth of verbose sensor logs into the prompt without watching the meter run.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"  # from https://portal.oxlo.ai
)

MODEL = "deepseek-v3.2"

Step 3: Define the system prompt

The system prompt constrains the model to short, actionable maintenance advice. Store it as a module-level constant so it is easy to tweak without touching request logic.

SYSTEM_PROMPT = """You are an industrial edge assistant.
Analyze the provided sensor telemetry and respond with:
1. A one-sentence health summary.
2. A bulleted list of any anomalies.
3. One recommended action.
Keep responses under 100 words."""

Step 4: Ingest local telemetry

On the edge device, sensor data is dumped to a local file. This helper reads the latest readings into a string.

def load_telemetry(path: str) -> str:
    with open(path, "r") as f:
        return f.read().strip()

For testing without hardware, create a dummy sensors.log:

echo "Pump_A: 120C, Vibration: 4.2mm/s, Flow: 80L/min
Pump_B: 115C, Vibration: 2.1mm/s, Flow: 82L/min" > sensors.log

Step 5: Build the inference loop

This function wraps the telemetry in a user message and sends it to Oxlo.ai. I set a 30-second timeout because edge LTE links can be slow.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def diagnose(path: str) -> str:
    telemetry = load_telemetry(path)
    user_message = f"Current telemetry:\n{telemetry}"

    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        timeout=30,
    )
    return response.choices[0].message.content

Step 6: Handle intermittent connectivity

Remote sites lose connectivity. I wrap the inference call in a simple retry loop so a brief dropout does not kill the script.

import time

def diagnose_with_retry(path: str, retries: int = 3) -> str:
    for attempt in range(1, retries + 1):
        try:
            return diagnose(path)
        except Exception as e:
            print(f"Attempt {attempt} failed: {e}")
            if attempt == retries:
                raise
            time.sleep(2 ** attempt)  # exponential back-off
    return ""

if __name__ == "__main__":
    import sys
    log_path = sys.argv[1] if len(sys.argv) > 1 else "sensors.log"
    print(diagnose_with_retry(log_path))

Run it

Install the dependency and execute the agent against the dummy log.

pip install openai
python agent.py sensors.log

Example output:

Health summary: Pump_A is operating above normal temperature.
- Anomaly: Pump_A vibration reads 4.2mm/s, exceeding the 3.0mm/s threshold.
- Pump_B readings are within normal limits.
Recommended action: Inspect Pump_A bearings and schedule maintenance within 24 hours.

Next steps

Add a local SQLite queue to buffer requests during network outages, then flush them when the link returns. If you need to analyze images from an edge camera, swap the model to kimi-k2.6 and pass base64-encoded frames to the same Oxlo.ai endpoint. See https://oxlo.ai/pricing to compare request-based costs against token-based providers for your expected payload sizes.

DEV Community