DEV Community

shashank ms
shashank ms

Posted on

The Future of LLMs in Autonomous Vehicles

#learnai #oxlo #ai

We are building a lightweight behavioral planning prototype for autonomous vehicles that turns structured sensor telemetry into structured driving decisions using an LLM. This gives AV teams a fast way to experiment with reasoning-based policies without retraining a dedicated model.

What you'll need

  • Python 3.10 or newer
  • The OpenAI SDK: pip install openai
  • An Oxlo.ai API key from https://portal.oxlo.ai
  • A working internet connection

Step 1: Initialize the Oxlo.ai client

I start by importing the OpenAI SDK and pointing it at Oxlo.ai. Because Oxlo.ai is fully OpenAI API compatible, this is a drop-in replacement. I picked Llama 3.3 70B for this planner because it responds reliably to structured instructions and has no cold starts on Oxlo.ai.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Model the telemetry payload

AV stacks generate JSON-like sensor fusion logs. I define a simple dataclass so the input shape stays consistent across scenes. This keeps the prompt clean and the simulation repeatable.

import json
from dataclasses import dataclass, asdict

@dataclass
class Telemetry:
    timestamp: str
    speed_mps: float
    weather: str
    obstacles: list
    lane_status: str
    traffic_light: str

scene = Telemetry(
    timestamp="2024-05-21T14:32:00Z",
    speed_mps=12.5,
    weather="light_rain",
    obstacles=[
        {"type": "pedestrian", "distance_m": 8.0, "bearing": "front_left"},
        {"type": "vehicle", "distance_m": 25.0, "bearing": "front"}
    ],
    lane_status="clear",
    traffic_light="green"
)

user_message = json.dumps(asdict(scene), indent=2)

Step 3: Define the system prompt and planner function

The system prompt locks the model into a safety-critical AV planner role. I force JSON output through instructions rather than external parsers so the code stays portable. The planner function sends the telemetry to Oxlo.ai and returns the parsed decision.

SYSTEM_PROMPT = """You are an autonomous vehicle behavioral planning module.
Your input is a JSON telemetry snapshot from the vehicle sensor stack.
You must output a single JSON object with no markdown formatting and no commentary outside the JSON.
Required keys:
- maneuver: one of [MAINTAIN_SPEED, DECELERATE, STOP, CHANGE_LANE_LEFT, CHANGE_LANE_RIGHT]
- target_speed_mps: float
- reasoning: string explaining the safety rationale
- confidence: float between 0.0 and 1.0

Rules:
1. If any obstacle is closer than 10 meters, favor DECELERATE or STOP.
2. If the traffic light is red, maneuver must be STOP.
3. In light rain, reduce target_speed_mps by 20 percent.
4. Always prioritize pedestrian safety over traffic flow."""

def plan_maneuver(scene_json: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": scene_json},
        ],
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Simulate a driving sequence

To see how the planner behaves across time, I feed it three consecutive snapshots: normal cruising, a nearby pedestrian, and a red light. This mimics a real AV decision loop.

scenarios = [
    Telemetry("2024-05-21T14:32:01Z", 15.0, "dry", [], "clear", "green"),
    Telemetry("2024-05-21T14:32:02Z", 15.0, "dry", [{"type": "pedestrian", "distance_m": 6.0, "bearing": "front"}], "clear", "green"),
    Telemetry("2024-05-21T14:32:03Z", 0.0, "dry", [], "clear", "red"),
]

for s in scenarios:
    payload = json.dumps(asdict(s), indent=2)
    decision = plan_maneuver(payload)
    print(f"{s.timestamp} | {decision['maneuver']} | speed={decision['target_speed_mps']} | confidence={decision['confidence']}")
    print(f"  reasoning: {decision['reasoning']}")

Run it

Save the complete script as av_planner.py, replace YOUR_OXLO_API_KEY, and run python av_planner.py. The Oxlo.ai endpoint streams back decisions with low latency because there are no cold starts on popular models. You should see output similar to this:

2024-05-21T14:32:01Z | MAINTAIN_SPEED | speed=15.0 | confidence=0.95
  reasoning: Clear lane, green light, no obstacles. Safe to maintain current speed.
2024-05-21T14:32:02Z | DECELERATE | speed=5.0 | confidence=0.92
  reasoning: Pedestrian detected at 6 meters. Prioritizing pedestrian safety by decelerating rapidly.
2024-05-21T14:32:03Z | STOP | speed=0.0 | confidence=0.99
  reasoning: Traffic light is red. Regulatory stop required.

Because Oxlo.ai uses request-based pricing, sending these long telemetry payloads does not inflate cost the way token-based billing would. For a prototype running hundreds of verbose sensor logs per day, that difference adds up. You can see the exact pricing structure at https://oxlo.ai/pricing.

Next steps

  • Wire this planner into a real-time ROS 2 node or CARLA simulator bridge so the LLM consumes live LiDAR and camera summaries instead of hand-written JSON.
  • Swap in kimi-k2.6 or deepseek-v3.2 on Oxlo.ai to compare reasoning depth for edge cases like unprotected left turns or construction zones.

Top comments (0)