DEV Community

shashank ms
shashank ms

Posted on

Integrating LLM with Augmented Reality Applications: A Step-by-Step Guide

#learnai #oxlo #ai

We are building a web-based AR repair assistant that takes a plain-text description of a physical scene, asks an LLM for contextual annotations, and renders those instructions as overlays on a live camera feed. It helps developers prototype intelligent AR guidance without native app stores or unpredictable token bills. Oxlo.ai fits here because its request-based pricing means a long, detailed scene description costs the same flat rate as a short one.

What you'll need

  • Python 3.10 or newer
  • An Oxlo.ai API key from https://portal.oxlo.ai
  • The OpenAI SDK and FastAPI: pip install openai fastapi uvicorn python-multipart
  • A modern browser with camera access

Step 1: Scaffold the backend

Create a new project folder and a main.py file. I will set up a FastAPI app with CORS enabled so the frontend can reach it during local development.

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from openai import OpenAI
import json

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

class SceneRequest(BaseModel):
    description: str
    task: str = "diagnose"

@app.get("/health")
def health():
    return {"status": "ok"}

Step 2: Write the system prompt

The system prompt tells the model how to format AR overlays. I want valid JSON with label text and normalized screen coordinates so the frontend can position divs directly.

SYSTEM_PROMPT = """You are an AR repair assistant. The user describes what they see in their physical environment.

Respond with a JSON object containing:
- "thought": a brief analysis of the scene
- "overlays": a list of objects, each with:
  - "label": the text to display
  - "x": horizontal position from 0.0 (left) to 1.0 (right)
  - "y": vertical position from 0.0 (top) to 1.0 (bottom)
  - "color": a CSS color for the label

Keep labels under 60 characters. Position overlays near the relevant object described by the user. If the user describes multiple items, prioritize the one most relevant to the task."""

Step 3: Connect to Oxlo.ai and create the annotation endpoint

Here I wire the endpoint to Oxlo.ai. I use the OpenAI SDK as a drop-in client and call Llama 3.3 70B. Because Oxlo.ai uses request-based pricing, not token-based pricing, a long scene description does not increase the cost.

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

@app.post("/annotate")
def annotate_scene(req: SceneRequest):
    user_message = f"Task: {req.task}\nScene: {req.description}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    raw = response.choices[0].message.content
    # Strip markdown fences if the model wraps JSON
    cleaned = raw.strip()
    if cleaned.startswith("

```"):
        cleaned = cleaned.split("```

")[1]
        if cleaned.startswith("json"):
            cleaned = cleaned[4:]
        cleaned = cleaned.strip()

    return json.loads(cleaned)

Step 4: Build the AR frontend

Create a static/ folder and add index.html. The page requests camera access, shows the video full-screen, and provides a text box for the user to describe what they see.

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>AR LLM Overlay</title>
  <style>
    body { margin: 0; overflow: hidden; background: #000; font-family: system-ui, sans-serif; }
    #video { position: fixed; top: 0; left: 0; width: 100vw; height: 100vh; object-fit: cover; }
    #ui { position: fixed; bottom: 20px; left: 20px; right: 20px; z-index: 10; display: flex; gap: 10px; }
    input { flex: 1; padding: 12px; border: none; border-radius: 8px; font-size: 16px; }
    button { padding: 12px 20px; border: none; border-radius: 8px; background: #0af; color: #fff; font-weight: 600; }
    .overlay { position: fixed; padding: 6px 10px; border-radius: 6px; background: rgba(0,0,0,0.6); color: #fff; font-size: 14px; pointer-events: none; transform: translate(-50%, -50%); border: 2px solid; }
  </style>
</head>
<body>
  <video id="video" autoplay playsinline></video>
  <div id="ui">
    <input id="desc" placeholder="Describe what you see..." />
    <button onclick="analyze()">Analyze</button>
  </div>
  <div id="overlays"></div>

  <script>
    const video = document.getElementById('video');
    navigator.mediaDevices.getUserMedia({ video: { facingMode: 'environment' } })
      .then(stream => video.srcObject = stream);

    async function analyze() {
      const desc = document.getElementById('desc').value;
      const res = await fetch('http://localhost:8000/annotate', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ description: desc, task: 'diagnose' })
      });
      const data = await res.json();
      render(data.overlays);
    }

    function render(overlays) {
      const container = document.getElementById('overlays');
      container.innerHTML = '';
      overlays.forEach(o => {
        const el = document.createElement('div');
        el.className = 'overlay';
        el.textContent = o.label;
        el.style.left = (o.x * 100) + '%';
        el.style.top = (o.y * 100) + '%';
        el.style.borderColor = o.color || '#0af';
        container.appendChild(el);
      });
    }
  </script>
</body>
</html>

Step 5: Serve the static files from FastAPI

I need to mount the static folder so the browser can load the page. Add these lines to main.py near the top.

from fastapi.staticfiles import StaticFiles

app.mount("/static", StaticFiles(directory="static"), name="static")

Run it

Start the server with Uvicorn, then open http://localhost:8000/static/index.html on a phone or laptop with a camera.

uvicorn main:app --reload --host 0.0.0.0

I pointed my phone at a desk with a loose cable and typed: "I see a router with a blinking red light and a loose ethernet cable on the left side." The backend returned:

{
  "thought": "The blinking red light indicates no internet connection, likely caused by the unplugged cable.",
  "overlays": [
    { "label": "Plug in ethernet cable", "x": 0.25, "y": 0.55, "color": "#ff4444" },
    { "label": "Check power LED", "x": 0.60, "y": 0.40, "color": "#00ff88" }
  ]
}

The frontend rendered two labels directly over the video feed. Because Oxlo.ai uses request-based pricing, this interaction cost one flat request regardless of how verbose my scene description was.

Next steps

Wire in an Oxlo.ai vision model to auto-generate the scene description from camera frames before passing it to Llama 3.3 70B for overlay instructions. Or package the frontend as a PWA with the Web Share Target API so technicians can launch it directly from a work ticket.

Top comments (0)