We are going to build a VR world generator that turns text prompts into interactive A-Frame scenes. A FastAPI backend calls Oxlo.ai to generate 3D markup and atmospheric narration, while a lightweight frontend renders it in the browser. This is useful for rapid prototyping of virtual environments, interactive storytelling, or any application where non-developers need to create spatial experiences.
What you'll need
Before starting, make sure you have Python 3.10 or newer installed. You will also need an Oxlo.ai API key from https://portal.oxlo.ai and the OpenAI SDK. Oxlo.ai uses request-based pricing, so sending long scene descriptions or multi-turn context does not inflate your cost the way token-based billing would. For details, see https://oxlo.ai/pricing.
- Python 3.10+
pip install openai fastapi uvicorn- An Oxlo.ai API key
- A modern web browser with WebGL support
1. Set up the Oxlo.ai client
First, I verify that I can reach Oxlo.ai and get coherent scene descriptions. I create vr_builder.py and initialize the OpenAI-compatible client pointing at Oxlo.ai.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
user_message = "A bioluminescent cave with glowing crystals and still water."
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a VR scene designer. Respond with a one-sentence description of the environment."},
{"role": "user", "content": user_message},
],
)
print(response.choices[0].message.content)
2. Define the system prompt and scene generator
Now I lock down the output format. The model must return valid JSON containing the A-Frame scene markup, a narration string, and a list of interactive object IDs. I define the system prompt as a constant so I can tweak it without touching the rest of the logic.
SYSTEM_PROMPT = """You are a VR world builder. Given a user description, generate a JSON object with exactly three keys:
- "scene_html": A string containing a complete A-Frame <a-scene> element. Include a sky, ground plane, lighting, and at least three interactive objects. Every interactive object must have a unique id and class="clickable". Use only standard A-Frame primitives.
- "narration": A short atmospheric narration to display to the user.
- "objects": A list of strings containing the id of every interactive object.
Output only valid JSON. Do not wrap the output in markdown."""Then I wire it into a reusable function that calls Oxlo.ai and parses the result.
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def generate_vr_scene(user_message: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
content = response.choices[0].message.content
return json.loads(content)
3. Build the FastAPI backend
Next, I wrap the generator in a FastAPI application. The /generate endpoint accepts a prompt, calls Oxlo.ai, and returns the parsed scene data. I add CORS so the frontend can reach it during local development.
import json
from openai import OpenAI
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
import uvicorn
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a VR world builder. Given a user description, generate a JSON object with exactly three keys:
- "scene_html": A string containing a complete A-Frame <a-scene> element. Include a sky, ground plane, lighting, and at least three interactive objects. Every interactive object must have a unique id and class="clickable". Use only standard A-Frame primitives.
- "narration": A short atmospheric narration to display to the user.
- "objects": A list of strings containing the id of every interactive object.
Output only valid JSON. Do not wrap the output in markdown."""
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)
class PromptRequest(BaseModel):
prompt: str
def generate_vr_scene(user_message: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
content = response.choices[0].message.content
return json.loads(content)
@app.post("/generate")
def create_scene(req: PromptRequest):
return generate_vr_scene(req.prompt)
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)
4. Create the A-Frame frontend
The frontend is a single HTML file that hosts an iframe for the VR scene. It posts the user prompt to the FastAPI backend, then injects the returned HTML into the iframe using srcdoc. This keeps the generated world isolated and avoids A-Frame DOM merge issues.
<!DOCTYPE html>
<html>
<head>
<style>
body { margin: 0; font-family: sans-serif; display: flex; flex-direction: column; height: 100vh; background: #0b0c10; color: #c5c6c7; }
#controls { padding: 1rem; background: #1f2833; border-bottom: 1px solid #45a29e; }
input, button { padding: 0.5rem; font-size: 1rem; margin-right: 0.5rem; }
#vr-frame { flex: 1; border: none; }
#narration { margin-top: 0.5rem; font-style: italic; }
</style>
</head>
<body>
<div id="controls">
<input id="prompt" type="text" value="A cyberpunk alley with neon signs and rain puddles" size="50">
<button onclick="generate()">Generate World</button>
<div id="narration">Ready.</div>
</div>
<iframe id="vr-frame" allow="vr"></iframe>
<script>
async function generate() {
const user_message = document.getElementById('prompt').value;
const res = await fetch('http://localhost:8000/generate', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({prompt: user_message})
});
const data = await res.json();
document.getElementById('narration').innerText = data.narration;
const doc = `<!DOCTYPE html>
<html>
<head><script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script></head>
<body style="margin:0">${data.scene_html}</body>
</html>`;
document.getElementById('vr-frame').srcdoc = doc;
window.currentScene = data;
}
</script>
</body>
</html>
5. Add dynamic object interactivity
Static scenes are only half the story. I want users to click an object and receive AI-generated lore. I add an /interact endpoint that uses qwen-3-32b for creative reasoning, then expose buttons for each object. When a button is clicked, the frontend manipulates the iframe DOM directly to provide immediate visual feedback.
Add the new endpoint to vr_builder.py before the uvicorn block.
class InteractRequest(BaseModel):
object_id: str
scene_context: str
@app.post("/interact")
def interact(req: InteractRequest):
system_prompt = "You are a VR lore master. Given an object ID and scene context, return a JSON object with 'lore' (one mysterious sentence) and 'effect' (one of: glow, pulse, shrink). Output only JSON."
user_message = f"Object: {req.object_id}\nContext: {req.scene_context}"
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_message},
],
)
return json.loads(response.choices[0].message.content)
Then extend the frontend to render interaction buttons after each generation.
async function generate() {
const user_message = document.getElementById('prompt').value;
const res = await fetch('http://localhost:8000/generate', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({prompt: user_message})
});
const data = await res.json();
document.getElementById('narration').innerText = data.narration;
const doc = `<!DOCTYPE html>
<html>
<head><script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script></head>
<body style="margin:0">${data.scene_html}</body>
</html>`;
document.getElementById('vr-frame').srcdoc = doc;
const controls = document.getElementById('controls');
controls.querySelectorAll('.obj-btn').forEach(b => b.remove());
data.objects.forEach(obj => {
const btn = document.createElement('button');
btn.className = 'obj-btn';
btn.innerText = obj;
btn.onclick = async () => {
const ires = await fetch('http://localhost:8000/interact', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({object_id: obj, scene_context: data.narration})
});
const idata = await ires.json();
document.getElementById('narration').innerText = idata.lore;
const el = document.getElementById('vr-frame').contentDocument.getElementById(obj);
if (el && idata.effect === 'glow') {
el.setAttribute('material', 'color', '#ffaa00');
}
};
controls.appendChild(btn);
});
}
Run it
Start the backend from your terminal.
python vr_builder.py
Open index.html in your browser, or serve it with python -m http.server 8080 and visit http://localhost:8080. Enter a prompt and click Generate World. You should see an A-Frame scene load in the iframe along with a narration line. Example output after clicking Generate World:
{
"scene_html": "<a-scene>\n <a-sky color=\"#050510\"></a-sky>\n <a-plane position=\"0 0 -4\" rotation=\"-90 0 0\" width=\"20\" height=\"20\" color=\"#222\"></a-plane>\n <a-box id=\"crate\" class=\"clickable\" position=\"-1 0.5 -3\" color=\"#654321\"></a-box>\n <a-sphere id=\"orb\" class=\"clickable\" position=\"1 1 -3\" color=\"#00ffcc\"></a-sphere>\n <a-cylinder id=\"pillar\" class=\"clickable\" position=\"0 1 -4\" radius=\"0.3\" height=\"2\" color=\"#777\"></a-cylinder>\n <a-light type=\"ambient\" color=\"#444\"></a-light>\n <a-light type=\"point\" position=\"2 3 2\" color=\"#ffaa00\"></a-light>\n</a-scene>",
"narration": "The alley hums with hidden current. Three artifacts await your attention.",
"objects": ["crate", "orb", "pillar"]
}
Clicking the orb button might return:
{
"lore": "The orb pulses with a frequency that predates the city above.",
"effect": "glow"
}
Next steps
Swap the text buttons for speech input by piping microphone audio through Oxlo.ai's Whisper endpoint, then synthesize responses with Kokoro TTS for a fully hands-free VR interface. If you want the world to remember state across sessions, store object embeddings using Oxlo.ai's BGE-Large endpoint and retrieve them on each load to build persistent narrative memory.
Top comments (0)