We're going to build a customer support chat backend that drops into any existing web frontend. It streams responses from an LLM hosted on Oxlo.ai, using flat per-request pricing so long user messages and conversation history do not inflate costs. By the end you will have a working FastAPI endpoint and a minimal HTML page to exercise it.
What you'll need
You will need Python 3.10 or newer, an Oxlo.ai API key from https://portal.oxlo.ai, and the OpenAI SDK. Install the dependencies with pip.
pip install openai fastapi uvicorn python-dotenvStep 1: Scaffold the project
Create a directory for the service and a virtual environment to keep things isolated. I also create a .env file to hold the Oxlo.ai key so it is not hard-coded.
mkdir support-agent && cd support-agent
python -m venv .venv
source .venv/bin/activate # on Windows use .venv\Scripts\activate
# .env
OXLO_API_KEY=YOUR_OXLO_API_KEYStep 2: Configure the Oxlo.ai client and system prompt
Create main.py and initialize the OpenAI-compatible client pointing at Oxlo.ai. I load the key from the environment and define a system prompt that keeps the agent focused on our fictional store policies.
import os
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY"),
)The system prompt below governs tone and boundaries. Edit it to match your actual product.
SYSTEM_PROMPT = """You are a customer support agent for Acme Widgets.
- Answer questions about shipping, returns, and warranty.
- Be concise. Ask for clarification if a request is ambiguous.
- Do not make up policy. If you do not know, offer to escalate.
- Current policies: Free shipping over $50. Returns accepted within 30 days.
"""Step 3: Build the chat endpoint with conversation history
I use a simple in-memory dictionary to track sessions by a browser-generated ID. The endpoint appends the latest user message to the history and sends the full list to Oxlo.ai so the model retains context.
import os
from dotenv import load_dotenv
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict
load_dotenv()
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY"),
)
SYSTEM_PROMPT = """You are a customer support agent for Acme Widgets.
- Answer questions about shipping, returns, and warranty.
- Be concise. Ask for clarification if a request is ambiguous.
- Do not make up policy. If you do not know, offer to escalate.
- Current policies: Free shipping over $50. Returns accepted within 30 days.
"""
app = FastAPI()
sessions: Dict[str, List[dict]] = {}
class ChatRequest(BaseModel):
session_id: str
message: str
@app.post("/chat")
async def chat(req: ChatRequest):
if req.session_id not in sessions:
sessions[req.session_id] = [
{"role": "system", "content": SYSTEM_PROMPT}
]
sessions[req.session_id].append(
{"role": "user", "content": req.message}
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=sessions[req.session_id],
)
assistant_msg = response.choices[0].message.content
sessions[req.session_id].append(
{"role": "assistant", "content": assistant_msg}
)
return {"reply": assistant_msg}Step 4: Add streaming for real-time tokens
Waiting for the full completion feels slow in a browser, so I switch on streaming. The endpoint now returns a FastAPI StreamingResponse that yields text chunks as Oxlo.ai generates them.
import os
from dotenv import load_dotenv
from openai import OpenAI
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from typing import List, Dict
load_dotenv()
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY"),
)
SYSTEM_PROMPT = """You are a customer support agent for Acme Widgets.
- Answer questions about shipping, returns, and warranty.
- Be concise. Ask for clarification if a request is ambiguous.
- Do not make up policy. If you do not know, offer to escalate.
- Current policies: Free shipping over $50. Returns accepted within 30 days.
"""
app = FastAPI()
sessions: Dict[str, List[dict]] = {}
class ChatRequest(BaseModel):
session_id: str
message: str
@app.post("/chat")
async def chat(req: ChatRequest):
if req.session_id not in sessions:
sessions[req.session_id] = [
{"role": "system", "content": SYSTEM_PROMPT}
]
sessions[req.session_id].append(
{"role": "user", "content": req.message}
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=sessions[req.session_id],
stream=True,
)
def event_generator():
assistant_buffer = ""
for chunk in response:
delta = chunk.choices[0].delta.content or ""
assistant_buffer += delta
yield delta
sessions[req.session_id].append(
{"role": "assistant", "content": assistant_buffer}
)
return StreamingResponse(event_generator(), media_type="text/plain")Step 5: Create a test frontend
I add a static file mount so FastAPI serves an HTML test page. This file posts to /chat and appends each chunk to the page as it arrives, giving you a working demo without React or Webpack.
import os
from fastapi.staticfiles import StaticFiles
os.makedirs("static", exist_ok=True)
app.mount("/static", StaticFiles(directory="static"), name="static")Save this as static/index.html.
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Support Agent</title>
<style>
body { font-family: sans-serif; max-width: 600px; margin: 40px auto; }
#log { border: 1px solid #ccc; padding: 12px; height: 300px; overflow-y: auto; margin-bottom: 12px; }
.msg { margin: 4px 0; }
.user { color: #0366d6; }
.bot { color: #22863a; }
</style>
</head>
<body>
<h2>Acme Widgets Support</h2>
<div id="log"></div>
<input id="inp" type="text" placeholder="Ask about shipping or returns..." style="width: 70%;" />
<button onclick="send()">Send</button>
<script>
const sessionId = Math.random().toString(36).slice(2);
const log = document.getElementById('log');
const inp = document.getElementById('inp');
function append(cls, text) {
const d = document.createElement('div');
d.className = 'msg ' + cls;
d.textContent = text;
log.appendChild(d);
log.scrollTop = log.scrollHeight;
}
async function send() {
const text = inp.value.trim();
if (!text) return;
append('user', 'You: ' + text);
inp.value = '';
const res = await fetch('/chat', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({session_id: sessionId, message: text})
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let botText = '';
const botEl = document.createElement('div');
botEl.className = 'msg bot';
botEl.textContent = 'Agent: ';
log.appendChild(botEl);
while (true) {
const {done, value} = await reader.read();
if (done) break;
botText += decoder.decode(value, {stream: true});
botEl.textContent = 'Agent: ' + botText;
log.scrollTop = log.scrollHeight;
}
}
inp.addEventListener('keydown', e => { if (e.key === 'Enter') send(); });
</script>
</body>
</html>Run it
Start the server with Uvicorn and open http://localhost:8000/static/index.html in your browser.
uvicorn main:app --reloadTry pasting a long message with order details. Because Oxlo.ai charges per request rather than per token, a multi-paragraph user message costs the same as a one-word message. The agent should answer policy questions and remember prior turns.
Example interaction after asking "What is your return policy?" and then "Does that include opened items?":
Agent: We accept returns within 30 days of delivery.
Agent: Yes, opened items can be returned within the same 30-day window as long as they are not damaged.Wrap-up
The endpoint is ready to integrate into your actual frontend. Two concrete next steps: wire the session store to Redis so history survives restarts, and add a tool-calling loop so the agent can look up real order IDs via your existing API. Oxlo.ai supports function calling on models like Llama 3.3 70B and Qwen 3 32B, so you can expand this into a fully agentic workflow without changing your integration code.
Top comments (0)