shashank ms

Posted on Jul 21

Integrating LLM with Web Applications: A Step-by-Step Guide

#engineering #oxlo #ai

We are going to build a multi-turn product support agent behind a FastAPI endpoint. It uses Oxlo.ai's request-based pricing to keep costs predictable even when users paste long stack traces, and because Oxlo.ai is fully OpenAI SDK compatible, the integration is literally a base_url swap. By the end you will have a stateful HTTP API that any frontend can call.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK and FastAPI: pip install openai fastapi uvicorn pydantic

Step 1: Scaffold the FastAPI application

Create a new directory and a file named main.py. Start with the imports and a Pydantic model that validates incoming chat messages.

# main.py
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class ChatRequest(BaseModel):
    thread_id: str
    message: str

Step 2: Write the system prompt

The system prompt is the only part a product manager should need to edit. I keep it in a module-level constant so it is easy to find and tweak without touching route logic.

SYSTEM_PROMPT = """You are a support engineer for AcmeAnalytics, a web analytics platform.
Be concise, ask clarifying questions if the user is vague, and never share internal API keys.
If you do not know the answer, say so and offer to escalate to a human."""

Step 3: Connect to Oxlo.ai

Instantiate the OpenAI client with Oxlo.ai's base URL. Because Oxlo.ai is fully OpenAI SDK compatible, this is the only change needed to route traffic to Oxlo.ai. I am using Llama 3.3 70B here because it handles open-ended support conversations reliably.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 4: Build the chat endpoint

Add the POST handler that forwards the user message to Oxlo.ai and returns the reply. At this stage the agent is stateless, but it proves the plumbing works.

@app.post("/chat")
def chat(req: ChatRequest):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": req.message},
        ],
    )
    return {"reply": response.choices[0].message.content}

Step 5: Add conversation memory

A support thread without history is frustrating. Store each thread in an in-memory dictionary keyed by thread_id. In production you will want Redis, but a dict is enough to prove the concept.

from typing import List, Dict

memory: Dict[str, List[Dict[str, str]]] = {}

@app.post("/chat")
def chat(req: ChatRequest):
    history = memory.get(req.thread_id, [])
    history.append({"role": "user", "content": req.message})

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[{"role": "system", "content": SYSTEM_PROMPT}] + history,
    )

    assistant_msg = response.choices[0].message.content
    history.append({"role": "assistant", "content": assistant_msg})
    memory[req.thread_id] = history

    return {"reply": assistant_msg}

Step 6: Enable CORS and start the server

Add CORS middleware so a frontend on localhost:3000 can call the agent, then wire up uvicorn.

from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_methods=["*"],
    allow_headers=["*"],
)

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

Run it

Start the server.

python main.py

In another terminal, send a message.

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"thread_id":"thread_abc","message":"How do I export a CSV from the dashboard?"}'

The agent should reply immediately. Example response:

{"reply":"Go to Dashboard > Export and select CSV. If your dataset is over 100k rows, use the async export option to avoid timeouts. Need help with that?"}

Send a follow-up with the same thread_id to test memory.

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"thread_id":"thread_abc","message":"Where is the async option?"}'

Next steps

Swap the in-memory dict for Redis or PostgreSQL so threads survive restarts, and wire in Oxlo.ai's function calling to let the agent look up live order status before it replies. If your support tickets tend to include long logs, keep an eye on Oxlo.ai's per-request pricing at https://oxlo.ai/pricing. It stays flat regardless of how many tokens the user pasted.

DEV Community