Best Practices for Low-Latency LLM Deployment

Building a production LLM service that responds in under 300 ms requires optimizing the full request path, not just picking a fast model. In this tutorial, I will walk through a FastAPI agent I shipped that classifies support tickets and drafts replies using Oxlo.ai. We will apply each low-latency tactic, from prompt compression to streaming, as we build it.

What you'll need

Before we start, gather the following:

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• pip install openai fastapi uvicorn cachetools
• A virtual environment (recommended)

Step 1: Configure the Oxlo.ai client

Set up the client with a short timeout and connection reuse. I keep the timeout at 10 seconds so a stalled request fails fast, and I rely on the underlying HTTP session to reuse TLS connections across calls.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY"),
    timeout=10,
    max_retries=1,
)

Step 2: Write a minimal system prompt

Every token in the prompt adds serialization overhead and context-window processing time. On Oxlo.ai, request-based pricing means the system prompt length does not affect cost, but it still adds latency. Keep the instructions under 100 tokens.

SYSTEM_PROMPT = """You are a support triage agent. Respond ONLY in JSON.
Keys: intent (one of: billing, technical, refund), urgency (low, medium, high), reply (max 20 words)."""

Step 3: Add streaming and JSON mode

Streaming gives us the first token immediately instead of waiting for the full response, and JSON mode guarantees we can parse the output without regex cleanup. I use llama-3.3-70b because it handles structured output reliably and streams quickly on Oxlo.ai.

import json

def triage_ticket(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        stream=True,
        max_tokens=150,
    )

    chunks = []
    for chunk in response:
        delta = chunk.choices[0].delta.content
        if delta:
            chunks.append(delta)

    raw = "".join(chunks)
    return json.loads(raw)

Step 4: Wrap it in a FastAPI endpoint

Using a standard sync route lets the OpenAI client run inside FastAPI's thread pool without blocking the event loop. This keeps the code simple and the deployment stable.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Ticket(BaseModel):
    message: str

@app.post("/triage")
def triage(ticket: Ticket):
    return triage_ticket(ticket.message)

Step 5: Cache identical requests

In production, users often submit the same query twice or refresh the page. A 5-second cache removes the API round-trip entirely for duplicates. Because Oxlo.ai charges per request, not per token, caching identical requests directly reduces cost without any token-counting math.

from cachetools import TTLCache
import hashlib

_cache = TTLCache(maxsize=500, ttl=5)

def triage_ticket(user_message: str) -> dict:
    key = hashlib.sha256(user_message.encode()).hexdigest()
    if key in _cache:
        return _cache[key]

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        stream=True,
        max_tokens=150,
    )

    chunks = []
    for chunk in response:
        delta = chunk.choices[0].delta.content
        if delta:
            chunks.append(delta)

    raw = "".join(chunks)
    result = json.loads(raw)
    _cache[key] = result
    return result

Run it

Save everything as main.py and start the server.

uvicorn main:app --host 0.0.0.0 --port 8000

Test the endpoint with curl.

curl -X POST http://localhost:8000/triage \
  -H "Content-Type: application/json" \
  -d '{"message": "I was charged twice for my subscription this month."}'

You should see a structured response immediately.

{"intent":"billing","urgency":"high","reply":"I have escalated the duplicate charge to our billing team."}

Wrap-up and next steps

Wire the /triage endpoint into your existing support queue and add Prometheus instrumentation to track p50 and p99 latencies. If you need even faster responses for shorter prompts, try qwen-3-32b on Oxlo.ai. It is a strong multilingual alternative that streams tokens quickly and fits well into agentic workflows.