Solving the Cold Start Problem in LLM

#engineering #oxlo #ai

Cold starts make serverless LLM inference unpredictable, forcing teams to build warming scripts and keep-alive hacks. We will build a real-time support triage agent that routes tickets instantly without any pre-warming logic. It runs on Oxlo.ai, which serves popular models with zero cold starts.

What you'll need

An Oxlo.ai API key from https://portal.oxlo.ai
Python 3.10 or newer
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

We instantiate the OpenAI SDK with Oxlo.ai's base URL. There are no custom headers or warming endpoints to manage.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

Step 2: Define the system prompt

The prompt forces strict JSON output so downstream code can parse the model response reliably.

SYSTEM_PROMPT = """You are a support triage agent. Analyze the customer message and respond with valid JSON containing exactly these keys:
- category: one of [Billing, Technical, Account]
- urgency: one of [Low, Medium, High]
- draft_reply: a one-sentence internal note for the support team.
Do not wrap the JSON in markdown fences. Output raw JSON only."""

Step 3: Build the triage function

This function calls Oxlo.ai's Llama 3.3 70B and parses the returned JSON. Because Oxlo.ai has no cold starts on popular models, the first request of the day returns just as fast as the hundredth.

import json

def triage_ticket(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    raw = response.choices[0].message.content
    return json.loads(raw.strip())

Step 4: Add routing logic

We use the structured output to decide whether to escalate, queue, or auto-reply. Keeping this logic in pure Python makes it easy to unit test without calling the model.

def route_ticket(triage: dict) -> str:
    category = triage["category"]
    urgency = triage["urgency"]
    draft = triage["draft_reply"]

    if urgency == "High":
        action = "Escalate to on-call engineer immediately."
    elif urgency == "Medium":
        action = "Add to queue for next available agent."
    else:
        action = "Send auto-reply and resolve during business hours."

    print(f"Category: {category}")
    print(f"Urgency: {urgency}")
    print(f"Draft: {draft}")
    print(f"Action: {action}")
    return action

Step 5: Wire the agent together

The main block feeds a sample billing ticket through the pipeline. You can replace the string with live input from a webhook or queue.

if __name__ == "__main__":
    ticket = (
        "I was charged twice for my subscription this month "
        "and I need a refund ASAP."
    )

    result = triage_ticket(ticket)
    route_ticket(result)

Run it

Save the script as triage_agent.py, export your key, and run it.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python triage_agent.py

Category: Billing
Urgency: High
Draft: Customer reports duplicate subscription charge and requests immediate refund.
Action: Escalate to on-call engineer immediately.

Next steps

Deploy the script as a FastAPI endpoint and point your support form at it. If you receive multilingual tickets, swap the model string to qwen-3-32b without changing any other code, since Oxlo.ai uses the same OpenAI-compatible shape for every model.

DEV Community