Understanding Cold Start in LLM

#learnai #oxlo #ai

We are building a support ticket triage CLI that classifies incoming issues by urgency and category. Latency matters here because users abandon chat after a few seconds of silence, and the first request after an idle period is exactly where cold starts punish serverless inference providers. I will show you a lightweight Python tool that batches tickets through Oxlo.ai and logs every millisecond, proving that no cold starts means predictable latency from the first call.

What you'll need

You will need Python 3.10 or newer, the OpenAI SDK installed with pip install openai, and an Oxlo.ai API key from https://portal.oxlo.ai. I also use a small JSON file of fake tickets so we can test against realistic text lengths.

Step 1: Create sample tickets

Create a file named tickets.json with three sample support tickets. These give us enough variety to exercise the classifier.

[
  {
    "id": "T-001",
    "subject": "Cannot reset password",
    "body": "The reset link expires immediately after I click it. I have tried three different browsers."
  },
  {
    "id": "T-002",
    "subject": "Invoice PDF shows wrong total",
    "body": "We were charged for 5 seats instead of 3 on the March invoice. Please correct and reissue."
  },
  {
    "id": "T-003",
    "subject": "Feature request: dark mode",
    "body": "Would love a dark theme for the dashboard. Not urgent, just nice to have."
  }
]

Step 2: Define the triage prompt

The system prompt forces the model to return only JSON. This keeps parsing trivial and avoids explanation tokens that waste time.

SYSTEM_PROMPT = """You are a support triage agent.
Read the ticket and respond ONLY with a JSON object containing two keys:
- category: one of Billing, Bug, FeatureRequest, Account
- urgency: one of Low, Medium, High
Do not wrap the JSON in markdown. Do not add commentary."""

Step 3: Initialize the Oxlo.ai client

Oxlo.ai exposes a fully OpenAI-compatible endpoint, so we can use the standard SDK by swapping the base URL and API key. Because Oxlo.ai keeps popular models warm, we skip the cold-start delay that occurs when a provider spins down GPUs after idle time.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 4: Build the classifier

This function formats the ticket, calls Llama 3.3 70B on Oxlo.ai, and times the round trip. Notice that the first request in the batch is just as fast as the rest because the model is already warm.

import json
import time

def triage_ticket(ticket: dict) -> dict:
    user_message = f"Subject: {ticket['subject']}\nBody: {ticket['body']}"

    start = time.perf_counter()
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    elapsed = time.perf_counter() - start

    raw = response.choices[0].message.content
    result = json.loads(raw)
    result["ticket_id"] = ticket["id"]
    result["latency_ms"] = round(elapsed * 1000, 1)
    return result

Step 5: Run the batch

Finally, load the JSON array and print each result. I keep the output as raw Python dicts so you can pipe it to jq or another tool.

import json

def main():
    with open("tickets.json") as f:
        tickets = json.load(f)

    for t in tickets:
        out = triage_ticket(t)
        print(out)

if __name__ == "__main__":
    main()

Run it

Set your API key and run the script. Your exact latency will depend on network conditions, but you should see sub-second responses for every ticket, including the very first call.

export OXLO_API_KEY="sk-oxlo.ai-..."
python triage.py

{'category': 'Account', 'urgency': 'High', 'ticket_id': 'T-001', 'latency_ms': 820.4}
{'category': 'Billing', 'urgency': 'High', 'ticket_id': 'T-002', 'latency_ms': 845.1}
{'category': 'FeatureRequest', 'urgency': 'Low', 'ticket_id': 'T-003', 'latency_ms': 801.2}

If your tickets grow into long context threads, Oxlo.ai's request-based pricing stays flat regardless of input length. See https://oxlo.ai/pricing for details.

Wrap-up

You now have a cold-start-free triage agent that is ready to drop into a serverless function or a CI pipeline. Try wiring it to a Slack webhook so high urgency tickets ping the on-call channel immediately, or swap in deepseek-v3.2 if you need heavier reasoning for ambiguous escalations.

DEV Community