Introduction to Cold Start in LLMs

#learnai #oxlo #ai

We're building a customer support agent that looks up order status and answers questions instantly. Cold start, the latency spike when a serverless model wakes up from idle, can ruin that experience. We'll assemble the agent, see where cold start bites, and run it on Oxlo.ai where popular models stay warm.

What you'll need

Python 3.10 or newer
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Bootstrap the Oxlo.ai client

Oxlo.ai is fully OpenAI SDK compatible, so a base URL swap is all we need. I will use Llama 3.3 70B as the general-purpose engine.

from openai import OpenAI
import json

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"  # from https://portal.oxlo.ai
)

MODEL = "llama-3.3-70b"

Step 2: Define the system prompt

The agent must stay concise, ask for a missing order ID, and rely on the lookup tool rather than guessing.

SYSTEM_PROMPT = """You are a support agent for an electronics store.
Your job is to answer order status questions.
If the user provides an order ID, use the lookup_order_status tool to get the current status.
If they do not provide an order ID, ask for it.
Keep responses under two sentences.
"""

Step 3: Build the mock tool and schema

In production this would query a database, but a dictionary is enough to demonstrate the tool-calling loop. We also declare the schema so the model knows when to invoke it.

ORDERS_DB = {
    "ORD-1001": {"status": "shipped", "eta": "2025-06-15"},
    "ORD-1002": {"status": "processing", "eta": "2025-06-18"},
}

tools = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order_status",
            "description": "Retrieve status and ETA for an order ID.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The order identifier, e.g. ORD-1001."
                    }
                },
                "required": ["order_id"]
            }
        }
    }
]

Step 4: Send the first request

This is the moment cold start usually strikes. On many serverless inference platforms, an idle model container has to wake up, so the user waits before seeing the first token. Oxlo.ai keeps popular models warm, which means the first request after quiet hours still returns quickly.

user_message = "Where is my order ORD-1001?"

response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    tools=tools,
    tool_choice="auto"
)

Step 5: Execute the tool call

When the model emits a tool call, we parse the arguments, run our lookup, and append the result as a tool message. Then we call the model again to produce the final, human-readable answer.

msg = response.choices[0].message

if msg.tool_calls:
    tool_call = msg.tool_calls[0]
    args = json.loads(tool_call.function.arguments)
    order_id = args.get("order_id", "")
    result = ORDERS_DB.get(order_id, {"status": "unknown", "eta": "N/A"})

    tool_response = {
        "role": "tool",
        "tool_call_id": tool_call.id,
        "content": json.dumps(result)
    }

    assistant_msg = {
        "role": "assistant",
        "content": msg.content,
        "tool_calls": [
            {
                "id": tool_call.id,
                "type": tool_call.type,
                "function": {
                    "name": tool_call.function.name,
                    "arguments": tool_call.function.arguments
                }
            }
        ]
    }

    final_response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
            assistant_msg,
            tool_response,
        ]
    )

    print(final_response.choices[0].message.content)
else:
    print(msg.content)

Run it

Save the complete script as support_agent.py, replace YOUR_OXLO_API_KEY, and run:

python support_agent.py

Expected output:

Your order ORD-1001 has shipped and is expected to arrive by 2025-06-15.

The assistant first emitted the tool call internally, then synthesized the final sentence after receiving the JSON result. Because Oxlo.ai has no cold starts on popular models, both turns feel instant even if this is the first request of the day.

Wrap-up

We now have a working support agent that introduces the cold start problem in practical terms. To push it further, wire the lookup_order_status function to a real REST API, or add streaming responses so the user sees tokens appear character by character. For flat per-request pricing that does not scale with prompt length, see https://oxlo.ai/pricing.

DEV Community