When you ship an LLM feature to production, latency is usually the first metric users notice. A sub-second response feels instant, but a multi-second hang breaks flow. One of the most common causes of unpredictable latency is the cold start: the delay between sending an API request and the model actually generating its first token. Unlike traditional software cold starts, which are measured in milliseconds, LLM cold starts can stretch to tens of seconds while massive parameter weights are loaded into GPU memory and attention caches are initialized. For agentic workflows, multi-turn conversations, and long-context pipelines, these delays compound quickly.
What Are Cold Starts in LLM Inference?
In the context of large language models, a cold start refers to the latency penalty incurred when an inference server must load a model from storage into GPU VRAM before it can process a request. If the model weights are not resident in memory, or if the serving container has scaled to zero, the first request triggers a sequence of expensive operations: container initialization, CUDA context creation, weight decompression and transfer, and KV cache allocation. Once the model is warm, subsequent requests from the same instance bypass most of these steps and travel straight to the forward pass.
The distinction matters because LLM weights are unusually large. A 70B parameter model at FP16 requires roughly 140 GB of VRAM, and a 671B Mixture-of-Experts model can demand substantially more. Moving these weights from disk or network storage into HBM is bounded by PCIe or interconnect bandwidth, not compute. The result is a fixed cost that no amount of optimization in the attention kernel can eliminate.
Why Cold Starts Happen
Cold starts are almost always a side effect of cost optimization. GPU time is expensive, so serverless inference platforms often scale workloads to zero when request volume drops. While this saves money during idle periods, it guarantees that the next request pays the full loading tax. Even in always-on clusters, cold starts can occur when:
- A new model replica is spun out to handle load spikes.
- A different model version or LoRA adapter is swapped into the same GPU.
- The KV cache is evicted and must be rebuilt for a new sequence length or batch size.
- A container crashes or is rescheduled onto fresh hardware.
Multi-node inference compounds the problem. When a single model is sharded across several GPUs or nodes, every participant must synchronize weights before the first forward pass. If one node is slow to join, the entire request stalls.
Impact on Production Systems
The most visible symptom of cold starts is variance in time-to-first-token, or TTFT. A healthy system might deliver a sub-second TTFT during peak hours, then spike to multiple seconds at low traffic. For streaming UIs, this manifests as a frozen typing indicator. For backend agents, it creates cascading timeouts.
Agentic workloads are especially vulnerable. A single task might chain ten tool calls across multiple models. If each hop has even a small probability of hitting a cold start, the tail latency of the entire chain degrades exponentially. Long-context requests make the problem worse because the KV cache initialization itself becomes a nontrivial memory operation. Users end up paying for latency twice: once in wall-clock time, and once in engineering hours spent adding retries, timeouts, and fallback logic.
Measuring Cold Start Latency
The simplest way to detect cold starts is to measure TTFT across a series of requests at irregular intervals. A stable system will show low variance; a serverless or oversubscribed system will show periodic spikes. Below is a lightweight script that records TTFT using the OpenAI SDK. Because Oxlo.ai is fully OpenAI SDK compatible, you can point the same client at https://api.oxlo.ai/v1 and observe consistent latencies on popular models.
import time
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_API_KEY"
)
def get_ttft(model, message):
start = time.perf_counter()
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": message}],
stream=True
)
# TTFT is the time until the first chunk arrives
first_chunk = next(response)
ttft = time.perf_counter() - start
return ttft
# Measure across intermittent requests
model = "llama-3.3-70b"
for i in range(5):
# Space requests apart to test for cold start behavior
time.sleep(30 if i > 0 else 0)
ttft = get_ttft(model, f"Ping {i}")
print(f"Request {i}: TTFT = {ttft:.3f}s")
On infrastructure with cold starts, you will often see the first or fourth request lag behind the others. On Oxlo.ai, popular models such as Llama 3.3 70B, Qwen 3 32B, and DeepSeek V4 Flash are kept warm, so TTFT remains stable even after idle gaps.
How Oxlo.ai Eliminates Cold Starts
Oxlo.ai takes a different approach from serverless inference providers. Rather than scaling popular models to zero, Oxlo.ai maintains always-on capacity for its flagship and most frequently used weights. This eliminates the weight-loading penalty entirely for models such as Llama 3.3 70B, DeepSeek R1 671B MoE, Kimi K2.6, and others.
The economic model makes this sustainable. Oxlo.ai charges a flat rate per API request, not per token. On token-based platforms, keeping a 70B model hot for sporadic traffic is economically hostile because long prompts generate high token counts and unpredictable bills. Oxlo.ai’s request-based pricing decouples infrastructure warmth from input length, so a long-context agent call costs the same as a short greeting. Developers get predictable bills and predictable latency simultaneously. For exact plan details, see the Oxlo.ai pricing page.
This design is particularly effective for workloads that mix context lengths. A coding agent might send a 50K token file followed by a 200 token follow-up. On token-based serverless tiers, the large prompt is expensive and may itself trigger a cold start if the provider optimizes for cost. On Oxlo.ai, both requests pay one flat fee, and both hit a warm model.
Choosing Infrastructure for Predictable Latency
When evaluating an inference provider, ask two questions. First, what is the P99 TTFT after five minutes of inactivity? Second, does the pricing model reward or punish the context lengths your application actually uses?
If your workload involves long documents, agent loops, or intermittent traffic, cold starts are not just an annoyance. They are a structural tax on your architecture. Oxlo.ai removes that tax by keeping 45+ models warm across LLMs, code models, vision, audio, and embeddings, with full OpenAI SDK compatibility and no cold starts on popular variants. You point your existing client at https://api.oxlo.ai/v1, and the first token arrives on the same predictable schedule as the hundredth.
Top comments (0)