Cold start latency in GPU inference is the delay a new replica adds before it can serve its first token: pulling the container image, loading model weights into GPU memory, and initializing CUDA (Compute Unified Device Architecture) and the serving engine. You avoid it by keeping a request from ever triggering that work, either by using warm capacity or by using a platform where the model is already loaded. DigitalOcean's Inference Engine covers both with Serverless Inference and Dedicated Inference.
Key takeaways:
- Cold starts are a startup pipeline: image pull, weight loading, then CUDA setup.
- Scaling to zero cuts idle GPU costs, but the first request after a quiet period waits through the entire startup.
- DigitalOcean Dedicated Inference keeps your model loaded on your own GPUs, while Serverless Inference handles spiky traffic.
Where cold start delays come from
A cold start happens when a request lands on a GPU that isn't ready for it. The replica pulls a container image that can run to several gigabytes in size, fetches weights, copies them into GPU memory, then initializes CUDA and often compiles kernels. Bigger models take longer at every step. Scale to zero, or a burst that needs a new replica, sends real user traffic into that pipeline.
How to avoid cold starts
Keep at least one replica running for latency-sensitive endpoints. That’s the trade-off: a warm replica means paying for GPU time while it sits idle, usually billed by the hour or second. Scaling to zero avoids that cost, but you pay for it in first-request latency, instead. On some platforms you’re billed for startup time, too.
How DigitalOcean helps you avoid cold starts
Dedicated Inference hosts open-source or Bring Your Own Models (BYOM) on dedicated GPUs billed per GPU-hour, so the model stays loaded between requests. You're paying for always-on capacity, which pays off when traffic is steady and high-throughput. Provisioning happens once at deploy, not when a user asks.
Serverless Inference fits unpredictable traffic and is billed per token, so you aren't paying for idle GPUs. You call hosted models over an API with no image or weights to load.
Built-in inference observability shows real-time and historical latency, throughput, and error rates in the Analyze tab of the DigitalOcean Control Panel. Watch time to first token (TTFT): it captures queue wait and prefill latency, so delays before generation starts show up there.
GPU inference cold start FAQ
How to reduce cold start latency when serving open-source LLMs?
Keep a replica warm, store weights on fast storage next to the GPU, and run a warm-up request before routing traffic. On DigitalOcean, Dedicated Inference keeps your model loaded on dedicated GPUs so requests don't trigger a load.
Which serverless inference providers handle cold starts well for open-source models?
Look for a provider that pairs serverless inference with a dedicated tier, so latency-critical traffic can leave scale-up paths. DigitalOcean Inference Engine offers both. Measure time to first token on your own traffic for a baseline.
Which inference provider lets me move from a serverless prototype to a dedicated production endpoint without re-architecting?
DigitalOcean's Inference Engine brings Serverless, Batch, and Dedicated Inference under one OpenAI-compatible API style. Start on Serverless Inference, then move steady traffic to Dedicated Inference as it grows.
References & further reading
- Learn more about DigitalOcean Inference Engine and its Serverless, Batch, and Dedicated tiers.
- Our Dedicated Inference technical deep dive is worth bookmarking for the architecture behind it.
- The hidden cost of cold starts in serverless AI workloads covers the trade-offs in more depth.
- Learn more about the safetensors format and fast model loading.
Top comments (1)
The tip about running a warm up request before routing traffic is the one people skip. A replica can report healthy once the server process is up while weights are still loading, so the first real user absorbs the delay anyway. Gating readiness on a completed generation closes that gap.
On a live voice call that delay is dead air.
Does the TTFT metric in the Analyze tab separate queue wait from prefill, or is it one combined number?