Deploying open-source LLMs (like Llama-3) or real-time Whisper transcription in production often forces a difficult architectural trade-off: keep dedicated GPUs running 24/7 (expensive) or rely on serverless scale-to-zero (cold start latency penalty).
To evaluate container spin-up overhead, we benchmarked median cold start latencies and per-second execution costs across the major serverless GPU platforms.
Benchmark Results
| Provider | GPU | Median Cold Start | Equiv. Hourly Rate | Scale-To-Zero |
|---|---|---|---|---|
| Modal | A100 (40GB) | 1.8s | ~$2.85 / hr | Yes |
| RunPod Serverless | A100 (80GB) | 4.2s | ~$2.59 / hr | Yes |
| Replicate | A100 (80GB) | 6.5s | ~$4.14 / hr | Yes |
| Together AI | H100 Cluster | Instant (Pooled) | Token-based | N/A |
| Lambda Labs | A100 (80GB) | VM Boot (~45s) | $1.89 / hr | No |
Key Observations
- Snapshot Restoration: Modal's filesystem and memory snapshotting drastically cut container initialization down to under 2 seconds, making user-facing on-demand LLM calls viable.
- Cost-Efficiency: For asynchronous batch jobs or high-throughput queues, RunPod Serverless remains the most cost-effective choice per GPU second.
- Managed Inference: If you do not require custom CUDA extensions or proprietary container code, pooled inference APIs like Together AI provide instantaneous responses without cold-start engineering.
The full benchmark dataset, hardware configurations, and testing scripts are maintained at ServerlessGPUBench.
Raw benchmark metrics are also open-sourced on GitHub: awesome-serverless-gpu-latency.
Top comments (0)