Benchmarking Serverless GPUs: Modal vs RunPod vs Replicate Cold Starts (2026)

mrzitoun — Thu, 03 Sep 2026 20:13:13 +0000

Deploying open-source LLMs (like Llama-3) or real-time Whisper transcription in production often forces a difficult architectural trade-off: keep dedicated GPUs running 24/7 (expensive) or rely on serverless scale-to-zero (cold start latency penalty).

To evaluate container spin-up overhead, we benchmarked median cold start latencies and per-second execution costs across the major serverless GPU platforms.

Benchmark Results

Provider	GPU	Median Cold Start	Equiv. Hourly Rate	Scale-To-Zero
Modal	A100 (40GB)	1.8s	~$2.85 / hr	Yes
RunPod Serverless	A100 (80GB)	4.2s	~$2.59 / hr	Yes
Replicate	A100 (80GB)	6.5s	~$4.14 / hr	Yes
Together AI	H100 Cluster	Instant (Pooled)	Token-based	N/A
Lambda Labs	A100 (80GB)	VM Boot (~45s)	$1.89 / hr	No

Key Observations

Snapshot Restoration: Modal's filesystem and memory snapshotting drastically cut container initialization down to under 2 seconds, making user-facing on-demand LLM calls viable.
Cost-Efficiency: For asynchronous batch jobs or high-throughput queues, RunPod Serverless remains the most cost-effective choice per GPU second.
Managed Inference: If you do not require custom CUDA extensions or proprietary container code, pooled inference APIs like Together AI provide instantaneous responses without cold-start engineering.

The full benchmark dataset, hardware configurations, and testing scripts are maintained at ServerlessGPUBench.

Raw benchmark metrics are also open-sourced on GitHub: awesome-serverless-gpu-latency.

Benchmarking Real-Time Voice AI APIs: Cartesia vs Deepgram vs ElevenLabs (2026)

mrzitoun — Thu, 03 Sep 2026 19:32:42 +0000

When building conversational agents or real-time voice applications, latency is the defining metric. If Time-to-First-Byte (TTFB) exceeds 200ms, natural turn-taking breaks down and conversational interruption becomes clunky.

We recently recorded and aggregated median latency and pricing metrics across the primary streaming Text-to-Speech APIs using WebSocket connections (US-East endpoints, median across 1,000 requests).

Summary Table

Provider	Model	TTFB Latency	Pricing / 1M Chars	Real-Time Suitability
Cartesia	Sonic-3	85ms	$20.00	Excellent (Fastest turn-taking)
Deepgram	Aura-2	115ms	$15.00	Very Good (Lowest bulk cost)
ElevenLabs	Flash v2.5	135ms	$25.00	Best Voice Realism
PlayHT	PlayDialog	180ms	$25.00	Good
OpenAI	TTS-1	240ms	$15.00	Slower (Chunked HTTP)

Key Takeaways for Developers

Ultra-low latency: Cartesia Sonic-3 remains the fastest streaming engine for handling real-time interruptions and WebRTC voice bots.
Cost at scale: Deepgram Aura-2 provides the most competitive price per character when deploying high-volume voice automation pipelines.
Voice acting & naturalness: ElevenLabs Flash v2.5 remains the benchmark for emotional inflection, voice cloning nuance, and dialect stability.

The full benchmark dataset, methodology, and continuous updates are documented at VoiceAIBench.