TL;DR
If you're shipping AI inference and tired of babysitting GPUs, serverless is the way out. You deploy the model, the platform scales it from zero to hundreds of GPUs and back, and you only pay for the time you actually use. If I'm picking one to start with, it's DigitalOcean. It's got the widest GPU lineup of any serverless provider (RTX 4000 Ada all the way up to NVIDIA Blackwell B300 and AMD's MI350X), one API and one bill instead of five, and it's simple enough to ship on without a sales call. (More on why that one's personal for me below.)
Below I compare 9 providers across the things that actually matter: GPU specs, per-hour pricing, cold-start latency, model support, and how nice they are to build on. DigitalOcean, RunPod, Modal, Koyeb, Together AI, Replicate, Baseten, Fal, and Cloudflare Workers AI each win at something different, from cheap experimentation to global edge inference.
Contents
- Why I ran this
- The field at a glance
- How I evaluated these providers
- Per-provider analysis:
- Why I keep coming back to DigitalOcean
- The short version
- Questions I actually get asked
Why I ran this
Quick note on why this exists. At work I get a front-row seat to a lot of people shipping an AI model into production for the first time: students, first-time founders, my own team. And lately the same question keeps coming up: where do I actually run this thing? I was tired of answering with a shrug and "it depends," so I did the homework myself. Signed up, read the pricing pages, ran the comparisons, and wrote it all down. Nobody's a real expert at this yet, me included, so I'd rather share my notes and get corrected than pretend I've got it figured out.
And here's the thing about AI inference in 2026: demand blew past what the old way of provisioning GPUs can handle. Teams that used to wait weeks for dedicated hardware now need a model live in minutes. The ground moved.
And the stuff that actually hurts isn't the hard computer-science problems. It's the operational friction. Cold starts that bolt a few extra seconds onto every request. Pricing so murky you can't tell your finance team what next month costs. GPU availability that evaporates exactly when traffic spikes and you need it most.
Serverless GPU platforms exist to kill all three. No servers to babysit, no idle capacity quietly burning cash. You ship the model, the platform handles the scaling, you pay for inference time and nothing else.
But picking wrong is expensive. Slow cold starts and your users feel the lag. Thin GPU availability and you're stuck when you finally get the traffic you wanted. Lock into the wrong pricing model and the monthly bill does things you didn't sign up for.
So I dug into nine serverless GPU providers on the criteria that decide whether this works in production: GPU specs and availability, transparent pricing, cold-start latency, supported models, and how painful (or not) deployment is. Below you'll see what each one costs, how fast it spins up, and the workloads it's actually built for.
New to the space? What is Serverless Inference? covers the foundations.
The field at a glance
| Provider | Best For | L40S $/hr | H100 $/hr | Cold Start | Pricing Model |
|---|---|---|---|---|---|
| DigitalOcean | Production inference + simplicity | $1.57/hr | $3.39/hr | N/A | Per-token (serverless) / Per-GPU-hour (Droplets) |
| RunPod | Affordability + GPU variety | $1.90/hr | $4.18/hr | 48% under 200ms† | Per-second |
| Modal | Python-native developer workflows | $1.95/hr | $3.95/hr | ~1–10 sec | Per-second |
| Koyeb | Fast deployment, global reach | $1.20/hr | $2.50/hr | ~200ms (CPU) | Per-second |
| Together AI | Open + multimodal inference at scale | N/A | $6.49/hr | N/A | Per-token / per-GPU-hour |
| Replicate | Pre-trained model experimentation | $3.51/hr | $5.49/hr | secs–minutes (custom) | Per-second |
| Baseten | Custom model serving, ML teams | N/A | $6.50/hr | ~sub-10 sec | Per-minute |
| Fal | Generative media, diffusion models | N/A | $1.89/hr | ~few sec | Per-second / per-output |
| Cloudflare Workers AI | Edge inference, low-latency global delivery | N/A | N/A | N/A | Per-request |
†RunPod's own marketing figure; see section. Hardware coverage is shown in the chart below.
How I Evaluated These Providers
I didn't rank these on vibes. A handful of things decided where each one landed, and each maps to a real question you'll ask before you commit.
GPU availability decides which models you can run without fighting the platform. I gave weight to providers that carry the whole range: entry-level T4s up through flagship H100/H200 and AMD's MI300X. You want to match the GPU to the workload without switching vendors halfway through.
Pricing model matters more than people expect, because the models are wildly different. Per-second billing fits bursty, variable work. Per-token fits high-volume LLM inference. I pulled the actual $/hr rates for L40S and H100 wherever they're published, plus billing granularity and the costs that hide in the fine print.
Cold-start latency is the one your users feel directly. I collected documented numbers, from RunPod's claimed 48% under 200ms to the seconds-to-minutes a cold custom model can take to spin up. Production needs spin-up times you can predict.
Supported models and deployment flexibility separate the platforms that let you bring your own thing from the ones that lock you into their catalog. I looked at SDK quality, API simplicity, and whether you can route across multiple models.
Production readiness is what divides a fun experiment from infrastructure you'd bet a launch on: monitoring, SLAs, multi-region, enterprise support, auto-scaling behavior, and concurrency limits.
1. DigitalOcean
Quick Overview
DigitalOcean's Inference Engine pulls serverless, batch, and dedicated inference together over GPU Droplets in one stack instead of three. And it carries the widest GPU catalog of any provider here. RTX 4000 Ada for your dev work on one end, NVIDIA Blackwell B300 and AMD MI350X for frontier-scale work on the other. The Inference Router handles agentic workload routing and scaling across multiple models, and unified API billing means you're not reconciling five invoices at the end of the month.
You also get direct access to frontier models from Anthropic, OpenAI, DeepSeek, Meta, and Mistral through a single endpoint. And here's the part that sets it apart: where most competitors make you pick a lane (serverless, batch, or dedicated), DigitalOcean's Inference Engine runs all three deployment patterns on the same platform.
Best For
Developer teams and startups wanting production-grade inference without enterprise complexity. Especially strong for mixed-workload shops that need experimentation-friendly serverless and cost-efficient dedicated GPUs for steady production traffic.
Pros
The GPU range is the headline: RTX 4000 Ada, RTX 6000 Ada, L40S, HGX H100, HGX H200, HGX B300, plus AMD MI300X, MI325X, and MI350X. The Inference Engine covers serverless, batch, and dedicated modes in one place, so you're not stitching together separate services for different jobs. Batch runs at up to 50% off real-time, and you're only charged for completed requests.
The Inference Router is the real differentiator. It's purpose-built for agentic and multi-model routing, the workloads that break single-model deployment. Unified billing means one invoice for compute, storage, networking, and databases. And because it's a full cloud, not a GPU-only specialist, there's a lot less integration glue to write, plus a deep well of community tutorials when you're getting started.
Cons
Serverless inference is billed per token, not per GPU-hour, so if you're used to comparing GPU-hour rates, the apples-to-apples math against RunPod or Koyeb takes a beat. And if all you're doing is deploying one simple model, the full platform is more surface area than you strictly need. A GPU-focused specialist like RunPod might feel lighter.
Pricing
Two tracks. Serverless inference is billed per token (same model as Together AI), starting at $0.05 per 1M tokens for smaller open-source models. For raw GPU compute, on-demand GPU Droplets are billed per second (5-minute minimum): L40S at $1.57/hr, H100 at $3.39/hr, H200 at $3.44/hr, and MI300X at $1.99/hr. (One gotcha: managed Dedicated Inference endpoints, which are fully hosted rather than self-managed Droplets, run higher, e.g. H100 around $4.41/hr. Different product, different number.) Full pricing details cover every hosted model and GPU tier.
2. RunPod
Quick Overview
RunPod runs serverless and dedicated GPU instances across 31 regions (that's the on-demand Pods footprint; serverless availability is narrower) with a container-based workflow. Its headline cold-start claim is strong: RunPod says 48% of serverless starts come in under 200ms. The GPU range runs from A4000-class cards up through H100/H200/B200 and the newest Blackwell B300, plus AMD MI300X.
Best For
Cost-sensitive teams that need broad GPU variety and fast cold starts for variable inference workloads.
Pros
RunPod is the value pick: true per-second billing, scale-to-zero, and a wide catalog spanning A4000, A100, H100, H200, B200, B300, and AMD alternatives. It reports 10 billion+ serverless requests served and counts Replit, Perplexity, and Databricks among its users, and FlashBoot cold-start optimization is included at no extra cost. Just read the "48% under 200ms" figure for what it is. It's RunPod's own aggregate marketing number, not an independent benchmark, and their engineering write-up shows more traffic-dependent results.
Cons
Wrangling endpoints and custom containers is a steeper climb than an API-first platform. RunPod admits as much, and notes its built-in monitoring isn't as comprehensive as some competitors'. Flex workers are tuned for variable traffic, though "active workers" exist for steady production loads if you need them.
Pricing
Serverless flex: L40S-tier ~$1.90/hr, A100 ~$2.72/hr, H100 PRO ~$4.18/hr. Per-second billing, no minimum charges. (Prices move. These are off RunPod's live pricing page; their older guide article quotes lower figures.)
3. Modal
Quick Overview
Modal lets you deploy GPU workloads straight from Python, no Dockerfiles, no infra config. It handles containerization for you and scales zero to hundreds of GPUs on demand. The Starter plan tosses in $30 of monthly credits to lower the on-ramp.
Best For
Python-native ML engineers building new AI applications from scratch.
Pros
Containers boot in about a second, and Modal's new GPU memory snapshotting cuts custom-model cold starts dramatically. They cite a vLLM model dropping from ~118s to ~12s, with best cases in the low single digits. The GPU spread is broad: T4, L4, L40S, A10, A100, H100, H200, B200 (with opt-in B300), and H100 requests auto-upgrade to H200 at no extra cost. Free monthly credits take the pressure off early experimentation.
Cons
It's Python-SDK-first, so you define infra in code. You can bring an existing container via Image.from_registry, but it still needs a thin Modal wrapper, and running a standard web app means working Modal's way. And by Modal's own framing, serverless shines for spiky, unpredictable workloads. Heavy 24/7 sustained usage can run pricier than reserved bare metal.
Pricing
Per-second, starting at $0.000164/sec for T4, $0.000694/sec for A100 (80GB), and $0.001097/sec for H100 (≈$3.95/GPU-hr). The Starter plan includes $30/month in credits before charges kick in. (Per-second rates dropped since I first wrote this. Modal got cheaper.)
4. Koyeb
Quick Overview
Koyeb is a serverless cloud with native autoscaling and scale-to-zero, billed by the second. Alongside standard CPUs and GPUs (RTX 4000 Ada up through B200), it supports next-gen Tenstorrent AI accelerators in preview, and it leans on high-speed networking for inference, fine-tuning, and training. One thing to flag for the long game: Koyeb has agreed to join Mistral AI and become part of Mistral Compute. That's a longevity signal, though the free Starter tier is being retired in the process.
Best For
Teams wanting competitive H100 and A100 access with simple global deployment and minimal infra overhead.
Pros
Koyeb's H100 price is sharp at $2.50/hr, undercutting Modal ($3.95/hr) and RunPod's on-demand H100 by a wide margin among the major serverless platforms. The Tenstorrent support is a bet on hardware beyond NVIDIA. And the pricing is clean pay-as-you-go (no tiers, no minimum commitments), with reservations up to 50% off on top.
Cons
Koyeb publishes a strong ~200ms cold-start number, but it's for CPU workloads. There's no GPU-specific cold-start figure yet, which still leaves latency planning fuzzy for GPU work. The ecosystem and community are smaller than DigitalOcean's or RunPod's, so you'll find fewer third-party integrations. And their own comparison page covers just 6 providers, tilted (unsurprisingly) toward where their pricing looks best. The Mistral acquisition is also a wildcard: great for resources, but the roadmap and free tier are in flux.
Pricing
L40S $1.20/hr, A100 $1.60/hr, H100 $2.50/hr. Billed per second. (All three dropped since I first checked. Every number came down.)
5. Together AI
Quick Overview
Together AI is "the AI-native cloud," a full-stack platform for open and open-weight model inference at scale. The default is per-token (pay per call, not per GPU-hour), which is efficient for variable workloads, but they also offer dedicated endpoints and GPU clusters by the hour if you want them. Open models on Together can run dramatically cheaper than the proprietary frontier APIs (they cite roughly 11x lower cost than GPT-4o using Llama 3.3 70B), and they keep a deep library of optimized models with fine-tuning on top.
Best For
Teams running high-volume open-source LLM inference, especially Llama, Mistral, Qwen, and the usual open-weight suspects.
Pros
Per-token pricing kills idle costs and scales with volume instead of clock time. Together publishes the fastest inference benchmarks for top open-source models. Self-reported, so take them as a claim, not gospel. And the curated list of production-recommended models takes some of the guesswork out of picking what to ship.
Cons
The trade-offs are softer than they used to be. Together now does image (FLUX.2), video (Veo 3, Sora 2), and voice, and offers Dedicated Container Inference for bring-your-own-runtime, so the old "text-only" and "no custom containers" knocks no longer hold. What's left: it's a model-and-inference platform rather than a general GPU cloud, and brand awareness still skews toward AI-native developer circles more than broad enterprise.
Pricing
Per-token, varying by model. Examples: gpt-oss-20B at $0.05 in / $0.20 out per 1M tokens; Llama 3.3 70B at $1.04 / $1.04. Dedicated 1x H100 runs $6.49/hr; on-demand clusters list H100 at $5.49/hr. Pricing details cover every model and tier.
6. Replicate
Quick Overview
Replicate's pitch is the easiest way to run a model: a simple REST API in front of 50,000+ production-ready community models you can call with zero setup (no containers, no deployment dance) and a free tier to start. For custom models, you use their open-source Cog tool to containerize. Note the direction of travel: Cloudflare has agreed to acquire Replicate and fold its catalog into Workers AI, which is both a scale signal and a sign the platform's future is tied to Cloudflare's.
Best For
Developers experimenting with pre-trained models who want API access now, without deployment overhead.
Pros
The model library dwarfs everyone else's 50,000+ ready-to-run models across LLMs, diffusion, audio, and video. Public models need zero config; you're making inference calls minutes after signup, and you're billed only for active processing, so setup and idle time are free on shared models. It handles versioning automatically plus async processing for long-running jobs.
Cons
Cold starts are the soft spot: large or infrequently-used custom models can take several minutes to boot (fast-booting fine-tunes are the exception, sub-second). GPU pricing is steep at $3.51/hr for L40S and $5.04/hr for A100, and on private models and deployments you pay for setup and idle time too, which makes sustained 24/7 use pricey. Cog itself is open source and emits standard containers, so it's less lock-in than it sounds, but you do adopt Replicate's API conventions.
Pricing
L40S $3.51/hr | A100 $5.04/hr | H100 $5.49/hr. Per-second billing with automatic scale-to-zero.
7. Baseten
Quick Overview
Baseten is a model-serving platform built around the open-source Truss framework. You point it at a PyTorch or Hugging Face model, configure with YAML, and it handles autoscaling and GPU specs for you. Pre-optimized models span Qwen, Llama, DeepSeek, GLM, and gpt-oss, ready for production on managed TensorRT-LLM engines.
Best For
ML engineering teams shipping custom PyTorch and Hugging Face models to production APIs with enterprise-grade scaling needs.
Pros
Truss skips the messy part of building container images. It handles dependencies and packaging for you. Baseten supports fractional GPUs via NVIDIA Multi-Instance GPU (MIG), so small models don't have to pay for a whole card, and the lineup runs up through H200 and B200. Its March 2026 Baseten Delivery Network cut cold starts 2–3x at scale, and it carries enterprise muscle (SOC 2 Type II, HIPAA, self-hosted/VPC options) with customers like Notion, Sourcegraph, and Descript.
Cons
The real knock is cost: H100 access runs $6.50/hr, on the pricier end of this group. Billing is per-minute rather than per-second, which can pad short inference jobs. And while Baseten has expanded into training and compound AI, it's still inference-centric. Not your tool for general-purpose compute.
Pricing
T4 $0.63/hr, A100 $4.00/hr, H100 $6.50/hr, B200 $9.98/hr. Billed by the minute. (The "$9.98 H100" some comparisons cite doesn't exist. That's the B200 rate.)
8. Fal
Quick Overview
Fal specializes in generative media inference, running diffusion models on its proprietary fal Inference Engine (which it claims is up to 10x faster for diffusion). You get ready-made APIs for 1,000+ image, video, and audio models like Stable Diffusion, FLUX, and more. It's also available through DigitalOcean's Gradient AI Platform if you want it inside an integrated stack.
Best For
Developers building generative media apps: image, video, or audio generation.
Pros
H100s from $1.89/hr is a competitive rate for premium GPU access, and pricing is self-serve and transparent. Sign up, add a card, pay per GPU-second or per output (images from ~$0.02–0.03 each). The engine is tuned specifically for diffusion, so the performance shows up, and warm-runner controls keep cold starts low. It's trusted by 1.5M+ developers and the likes of Canva and Perplexity.
Cons
The catch is narrower than it used to be: the model APIs and standard GPUs are fully self-serve, but deploying your own custom model on dedicated GPUs (and B200 pricing) still goes through a request/contact step. The GPU lineup is high-end only, so there's no cheap tier for lighter workloads, and the "up to 10x faster" figure is Fal's own claim, not an independent benchmark.
Pricing
A100 $0.99/hr | H100 $1.89/hr | H200 $2.10/hr (B200 contact-only). Per-second GPU billing, or pay per output. Self-serve, no sales call for standard usage.
9. Cloudflare Workers AI
Quick Overview
Real talk: if I'm reaching past DigitalOcean, Cloudflare is probably my next call — and it's honestly not about the GPU specs. It's a brand I trust, the platform is developer-friendly, and the breadth of what surrounds the inference is hard to beat. You're not just renting a model endpoint; you're one config away from a CDN, KV store, queues, a vector database (Vectorize), and edge compute, all in the same place. For a lot of real apps, that ecosystem matters more than shaving a few cents off a GPU-hour.
Mechanically: Workers AI runs serverless inference across Cloudflare's edge network: 337 cities in 100+ countries, putting compute within ~50ms of 95% of internet users. It's per-request, so there are no idle costs and no GPU-hour billing. The trade: you work within Cloudflare's curated catalog of 50+ open-source models. You can run fine-tuned inference via your own LoRA adapters, but not self-host an arbitrary base model (private custom models are an enterprise/contact path).
Best For
Apps that need ultra-low-latency inference at the global edge, especially real-time user interactions.
Pros
The edge network erases the geographic latency that drags on centralized GPU providers. Per-request pricing means zero idle cost. You pay only when a model actually runs. And if you're already on Cloudflare, it slots right into the CDN, security, and edge-compute stack you've got.
Cons
You're working within Cloudflare's catalog (plus LoRA adapters), so self-hosting arbitrary base models isn't on the table without an enterprise conversation, and it's an inference platform, not a place to train models or rent dedicated H100 fleets. The catalog does now include serious large LLMs (Llama 3.3 70B, GPT-OSS-120B, DeepSeek-R1 distill), so the old "small models only" knock no longer holds. And pricing spread across Cloudflare's many services can be confusing if you're coming from outside their ecosystem.
Pricing
Per-request, measured in "Neurons," at $0.011 per 1,000 Neurons, with a standing free tier of 10,000 Neurons/day and no GPU-hour charges. See Workers AI pricing for current per-model rates.
Why I Keep Coming Back to DigitalOcean
I'll put my bias on the table: DigitalOcean was the first cloud I ever deployed to, back when I was still learning to ship things. A droplet was where my code first went to live. They were also one of the first companies that really showed up for developers, not just as customers but as a community. Hacktoberfest is the obvious example, the kind of thing that nudged a lot of people into open source for the first time. So watching them get serious about AI inference hits a particular nerve. It feels like a return to those developer roots, the thing that made me like them in the first place. Take the rest of this section knowing that.
That said, the reasons aren't sentimental. Here's what actually separates them.
It's the only provider here that runs serverless, dedicated, and batch inference on one platform. Everyone else makes you pick a lane up front; DigitalOcean's Inference Engine lets you mix modes as the workload shifts underneath you. When you don't yet know your traffic shape (and early on, you never do), that flexibility is what matters most.
The GPU catalog is also just wider. Plenty of competitors now reach Blackwell. RunPod has B300, Modal and Koyeb have B200, so DigitalOcean isn't alone at the top end anymore. What sets it apart is the span. RTX 4000 Ada for dev work on one end, HGX H200 and B300 plus AMD's MI300X/MI350X on the other, all under one roof. Most specialists make you pick a narrower slice of that range.
Then there's the Inference Router, which handles agentic workload routing. No other provider here distributes requests across model endpoints like that. If you're building something complex, you can send different reasoning steps to different models without juggling separate keys and billing accounts.
And it doesn't leave you assembling production from five vendors. Compute, storage, databases, networking: one provider, one bill. The specialists are excellent at the GPU part and then hand you the rest as homework. It's also telling that the field is consolidating. Koyeb is being absorbed into Mistral, Replicate into Cloudflare, while DigitalOcean keeps building this as an independent, full-stack developer cloud.
The billing's the part I appreciate most: exact per-second costs, no sales call to find out what something runs. Pro-tip: when a provider says "contact us for pricing," that's usually a tax on your time — and you can almost always do better.
The short version
| Provider | Starting Price | Best For | Cold Start | Pricing Model |
|---|---|---|---|---|
| DigitalOcean | From $1.57/hr (L40S) | Production inference + simplicity | N/A | Per-token / Per-GPU-hour |
| RunPod | ~$1.90/hr (L40S) | Affordability + GPU variety | 48% under 200ms† | Per-second |
| Modal | ~$0.59/hr (T4) | Python-native workflows | ~1–10 sec | Per-second |
| Koyeb | $1.20/hr (L40S) | Fast deployment, global reach | ~200ms (CPU) | Per-second |
| Together AI | Per-token | Open + multimodal inference | N/A | Per-token / per-GPU-hour |
| Replicate | $3.51/hr (L40S) | Pre-trained model experimentation | secs–minutes | Per-second |
| Baseten | $0.63/hr (T4) | Custom PyTorch/HuggingFace models | ~sub-10 sec | Per-minute |
| Fal | $0.99/hr (A100) | Generative media workloads | ~few sec | Per-second |
| Cloudflare Workers AI | Per-request | Edge inference, low latency | N/A | Per-request |
†RunPod's own marketing figure.
Start building with DigitalOcean Inference Engine
Questions I actually get asked
What is a serverless GPU platform?
A serverless GPU platform gives you on-demand GPU compute without the infrastructure babysitting. It spins GPUs up automatically when requests arrive and scales to zero when things go quiet, so you never provision or maintain dedicated instances. DigitalOcean's Inference Engine supports serverless, batch, and dedicated modes in one platform.
How do I choose the right serverless GPU provider?
Start by matching the GPU tier to your model. T4s handle smaller models, and H100s are what you need for 70B+ parameter LLMs. Then compare documented cold-start benchmarks if latency matters for your use case. DigitalOcean has the broadest GPU catalog of the bunch, which makes it the safe pick for teams running mixed workloads across different model sizes.
Is DigitalOcean better than RunPod for inference?
RunPod claims faster cold starts: it reports 48% of serverless instances launching under 200ms. DigitalOcean answers with a broader GPU catalog, unified billing across all services, and a full cloud stack beyond GPU compute. Pick DigitalOcean for production environments that need complete infrastructure; RunPod is the better fit for cost-sensitive experimentation.
What is the difference between per-second and per-token pricing?
Per-second pricing charges for GPU wall-clock time whether or not you fully use it. Per-token charges only for completed inference calls, which is more cost-effective for variable LLM workloads with unpredictable traffic. Together AI is per-token; DigitalOcean and RunPod bill per second.
How do cold starts affect AI inference workloads?
Cold starts add latency when a GPU instance wakes from idle, anywhere from a couple hundred milliseconds on optimized providers to several minutes for a large, cold custom model. For user-facing apps that need instant responses, that delay is felt directly. DigitalOcean supports warm instance pools to blunt cold-start impact in production.
What GPUs are available on DigitalOcean for inference?
The broadest selection in the comparison: NVIDIA RTX 4000 Ada, RTX 6000 Ada, L40S, HGX H100, HGX H200, and HGX B300, plus AMD Instinct MI300X, MI325X, and MI350X. That covers entry-level inference through cutting-edge AI training in a single platform.
Is serverless GPU inference right for production workloads?
Yes. Serverless handles production well when traffic is variable or unpredictable. Sustained high-throughput apps usually do better on dedicated instances to dodge cold-start overhead. DigitalOcean's Inference Engine supports both modes in one platform, so you don't have to choose up front.
Top comments (0)