When I started building Pixova, the first infrastructure question was the most expensive one: how do I run GPU inference without going broke?
Here's what I tested, what it cost, and what I'd choose again.
The Two Options
| Dedicated GPU | Serverless GPU | |
|---|---|---|
| Pricing model | Per hour (running or idle) | Per second of actual use |
| Cold start | None | 3–8 seconds after idle |
| Scale to zero | ❌ No | ✅ Yes |
| Setup complexity | Medium | Low |
| Predictable cost | ✅ Yes | ❌ Varies with traffic |
| Best for | Consistent high traffic | Variable/unpredictable traffic |
The Math on Dedicated
A mid-range A10G GPU on a major cloud provider runs roughly $1.50–2.50/hour.
That's $1,080–1,800/month whether anyone uses the tool or not.
For an early-stage product with unpredictable traffic, you're paying full price for a server that sits idle most of the day. A tool that gets 200 generations per day has maybe 30–40 minutes of actual GPU work happening in 24 hours.
The rest is dead cost.
The Math on Serverless
RunPod serverless charges per second of GPU execution. For image generation, a single inference run takes roughly 3–8 seconds of GPU time depending on settings.
At typical serverless rates, that's fractions of a cent per generation.
The catch: cold starts. When the endpoint sits idle and a request comes in, it has to spin up a container before inference can begin. This adds 5–15 seconds to the first generation after a quiet period.
For a user experience standpoint, that's noticeable. For a cost standpoint, it's transformative.
What I Actually Did
I run Pixova on RunPod serverless. The decision came down to one question:
What's the cost of a cold start versus the cost of keeping a GPU warm 24/7?
Option A — Always warm:
Keep a minimum of 1 worker running at all times to eliminate cold starts. Adds roughly $30–50/month depending on GPU tier.
Option B — Scale to zero:
Accept cold starts, pay only for actual inference. Near-zero cost during quiet periods.
For traffic patterns that spike unpredictably, I chose Option B with a scheduled warm-up ping every 15 minutes during peak hours. This significantly reduces cold starts during active periods without paying for 24/7 uptime.
The Hidden Costs Nobody Talks About
Image delivery
Generated images need to go somewhere before hitting the user's browser. Options:
Option 1: Return base64 directly → simple, but large payloads
Option 2: Upload to storage, return URL → adds latency, adds cost
Option 3: Signed temporary URLs → best UX, slightly complex
I use Cloudinary for delivery optimization. The cost is minimal for typical traffic but worth factoring into the total.
Egress fees
Cloud providers charge for data leaving their network. GPU outputs (images) are large. At scale, egress adds up faster than compute.
Concurrency limits
Serverless endpoints have maximum concurrent worker limits. Traffic spikes beyond that limit queue — users wait. Plan your concurrency ceiling based on your expected peak, not average.
When Dedicated Makes Sense
Dedicated GPUs make sense when:
- You have consistent, predictable traffic that fills the GPU most of the day
- Latency requirements are strict — cold starts are unacceptable
- You're running custom models that take minutes to load (not viable on serverless)
- You need GPU resources for multiple services simultaneously The crossover point is roughly when your serverless bill approaches 60–70% of the equivalent dedicated cost consistently. At that point, predictability becomes worth paying for.
TL;DR
| Scenario | Choose |
|---|---|
| Early stage, variable traffic | Serverless |
| Consistent high volume | Dedicated |
| Need zero cold starts | Dedicated or warm minimum workers |
| Running custom large models | Dedicated |
| Minimal ops overhead | Serverless |
For the full breakdown of how Pixova is built — tech stack, architecture decisions, and what broke — I wrote a detailed technical post here.
What GPU infrastructure are you running for AI workloads? Drop it in the comments.
Top comments (0)