DEV Community

aco dog
aco dog

Posted on

I ran a GPU inference app for a month on Azure serverless GPU. Here's the actual bill.

#azure #gpu #cloud #devops

I run a small internal speech-synthesis service on an NVIDIA T4 via Azure Container Apps' serverless GPU, with min-replicas=0. The pitch is that you pay nothing while nobody's using it.

I wanted to know if that's actually true, so I pulled the real numbers out of Azure Cost Management, scoped to just that resource group.

Short answer: yes, it's true. But the biggest line item on my bill wasn't the GPU.

The setup 

[browser] → [Container App (T4, min=0 → max=1)]
               ├─ web UI      (FastAPI + uvicorn, :8000)
               └─ inference   (:9880)
Enter fullscreen mode Exit fullscreen mode

Idle for ~5 minutes and it scales to zero replicas. Next request wakes it back up. Cold start is about 5 minutes in my case — a CUDA base image plus loading model weights.

The actual numbers

Measured over 48 days, scoped to one resource group. (Billed in JPY; USD figures are approximate.)

Service Per month Notes
Container Apps (GPU) $2–7 Only charged on days I used it
Container Registry ~$11 Fixed cost. The biggest line item.
Log Analytics $0

Total: around $13/month, including the month where I was actively building and testing.

"You only pay when you use it" is literal

Here's the daily distribution. Out of 48 days, GPU charges appeared on only 12.

Charge that day Days
$0 (unused) 36
$0.01–0.20 4
$0.20–0.80 5
~$1.05 2
$2.30 1

A day of real use costs somewhere between four cents and a dollar. An always-on T4 VM of comparable spec runs a few hundred dollars a month. That's close to two orders of magnitude.

The tradeoff is the cold start. For an internal tool where people know to expect a few minutes on first use, that's an easy trade. For a customer-facing endpoint, it isn't.

The $2.30 day was me forgetting to scale back down

That one outlier day has a boring explanation. I deployed a new image and temporarily forced the app to stay up so I could read the startup logs:

# force it up to inspect logs
az containerapp update -n $APP -g $RG --image $ACR.azurecr.io/my-app:2 \
  --min-replicas 1 --max-replicas 1

# ...and this is the line people forget
az containerapp update -n $APP -g $RG --min-replicas 0
Enter fullscreen mode Exit fullscreen mode

In a min=0 setup, the expensive mistake is never the GPU's hourly rate. It's leaving min-replicas 1 on after a debugging session. Put the scale-back-down step in your deploy checklist.

Checking whether you're being billed right now

The Azure portal showing "Status: Running" means the app exists, not that a replica is up. That confused me early on. The CLI tells the truth:

# empty output = no replicas = not being billed
az containerapp replica list -n $APP -g $RG -o table

# ScaledToZero or Running
az containerapp revision list -n $APP -g $RG \
  --query "[?properties.active].properties.runningState" -o tsv
Enter fullscreen mode Exit fullscreen mode

The actual surprise: the registry cost more than the GPU

Container Registry was ~$11/month against the GPU's $2–7. Two reasons compounding:

The images are huge. A CUDA base image plus bundled model weights runs several tens of GB per image. I bundle the weights deliberately — downloading them at startup would add minutes to an already slow cold start.

Old images never leave. I tag every deploy (my-app:1, :2, :3…), and nothing removes the old ones. Storage billing is on the total, so the bill creeps up with every deploy.

The fix is housekeeping. Keep what's deployed plus one for rollback:

# what's actually running
az containerapp show -n $APP -g $RG \
  --query "properties.template.containers[0].image" -o tsv

# tags, newest first
az acr repository show-tags -n $ACR --repository my-app --orderby time_desc -o tsv

# drop the old ones
for t in 1 2 3 4 5 6 7 8; do
  az acr repository delete -n $ACR --image my-app:$t --yes
done
Enter fullscreen mode Exit fullscreen mode

I went from 10 images down to 2. Storage billing catches up over a few hours, via the registry's garbage collection.

If you're evaluating serverless GPU, budget for registry storage. It's the line item nobody warns you about, and for a low-traffic workload it can quietly become your largest one.

What I'd tell someone starting out

  • Zero idle cost is real. A day of use costs cents, not dollars.
  • It gets better the less you use it. Bursty, low-frequency workloads are the sweet spot.
  • Cold start (~5 min for a big CUDA image) decides whether this fits your use case. Be honest about that up front.
  • Your worst cost accident will be forgetting --min-replicas 0.
  • Watch registry storage, not just GPU seconds.

Getting from zero to a working deployment took considerably longer than it should have. Pinning torch against transformers' requirements, a 504 caused by the order services start in, and — on Windows — az acr build crashing the CLI on a character encoding issue mid-build. None of that is in the docs.

I wrote all of it up, including the full deployment walkthrough, here:

Zero-Idle GPU: Running Inference on Azure Container Apps

Happy to answer questions in the comments.

Top comments (0)