Rent a GPU for open-weight models: do the VRAM math first

#llm #ai #devops #python

Disclosure: this post contains affiliate links; I may earn a commission at no cost to you.

TL;DR

Before you rent a GPU for an open-weight model, do the VRAM math. Weights take roughly parameters x bytes per weight, then add 15 to 25 percent for KV cache and runtime. A 70B model at 4-bit fits one 80 GB card. A 8B model fits a cheap 24 GB card. Rent by the second, stop the pod when idle, and compare against API pricing before committing.

The VRAM calculator (copy-paste)

def vram_gb(params_b, bits=4, overhead=0.2):
    weights = params_b * bits / 8          # GB for weights
    return round(weights * (1 + overhead), 1)

for name, p in [("8B", 8), ("32B", 32), ("70B", 70)]:
    print(name, {b: vram_gb(p, b) for b in (16, 8, 4)})

Model size	BF16 (16-bit)	8-bit	4-bit	Fits on
8B	19.2 GB	9.6 GB	4.8 GB	one 24 GB card
32B	76.8 GB	38.4 GB	19.2 GB	4-bit on 24 GB, 8-bit on 48 GB
70B	168 GB	84 GB	42 GB	4-bit on one 80 GB card

Long contexts and many concurrent users grow the KV cache, so treat the table as a floor, not a promise.

Launch an OpenAI-compatible endpoint with vLLM

pip install vllm
vllm serve <your-huggingface-model-id> \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --api-key $MY_KEY

Then point any OpenAI SDK at http://<pod-host>:8000/v1. For quantized weights add --quantization awq.

What it costs (assumptions stated)

RunPod Secure Cloud on-demand rates seen on runpod.io/pricing in late October 2026, billed per second (they move often): A100 80GB about $1.79/hr, H100 PCIe about $2.89/hr.

Worked example: test a 4-bit 70B model for 4 hours on an A100 80GB, then stop the pod: about $7. Left running all month at the same rate: about $1,290 (1.79 x 720 h). The money-saving habit is simply stopping the pod. Serverless endpoints bill only while workers run, but their hourly rate is higher, so check utilization first.

Situation	Better choice
Spiky, a few hours a day	API, or serverless GPU
Steady high volume, one model	Dedicated pod
Data must stay in your control	Rented GPU
Fine-tuned or niche model	Rented GPU

Try it

To test this without buying hardware, spin up a pod on RunPod here and be serving a model in about ten minutes. Start with a 24 GB card for 8B models and step up only when the math says you must.

Full disclosure: TokenLean participates in affiliate programs, including RunPod's. If you sign up or buy through links on this page, I may earn a commission at no extra cost to you. Prices are from public pages at the time of writing and may change.

DEV Community