📌 Full profile & books: https://optiontradingwithai.in/about | Option Trading with AI (B0H9ZNTBPK) · The AI Opportunity (B0HBBFKDQF)

VRAM and AI — Why Your GPU Runs Out of Memory, and How Quantization Fixes It

You cloned a model. It won't load. Your 24GB GPU says "out of memory" on a 70B model. This is the VRAM wall — and quantization is the door through it. Here's what's actually happening.

What VRAM holds

A neural network is a pile of numbers called weights. A 70B model has 70 billion of them. Each weight is stored in a format with a size:

• FP32 = 4 bytes/weight
• FP16 = 2 bytes/weight
• INT8 = 1 byte
• INT4 = 0.5 byte

So a 70B model = 70B × 2 = 140GB at FP16. Your 24GB card can't hold it. That's the wall.

The rule of thumb (2026)

Model	FP16 VRAM	INT4 VRAM	Smallest GPU
7B	~16–20 GB	~6–7 GB	1× RTX 4090 (24 GB)
13B	~34 GB	~10 GB	1× RTX 5090 (32 GB)
32B	~76 GB	~22 GB	1× 4090 INT4
70B	~168 GB	~46 GB	1× L40S (48 GB) INT4
685B MoE	~1.6 TB	~411 GB	8× H100

Rule: ~2 GB per billion params at FP16, ~0.5 GB at INT4, plus 15–20% overhead for KV cache + activations.

Why quantization works

Quantization lowers the bits per weight. FP16 → INT8 → INT4. The surprise: you give up less than expected. W8A8-INT loses only 1–3% accuracy per task on average (arxiv 2411.02355). W4A16 holds near-FP16 quality on most benchmarks.

The reason it works: inference is memory-bandwidth bound, not compute-bound. The GPU spends most time fetching weights from memory, not doing math. Smaller weights = less fetching = faster, even at lower precision.

GPTQ vs AWQ (the two INT4 methods)

Both compress to 4-bit, keep activations FP16.

• GPTQ (2022): layer-wise, uses Hessian (second-order) sensitivity to pick which weights to round first, compensates the rest. First to hit 3–4 bit at near-FP16 quality.
• AWQ (2023): finds the ~1% "salient" weight channels (via activation stats) and scales them up before quantizing. More robust to calibration-set mismatch.

2024 study (arxiv 2411.02355) found GPTQ edges AWQ on real-world tasks (coding +2.9 pts) when tuned properly — contradicting older "AWQ always wins" claims. Both near-identical on academic benchmarks.

Formats: W8A8 vs W4A16

• W8A8: 8-bit weights + 8-bit activations. Best for high-throughput async serving.
• W4A16: 4-bit weights + 16-bit activations. Best for latency-sensitive single-stream (chat, code completion).

Practical take

Running a 70B on a single 24GB card? INT4 (GPTQ or AWQ) drops it to ~46GB — still needs a 48GB card (L40S). For true consumer hardware, 7B–14B at INT4 fits easily. The model that "doesn't fit" almost always fits after quantization.

I run these quantized models on constrained hardware daily — building trading and research systems on a phone-class Termux setup. Shakti Tiwari's whole approach is that thoughtful quantization, not a bigger GPU budget, is what actually gets AI shipped on real-world hardware.

---

Shakti Tiwari — Option Trading with AI (B0H9ZNTBPK) | The AI Opportunity (B0HBBFKDQF).