DEV Community

mujin
mujin

Posted on Edited on

Every VRAM dial on a 4GB card, ranked by what it actually buys

#ai #machinelearning #performance

Stable Diffusion 1.5 in fp32 on a GTX 1650 with 4GB. About 150 seconds per 512x768 image with model offload on. Every number below is measured on that card, not estimated, because on 4GB the difference between "fits" and "doesn't" is a few hundred MB.

1. The four knobs, cost and savings

Knob Wall-clock cost Peak VRAM saved
fp32 → fp16 none measured roughly a third of the weights
Attention slicing a few seconds per step enough to matter at long prompts
VAE tiling only during decode the whole VAE pass
Model offload the dominant cost of the 150s the weights between steps

I can't give you absolute GB for each because I didn't isolate them cleanly; what I can tell you is the ranking is stable. Offload is the only one that turns a hard OOM into a working render. fp16 is the only one that costs nothing you can measure. Slicing and tiling are cheap insurance you leave on.

The 150 seconds per 512x768 is the number to beat. That is the fp32-with-offload figure. I never got fp16 to render a piece I shipped.

2. The order to try them

Cheapest first. Never touch offload until the three below it have failed.

  1. fp16. Free. Try it first every time.
  2. Attention slicing. Costs seconds. Removes the attention peak, which is what usually spikes on a batch.
  3. VAE tiling. Costs nothing until decode. Removes the decode spike, which is where a batch that trained fine dies.
  4. Model offload. Costs you the render budget. Last resort, and it works.

The command that tells you whether a change bought anything:

python -c "
import torch
print(torch.cuda.max_memory_allocated()/2**20, 'MiB peak')
"
Enter fullscreen mode Exit fullscreen mode

Reset the peak counter between runs or you read the high-water mark from the last attempt:

torch.cuda.reset_peak_memory_stats()
Enter fullscreen mode Exit fullscreen mode

I run the render, then print the peak, then change exactly one knob and run it again. One knob per run. If you change two and it fits, you don't know which one earned it.

3. The knob that looked free

VAE tiling. It costs nothing in wall-clock, so I assumed it was free and left it on. It is not free in quality. Tiled decode seams at tile boundaries. On a 512x768 render the seam runs through a flat region and you will not see it at thumbnail size.

I saw it because I crop. A 1:1 crop at the tile boundary shows a visible discontinuity in gradient where the untiled decode is smooth. That crop is what made me stop trusting "no measurable cost" as "no cost." Wall-clock and VRAM are easy to log. Quality is not, so it needs its own check, and cropping is the check.

Rule I use now: tiling on for drafts, off for anything that goes on chain. If it OOMs with tiling off, fix the batch size, not the knob.

The other thing I log is on-chain cost: about 0.0000008 ETH per piece to preload, mint price 0.00063 ETH on Base. I generated and pinned 77 pieces, minted 7, 70 still to mint, 5 per wallet. The test suite is around 271 tests, which is how I keep the render path stable while I change knobs.

Takeaway: before you reach for offload, print peak VRAM after a single knob change and write the number down. If you didn't log it, you didn't measure it, and you'll be back on the same dial next week.

Collection: https://thunderous-sunflower-a910b4.netlify.app

Top comments (0)