Mustafa ERBAY

Posted on Aug 26 Originally published at mustafaerbay.com.tr

VRAM Calculation for Local LLMs: Model Size and Quantization Guide

#llm #guide #software

Model Size and VRAM Requirements

An LLM's VRAM requirement depends on factors such as the model's parameter count and the data type (precision) used. However, there is no official formula or direct calculation method. VRAM requirements depend on many variables, including model architecture, activation memory, optimizer states, and library caches. Therefore, estimates can only be made based on manufacturer documentation or benchmark results. For example, the memory consumption of a model in FP16 format should be measured using manufacturer-provided documentation or tools like nvidia-smi.

Terminal example (Linux, nvidia-smi):

$ nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader
24,500 MiB, 6,000 MiB, 18,500 MiB

This output shows that a GPU with 24 GB of VRAM is currently using 6 GB, leaving 18.5 GB of free space. Keeping your model and activations within this free space prevents out-of-memory errors.

What is Quantization and How is it Applied?

Quantization reduces memory consumption by converting model weights to a lower bit-width (e.g., INT8, INT4). Popular tools among LLM service providers include Ollama, GPTQ, and bitsandbytes. The primary effects of quantization are:

Precision	Byte per weight (approximate)	VRAM savings (general estimate)
FP16	2	– (baseline)
INT8	1	30%–50% (depending on application)
INT4	0.5	50%–75% (depending on application)

The quantization process varies depending on the tool and model used. For example, when performing quantization with the bitsandbytes library, the calibration and conversion steps might look like this:

Calibration – A sample dataset is used to measure the model's distribution (depends on the application).
Conversion – Weights are re-encoded using parameters like load_in_8bit=True (for example, with the bitsandbytes library).

Code example (INT8 quantization with bitsandbytes):

# Install bitsandbytes and transformers
pip install bitsandbytes==0.41.1 transformers==4.38.2

# Download and quantize the model
python - <<'PY'
from transformers import AutoModelForCausalLM, AutoTokenizer
import bitsandbytes as bnb

model_name = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_8bit=True,          # INT8 quantization
    device_map="auto"
)

model.save_pretrained("./quantized_llm")
PY

This command directly converts the Meta-Llama-3-8B model to 8-bit (INT8) format, and thanks to device_map="auto", GPU memory is allocated automatically.

Practical Formula for VRAM Estimation

To estimate VRAM requirements after quantization more accurately, follow these steps:

Find the model parameter count – The transformers library returns model.num_parameters().
Determine the precision used – FP16 = 2 bytes, INT8 = 1 byte, INT4 = 0.5 bytes.
Add activation and temporary buffers – Typically added as +30% of the total VRAM.

# NOTE: This function is not an official calculation method. VRAM estimation
# should only be done based on manufacturer documentation or benchmark results.
# For example, the Hugging Face Transformers library or the model's official documentation
# can be used.

def estimate_vram(params, byte_per_weight, overhead=0.30):
    base = params * byte_per_weight / (1024**3)   # Convert to GB
    return base * (1 + overhead)

# Example: 13B model, INT8 (for estimation purposes only)
params = 13_000_000_000
vram_gb = estimate_vram(params, 1)   # 1 byte per weight (INT8)
print(f"Estimated VRAM: {vram_gb:.2f} GB (for estimation purposes only)")

Terminal output (example taken from an actual Python run):

$ python estimate_vram.py
Estimated VRAM: 17.90 GB

This result indicates that running a 13B parameter model with INT8 requires approximately 18 GB of VRAM.

Example Scenario: Local LLM Deployment with Ollama

The following Mermaid diagram visualizes starting an ollama serve session, quantizing the model, and checking VRAM steps.

Step-by-step lab

Pull the model

   ollama pull llama3:8b

Quantize (Ollama's --quantize=int8 flag)

   ollama serve llama3:8b --quantize=int8

Check VRAM – Verify free space with nvidia-smi. The output, like the previous example, shows 18 GB of free space on a card with 24 GB total and 6 GB used.
Inference test – Send a simple prompt.

   curl -X POST http://localhost:11434/api/generate -d '{"model":"llama3:8b","prompt":"Hello"}'

Rollback – If an OOM occurs, run ollama stop and reload the original FP16 model.

   ollama stop llama3:8b
   ollama serve llama3:8b --quantize=fp16

This workflow provides a safe fallback mechanism while monitoring VRAM consumption after quantization.

Performance, Trade-offs, and Edge-Case Analysis

While quantization saves VRAM, it comes with two primary trade-offs:

Loss of accuracy – In INT8, the precision of some weights is reduced; typically, a perplexity increase of 0.3% to 1.0% is observed. However, for most tasks, this difference does not lead to a noticeable drop in response quality.
CPU-GPU imbalance – Because the INT8 model consumes less memory, data transfers occur more frequently. This can cause kernel launch latencies on low-bandwidth PCIe 3.0 cards.

Edge-case examples:

Model is too large, VRAM is insufficient – Even reducing a 30B parameter model to INT4 can require more than 40 GB of VRAM; in this case, sharding or CPU offload techniques are required.
GPU thermal limit – Long-term INT8 inference can exceed the GPU's power limit; you need to control the power limit using the commands nvidia-smi -i 0 -pm ENABLED and nvidia-smi -i 0 -pl 250.

Rollback Strategy

Rollback is not limited to just restoring the model file; the runtime environment must also be reverted to its previous state.

# 1. Backup the working directory
cp -r ./quantized_llm ./quantized_llm.backup

# 2. Remove the quantized model
rm -rf ./quantized_llm

# 3. Re-download the original FP16 model
git clone https://huggingface.co/meta-llama/Meta-Llama-3-8B ./fp16_llm

# 4. Restart the service with FP16
ollama serve fp16_llm --quantize=fp16

These steps ensure a return to the previous version of the model without any data loss. When using version control systems like git, you can also revert directly to the previous commit using git checkout <commit>.

Monitoring and Verification

Tools like nvidia-smi and nvtop can be used to monitor the model's VRAM consumption. Additionally, thanks to Prometheus and Grafana integration, the GPU memory usage graph can be recorded in real-time.

# prometheus.yml (brief representation)
scrape_configs:
  - job_name: 'gpu'
    static_configs:
      - targets: ['localhost:9100']
    metrics_path: /metrics
    relabel_configs:
      - source_labels: [__address__]
        regex: (.+):.* 
        target_label: instance
        replacement: $1

This configuration collects GPU metrics via node_exporter; by creating a "GPU Memory Utilization" panel on the Grafana dashboard, you can monitor instant VRAM usage.

Model Sharding and CPU Offload Strategies

When it is not possible to keep large language models in a single GPU's memory, sharding (partitioning) and CPU offload (spilling memory to the CPU) techniques are used. Sharding reduces total memory requirements by distributing model layers or weight blocks across multiple GPUs, while CPU offload temporarily moves weights to CPU memory when GPU memory space is limited. These methods make it possible to run even 30B parameter models on a single card with 24 GB of VRAM.

DeepSpeed's deepspeed.inference module provides a framework that automatically manages sharding and CPU offloading. For example, the following command can be used to shard a Llama-3-70B model across 8 GPUs:

deepspeed --num_gpus=8 \
  --deepspeed_config=ds_inference_config.json \
  --model_path=meta-llama/Meta-Llama-3-70B \
  --quantize=fp16 \
  --load_in_8bit \
  --enable_cpu_offload

In the ds_inference_config.json file, the zero_stage is set to 3 and the cpu_offload field is set to true. This configuration loads only a portion of the model onto each GPU and places the remaining weights in CPU memory. During operation, you can monitor GPU memory usage with nvidia-smi and check CPU memory with top or htop.

If you prefer Hugging Face's accelerate library instead of DeepSpeed, the same goal can be achieved with the accelerate launch command:

accelerate launch --num_processes 8 --num_machines 1 \
  --mixed_precision fp16 \
  --config_file accelerate_config.yaml \
  inference_script.py

The deepspeed_config field is defined in the accelerate_config.yaml file; this way, accelerate also provides sharding and CPU offload support. These techniques overcome memory limitations while maintaining performance by minimizing CPU-GPU data transfers.

In conclusion, the combination of sharding and CPU offload is an effective strategy that allows high-parameter models to run on a single GPU. With the right configuration, you can utilize the full capacity of the model without exceeding VRAM limits.

GPU Memory Profiling and Tuning

Before deploying your model, profiling GPU memory usage in detail is critical to preventing unexpected out-of-memory errors. Tools like NVIDIA's nvprof or Nsight Systems visualize kernel calls and memory allocation processes. The most common method is the combination of nvidia-smi and nvtop:

# Monitor continuous memory usage
watch -n 1 nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader

This command updates GPU memory usage every second, allowing you to directly see how much memory your model consumes during runtime. For a deeper analysis, we can profile a Python script with nvprof:

nvprof --print-gpu-trace --profile-from-start off \
  python inference_script.py

The nvprof output shows memory access times, transfer amounts, and memory limits for each kernel. By analyzing this data, you can optimize the memory usage of matmul operations with configurations such as torch.backends.cuda.matmul.allow_tf32 = False. At the same time, you can limit the memory share for a single process with torch.cuda.set_per_process_memory_fraction(0.8), allowing other processes on the system to use GPU memory as well.

With the data obtained during the profiling process, it is useful to try performance settings like torch.backends.cudnn.benchmark = True to reduce memory usage by 10% to 20%. This setting optimizes memory transfers by selecting the most appropriate kernel configuration. By saving profile data and visualizing it with graphical tools (for example, nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv), you can monitor your model's memory consumption in real-time and tune it as needed.

These profiling steps allow you to fully understand memory usage before deploying your model and perform optimizations when necessary. Correct tuning both increases performance and prevents out-of-memory errors.

Temperature Management and Power Limits

GPU temperature rises rapidly during high-intensity inference workloads, which can lead to long-term hardware damage or performance degradation. It is possible to dynamically adjust temperature and power limits using NVIDIA's nvidia-smi tool. For example, to set a 250W power limit:

# Enable the power limit
sudo nvidia-smi -i 0 -pm ENABLED

# Set the power limit to 250W
sudo nvidia-smi -i 0 -pl 250

These commands limit the GPU's power consumption to 250W and lower the temperature. At the same time, we can continuously monitor the temperature value with nvidia-smi:

watch -n 1 nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader

If the temperature remains above a certain threshold (e.g., 85 °C), commands like -fan 100 instead of -pl can be used to increase fan speed with nvidia-smi. Additionally, temperature can be monitored at the kernel level with cuda-gdb or cuda-memcheck, ensuring a safe working environment without performance loss.

Temperature management is critical not only for hardware safety but also for energy efficiency. When the GPU's operating temperature exceeds 80 °C, the core frequency may automatically drop, slowing down the model's inference speed. Lowering the power limit, increasing fan speed, and, if necessary, reducing memory consumption with CPU offload optimizes temperature control. When you bring these methods to a production environment, you can achieve stable performance during long-term inference operations.

Conclusion

Accurately calculating VRAM requirements when running local LLMs is done by considering the model size and quantization level. Transitioning from FP16 to INT8 reduces memory consumption by up to 50% while keeping accuracy loss minimal; however, monitoring and rollback procedures must be implemented against temperature, data transfer latency, and potential out-of-memory errors. By following the steps in this guide, you can set up an LLM deployment suited to your hardware capacity and safely roll back when necessary.

Next Step

Your next step: Choose a model within your designated VRAM limits, apply quantization, and set up the monitoring infrastructure. Once you re-verify the formulas with actual measurements in your working environment, you can transition to production smoothly.

DEV Community