GPUs have become an important part of modern computing. They are used in Artificial Intelligence, Machine Learning, scientific simulations and High Performance Computing (HPC).
When people discuss GPU performance, they often focus on the number of GPU cores, processing power or memory capacity. However, there is another important part that can significantly affect performance: GPU memory.
You might have come across terms such as VRAM, System RAM, GPU Memory Usage and GPU Utilisation when monitoring a server or running an AI workload.
Although these terms are related, they do not mean the same thing.
Let's understand what each one means and how they work together.
1. What is GPU Memory?
Just like a CPU needs memory to work with data, a GPU also needs memory to perform its calculations.
When a GPU processes an application, it needs to access data such as model parameters, images, datasets and intermediate calculation results.
This information is stored in GPU memory so that the GPU can access it while performing computations.
For example, imagine running an AI model on a server. Before the GPU can process the model, its parameters and other required data need to be made available in GPU memory.
The amount of available memory can determine which models you can run, how large your datasets can be and how much data the GPU can process at once.
2. What is VRAM?
VRAM stands for Video Random Access Memory. It is the dedicated memory available on a graphics card or GPU.
Unlike normal system memory, VRAM is designed to work closely with the GPU and provide fast access to the data it needs.
Modern GPUs use high bandwidth memory technologies such as GDDR6, GDDR6X and HBM.
You will commonly find GDDR memory in graphics cards, while many high performance data centre GPUs use HBM.
For example, an NVIDIA data centre GPU might have 80 GB of HBM memory.
This means the GPU has 80 GB of dedicated memory available for storing data required by its workloads.
Why does VRAM matter for AI?
Consider running a Large Language Model (LLM).
The GPU needs memory to store:
- Model weights and parameters.
- Input and output data.
- Intermediate calculations.
- Activations and other working data.
If the model requires more memory than the GPU has available, it may fail to load or require techniques such as model quantisation, offloading or splitting the model across multiple GPUs.
This is why VRAM capacity is an important consideration when selecting GPUs for AI infrastructure.
3. What is System RAM?
System RAM is the main memory available to the CPU.
It stores the data and instructions that applications and the operating system need while running.
For example, when you start an AI application, its processes run on the CPU and use system RAM. When the application needs GPU processing, the required data is transferred to the GPU.
Think of a server with:
- 256 GB System RAM.
- One GPU with 80 GB VRAM.
These are two separate memory resources.
Having 256 GB of system RAM does not mean that the GPU has 256 GB available. The GPU still has its own 80 GB of dedicated memory.
The CPU and GPU can exchange data through interfaces such as PCIe. In some systems, technologies such as NVIDIA NVLink provide additional high speed communication between GPUs or supported processors.
System RAM and VRAM work together, but they serve different purposes.
4. How Do System RAM and VRAM Work Together?
Let's use a simple example.
Imagine you are running an AI image recognition application.
First, the application starts on the server and uses system RAM to load its files and prepare the data.
Next, the required model and input data are transferred to GPU memory.
The GPU processes the data and performs the calculations.
The resulting information can then be transferred back to system RAM for the CPU and application to use.
This is a simplified explanation, as real applications can move data between CPU and GPU memory multiple times.
The important point is that both memory resources are involved in the workload.
If system RAM is insufficient, the application may struggle before the GPU even starts processing.
If VRAM is insufficient, the GPU may not be able to load the model or process the required data efficiently.
5. What is GPU Utilisation?
GPU utilisation is a measurement of how busy the GPU is performing its processing work.
It is usually displayed as a percentage.
For example:
- 0% means the GPU is doing little or no processing work at that moment.
- 50% means the GPU is moderately busy.
- 100% means the GPU is fully busy according to the utilisation metric being reported.
However, GPU utilisation should not be confused with GPU memory usage.
These are two different measurements.
Imagine a GPU with 80 GB of VRAM.
An AI workload might use 70 GB of VRAM but show only 20% GPU utilisation.
This means a large amount of GPU memory is occupied, but the GPU is not necessarily performing calculations continuously.
Similarly, another workload might show 95% GPU utilisation while using only 15 GB of VRAM.
The GPU is working hard, but it does not need to occupy all its available memory.
High memory usage does not always mean high GPU utilisation, and high GPU utilisation does not mean all GPU memory is occupied.
6. How to Monitor GPU Memory and Utilisation
If you work with NVIDIA GPUs, one of the easiest ways to monitor GPU activity is through the nvidia-smi command.
Run:
nvidia-smi
This displays useful information, including:
- GPU model.
- Total GPU memory.
- Used GPU memory.
- Free GPU memory.
- GPU utilisation.
- Temperature and power consumption.
- Running GPU processes, where supported.
For example, you might see:
GPU Utilisation: 85%
Memory Usage: 65 GB / 80 GB
This tells you that the GPU is heavily utilised and that 65 GB of its 80 GB memory is occupied.
For continuous monitoring, you can use:
nvidia-smi -l 1
This refreshes the output every second.
These commands are particularly useful when troubleshooting AI and HPC workloads.
7. Why Does This Matter in HPC and AI Infrastructure?
Understanding GPU memory becomes especially important when designing and managing HPC and AI environments.
Consider a server containing four GPUs, each with 80 GB of VRAM.
Although the server has 320 GB of total GPU memory across all four devices, this does not automatically mean that a single application can use all 320 GB as one shared memory space.
The application needs to support multi GPU execution and distribute its workload across the GPUs.
This is particularly important when running large AI models that cannot fit into the memory of a single GPU.
Another important consideration is GPU memory bandwidth.
Memory capacity tells us how much data can be stored, while memory bandwidth tells us how quickly data can be transferred between GPU memory and the GPU processing units.
Both are important for performance.
A GPU might have plenty of memory capacity, but if its memory bandwidth is insufficient for a particular workload, performance can still be affected.
This is why GPU selection for HPC and AI infrastructure should consider memory capacity, bandwidth, processing power and the requirements of the applications being deployed.
Final Thoughts
GPU performance is not just about processing power. Memory plays an equally important role in determining what workloads a GPU can handle and how efficiently it can process them.
The key points to remember are:
- VRAM is the dedicated memory used by the GPU.
- System RAM is the main memory used by the CPU and applications.
- GPU utilisation indicates how busy the GPU is processing work.
- GPU memory usage and GPU utilisation measure different things.
- Memory capacity and memory bandwidth both matter for AI and HPC workloads.
Understanding these basics gives you a better foundation for working with GPUs, troubleshooting performance issues and designing infrastructure for modern AI and HPC environments.
Top comments (0)