DEV Community: Muhammad Zubair Bin Akbar

GPU Memory Explained: VRAM, System RAM and GPU Utilisation

Muhammad Zubair Bin Akbar — Sat, 31 Oct 2026 17:39:49 +0000

GPUs have become an important part of modern computing. They are used in Artificial Intelligence, Machine Learning, scientific simulations and High Performance Computing (HPC).

When people discuss GPU performance, they often focus on the number of GPU cores, processing power or memory capacity. However, there is another important part that can significantly affect performance: GPU memory.

You might have come across terms such as VRAM, System RAM, GPU Memory Usage and GPU Utilisation when monitoring a server or running an AI workload.

Although these terms are related, they do not mean the same thing.

Let's understand what each one means and how they work together.

1. What is GPU Memory?

Just like a CPU needs memory to work with data, a GPU also needs memory to perform its calculations.

When a GPU processes an application, it needs to access data such as model parameters, images, datasets and intermediate calculation results.

This information is stored in GPU memory so that the GPU can access it while performing computations.

For example, imagine running an AI model on a server. Before the GPU can process the model, its parameters and other required data need to be made available in GPU memory.

The amount of available memory can determine which models you can run, how large your datasets can be and how much data the GPU can process at once.

2. What is VRAM?

VRAM stands for Video Random Access Memory. It is the dedicated memory available on a graphics card or GPU.

Unlike normal system memory, VRAM is designed to work closely with the GPU and provide fast access to the data it needs.

Modern GPUs use high bandwidth memory technologies such as GDDR6, GDDR6X and HBM.

You will commonly find GDDR memory in graphics cards, while many high performance data centre GPUs use HBM.

For example, an NVIDIA data centre GPU might have 80 GB of HBM memory.

This means the GPU has 80 GB of dedicated memory available for storing data required by its workloads.

Why does VRAM matter for AI?

Consider running a Large Language Model (LLM).

The GPU needs memory to store:

Model weights and parameters.
Input and output data.
Intermediate calculations.
Activations and other working data.

If the model requires more memory than the GPU has available, it may fail to load or require techniques such as model quantisation, offloading or splitting the model across multiple GPUs.

This is why VRAM capacity is an important consideration when selecting GPUs for AI infrastructure.

3. What is System RAM?

System RAM is the main memory available to the CPU.

It stores the data and instructions that applications and the operating system need while running.

For example, when you start an AI application, its processes run on the CPU and use system RAM. When the application needs GPU processing, the required data is transferred to the GPU.

Think of a server with:

256 GB System RAM.
One GPU with 80 GB VRAM.

These are two separate memory resources.

Having 256 GB of system RAM does not mean that the GPU has 256 GB available. The GPU still has its own 80 GB of dedicated memory.

The CPU and GPU can exchange data through interfaces such as PCIe. In some systems, technologies such as NVIDIA NVLink provide additional high speed communication between GPUs or supported processors.

System RAM and VRAM work together, but they serve different purposes.

4. How Do System RAM and VRAM Work Together?

Let's use a simple example.

Imagine you are running an AI image recognition application.

First, the application starts on the server and uses system RAM to load its files and prepare the data.

Next, the required model and input data are transferred to GPU memory.

The GPU processes the data and performs the calculations.

The resulting information can then be transferred back to system RAM for the CPU and application to use.

This is a simplified explanation, as real applications can move data between CPU and GPU memory multiple times.

The important point is that both memory resources are involved in the workload.

If system RAM is insufficient, the application may struggle before the GPU even starts processing.

If VRAM is insufficient, the GPU may not be able to load the model or process the required data efficiently.

5. What is GPU Utilisation?

GPU utilisation is a measurement of how busy the GPU is performing its processing work.

It is usually displayed as a percentage.

For example:

0% means the GPU is doing little or no processing work at that moment.
50% means the GPU is moderately busy.
100% means the GPU is fully busy according to the utilisation metric being reported.

However, GPU utilisation should not be confused with GPU memory usage.

These are two different measurements.

Imagine a GPU with 80 GB of VRAM.

An AI workload might use 70 GB of VRAM but show only 20% GPU utilisation.

This means a large amount of GPU memory is occupied, but the GPU is not necessarily performing calculations continuously.

Similarly, another workload might show 95% GPU utilisation while using only 15 GB of VRAM.

The GPU is working hard, but it does not need to occupy all its available memory.

High memory usage does not always mean high GPU utilisation, and high GPU utilisation does not mean all GPU memory is occupied.

6. How to Monitor GPU Memory and Utilisation

If you work with NVIDIA GPUs, one of the easiest ways to monitor GPU activity is through the nvidia-smi command.

Run:

nvidia-smi

This displays useful information, including:

GPU model.
Total GPU memory.
Used GPU memory.
Free GPU memory.
GPU utilisation.
Temperature and power consumption.
Running GPU processes, where supported.

For example, you might see:

GPU Utilisation: 85%

Memory Usage: 65 GB / 80 GB

This tells you that the GPU is heavily utilised and that 65 GB of its 80 GB memory is occupied.

For continuous monitoring, you can use:

nvidia-smi -l 1

This refreshes the output every second.

These commands are particularly useful when troubleshooting AI and HPC workloads.

7. Why Does This Matter in HPC and AI Infrastructure?

Understanding GPU memory becomes especially important when designing and managing HPC and AI environments.

Consider a server containing four GPUs, each with 80 GB of VRAM.

Although the server has 320 GB of total GPU memory across all four devices, this does not automatically mean that a single application can use all 320 GB as one shared memory space.

The application needs to support multi GPU execution and distribute its workload across the GPUs.

This is particularly important when running large AI models that cannot fit into the memory of a single GPU.

Another important consideration is GPU memory bandwidth.

Memory capacity tells us how much data can be stored, while memory bandwidth tells us how quickly data can be transferred between GPU memory and the GPU processing units.

Both are important for performance.

A GPU might have plenty of memory capacity, but if its memory bandwidth is insufficient for a particular workload, performance can still be affected.

This is why GPU selection for HPC and AI infrastructure should consider memory capacity, bandwidth, processing power and the requirements of the applications being deployed.

Final Thoughts

GPU performance is not just about processing power. Memory plays an equally important role in determining what workloads a GPU can handle and how efficiently it can process them.

The key points to remember are:

VRAM is the dedicated memory used by the GPU.
System RAM is the main memory used by the CPU and applications.
GPU utilisation indicates how busy the GPU is processing work.
GPU memory usage and GPU utilisation measure different things.
Memory capacity and memory bandwidth both matter for AI and HPC workloads.

Understanding these basics gives you a better foundation for working with GPUs, troubleshooting performance issues and designing infrastructure for modern AI and HPC environments.

What Is an AI Factory? Understanding the Infrastructure Behind AI

Muhammad Zubair Bin Akbar — Wed, 28 Oct 2026 16:28:50 +0000

When we talk about artificial intelligence, we usually think about chatbots, large language models, image generators, and other AI applications.

But have you ever wondered what happens behind the scenes when an AI model is trained or responds to a question?

Behind every AI application is an infrastructure layer responsible for processing data, performing calculations, moving information between systems, and delivering results. This infrastructure can include thousands of GPUs, high speed networking, large storage systems, and software that manages workloads across multiple servers.

This is where the concept of an AI factory comes in.

An AI factory is not simply a room full of powerful GPUs. It is an integrated infrastructure designed to turn data into useful AI outputs as efficiently as possible.

Let's break down how it works and why it matters, especially if you are learning about High Performance Computing (HPC).

1. What Exactly Is an AI Factory?

Think of a traditional factory. Raw materials enter the factory, machines process them, and finished products come out at the other end.

An AI factory follows a similar concept, but instead of manufacturing physical products, it processes data to produce AI models, predictions, generated content, or other results.

Data enters the system, compute resources process it, and the output is delivered to an application or user.

For example, training a large language model requires enormous amounts of data and computational work. The infrastructure must load that data, distribute calculations across GPUs, coordinate communication between servers, and save the model as training progresses.

Once the model is trained, the same infrastructure, or a related deployment environment, can be used to serve requests from users.

The term AI factory describes this entire system working together.

2. The Main Components of an AI Factory

An AI factory depends on several infrastructure components. Each has a specific role, and a problem in one area can affect the performance of the entire system.

Compute: CPUs and GPUs

Compute is where the actual processing takes place.

CPUs handle general purpose tasks such as operating system operations, application logic, data preparation, and coordinating workloads.

GPUs are particularly important for AI because they can perform large numbers of mathematical operations in parallel. These operations are fundamental to training and running many modern AI models.

A single server might contain multiple GPUs, while a larger AI cluster can connect hundreds or thousands of GPUs across many servers.

However, adding more GPUs does not automatically guarantee better performance. The workload must be able to use them effectively, and the infrastructure must supply data and coordinate calculations fast enough.

Networking: Connecting the Compute Resources

When a workload runs across multiple GPUs or servers, those resources often need to exchange information.

For example, during distributed model training, GPUs may need to share intermediate results and synchronise updates. Slow communication can leave GPUs waiting instead of performing useful calculations.

This is why high performance networking is so important.

Technologies such as InfiniBand and high speed Ethernet are commonly used in AI and HPC environments. Depending on the design, these networks can support Remote Direct Memory Access (RDMA), which allows systems to exchange data with reduced CPU involvement.

The goal is to keep communication fast enough that the compute resources can continue working efficiently.

Storage: Feeding Data to the GPUs

AI workloads need access to datasets, trained models, checkpoints, and other files.

These files may be stored on local NVMe drives, shared network storage, or parallel file systems designed to serve many compute nodes at once.

Imagine a server with several powerful GPUs, but the dataset is stored on a system that cannot deliver data quickly enough. The GPUs may spend time waiting for the next batch of data rather than processing it.

Storage performance is therefore an important part of AI infrastructure.

Capacity also matters. Large datasets and model checkpoints can consume substantial storage, particularly when multiple training runs must be retained.

Software: Making Everything Work Together

Hardware alone cannot run an AI workload.

The software stack typically includes an operating system, GPU drivers, communication libraries, AI frameworks, and workload management tools.

For example, Linux provides the operating environment, NVIDIA drivers allow applications to use NVIDIA GPUs, and frameworks such as PyTorch provide tools for building and training AI models.

In an HPC environment, Slurm can schedule workloads and allocate resources across the cluster.

Each layer has a different responsibility, but all must work together for the application to run correctly.

Power and Cooling

There is another important part of AI infrastructure that is easy to overlook: power and cooling.

High density GPU servers can consume substantial electrical power and generate significant heat. As clusters grow, the data centre must provide sufficient electrical capacity and cooling to keep the equipment operating within safe limits.

This affects infrastructure design from the beginning. Server density, rack layouts, cooling methods, power distribution, and operational costs all influence how an AI system can be deployed and scaled.

3. How Do These Components Work Together?

Let's consider a simplified example of training an AI model across several GPU servers.

First, the training dataset is read from storage. The application loads the data and prepares it for processing.

The CPUs manage the application and help prepare the data, while the GPUs perform the computationally intensive operations involved in training.

If training is distributed across multiple GPUs or servers, the GPUs communicate through the cluster network to exchange information and coordinate their work.

This process repeats many times as the model learns from the data. Periodically, the training software saves checkpoints so that the work can be resumed if the job is interrupted.

Eventually, training finishes and the resulting model can be saved for evaluation or deployment.

Notice that the process depends on several components working together:

Storage must deliver the required data.
CPUs must keep the application and data pipeline moving.
GPUs must have enough compute and memory resources.
The network must support communication between distributed resources.
Software must coordinate the workload correctly.

If one component becomes a bottleneck, the performance of the entire workload may suffer.

For example, a slow data pipeline can leave GPUs underutilised. A network bottleneck can slow down distributed training. Insufficient GPU memory can prevent a model from fitting on a device.

This is why designing an AI factory requires more than selecting the fastest available GPU.

4. What Makes an AI Factory Different from a Traditional HPC Cluster?

At first glance, an AI factory and an HPC cluster can look very similar. Both can have compute nodes, high speed networks, shared storage, Linux, and workload management software.

The main difference is the workloads they are designed to support and the way their infrastructure is optimised.

Traditional HPC environments commonly run scientific simulations, engineering calculations, weather forecasting, and computational research. These workloads may depend heavily on CPU performance, memory bandwidth, low latency communication, or GPU acceleration.

AI infrastructure is often designed around model training, distributed GPU workloads, and inference, where trained models generate responses or predictions.

Large scale AI training can place particularly demanding requirements on GPU memory, network communication, data delivery, and power density.

Inference introduces a different set of considerations. A system serving AI requests must balance response time, throughput, memory capacity, and the number of users it can support.

There is considerable overlap between these environments. HPC clusters can run AI workloads, and AI infrastructure often uses technologies that have long been important in HPC.

In fact, many of the skills used to operate HPC systems, including Linux administration, cluster networking, storage management, automation, and workload scheduling, are directly relevant to AI infrastructure.

5. Where Does Slurm Fit Into an AI Factory?

If you have worked with HPC, you may already know Slurm.

Slurm is a workload manager that schedules jobs and allocates resources across a cluster. It helps multiple users share compute resources without each user having to manage individual servers manually.

In an AI environment, a user might submit a training job requesting two GPUs, several CPU cores, and a specific amount of memory.

Slurm determines when the requested resources are available, allocates them according to the cluster configuration and scheduling policies, and launches the job.

A simplified job request might look like this:

#!/bin/bash
#SBATCH --job-name=ai-training
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:2
#SBATCH --time=01:00:00

nvidia-smi
python train.py

This example requests one node, eight CPUs for the task, and two GPUs. The nvidia-smi command displays information about the available NVIDIA GPUs, and the Python script represents the training application.

The example assumes the cluster is configured to recognise GPUs through the gres mechanism and that the required software and training script are available.

Slurm does not train the model itself. It manages the resources and launches the workload, while the AI framework performs the actual training.

At larger scales, workload management becomes increasingly important because multiple teams and applications may compete for expensive compute resources.

6. Why GPU Utilisation Is Not the Whole Story

Suppose you have a powerful GPU server, but your monitoring shows that the GPUs are not being fully utilised.

It may be tempting to assume that the GPUs are underperforming. However, the actual problem could be elsewhere.

Perhaps the storage system cannot deliver data quickly enough. Maybe CPU resources are insufficient for preparing batches of data. The application might be waiting for communication from another server, or the model may not be configured to use the available GPUs effectively.

GPU utilisation is a useful metric, but it does not explain every performance problem.

A good troubleshooting process considers the entire workload: CPU usage, GPU activity and memory, storage performance, network communication, and application behaviour.

The same principle applies to traditional HPC. A compute node may have plenty of processing power, but overall performance depends on how effectively all the system components work together.

Understanding these relationships is a key skill for anyone working with AI infrastructure.

7. Why Is the AI Factory Concept Important?

AI workloads are growing in size and complexity. Organisations need infrastructure that can support demanding workloads while using expensive hardware efficiently.

This creates opportunities and challenges for infrastructure engineers.

Building an effective AI environment requires decisions about compute hardware, GPU memory, network topology, storage architecture, software deployment, scheduling, power, and cooling.

It also requires understanding how these components interact under real workloads.

For someone coming from a Linux or HPC background, this is an opportunity to build on existing knowledge rather than start from scratch.

Linux administration, Slurm, high speed networking, parallel storage, automation, and performance troubleshooting are all valuable foundations for working with AI infrastructure.

Learning how these technologies fit into the bigger picture is often more useful than focusing on one hardware component in isolation.

Conclusion

An AI factory is much more than a collection of powerful GPUs. It is an integrated infrastructure in which compute, networking, storage, software, power, and cooling work together to turn data into AI results.

The GPUs may perform the most visible calculations, but their performance depends on the systems surrounding them.

For anyone learning HPC, understanding the AI factory concept is a useful step towards understanding how modern AI systems are built, operated, and scaled.

The next time you see a large GPU cluster, think beyond the GPUs. Ask yourself how data reaches them, how they communicate, how workloads are scheduled, and how the entire system is kept running efficiently.

That is where the real infrastructure engineering begins.

CPU, Cores and Threads in HPC: Understanding What Your Job Actually Uses

Muhammad Zubair Bin Akbar — Sun, 25 Oct 2026 16:19:42 +0000

When you submit a job to an HPC cluster, you might request 4 CPUs, 8 CPUs, or perhaps an entire compute node.

But what does that actually mean?

Does one CPU mean one physical processor? Does one core mean one CPU? And where do threads fit into the picture?

Understanding these terms makes it much easier to understand how Slurm allocates resources and how your application uses them.

1. What is a CPU?

A CPU is the processor that executes instructions. In an HPC compute node, you might find one or two physical processor packages installed in the server.

For example, a server could have two Intel Xeon processors or two AMD EPYC processors.

Each physical processor contains multiple cores. These cores allow the processor to execute several tasks in parallel.

Think of the physical processor as a building and the cores as individual workers inside it.

2. What is a core?

A core is an independent processing unit within a physical processor.

Imagine a compute node with two processors, each containing 32 physical cores.

The total number of physical cores is:

2 processors × 32 cores = 64 physical cores.

This gives the node 64 physical cores available to execute workloads.

More cores can help applications run faster when they can divide their work into parallel tasks. However, adding cores does not automatically make every application faster.

Some applications are limited by memory bandwidth, storage performance, communication overhead, or work that cannot be parallelised.

3. What is a thread?

The word thread can mean two different things in HPC.

The first is a hardware thread. Modern Intel and AMD processors can support simultaneous multithreading, allowing one physical core to expose multiple logical CPUs to the operating system.

Intel calls its implementation Hyper Threading. AMD commonly refers to its implementation as SMT.

For example, if a server has 64 physical cores and each core supports two hardware threads, Linux may report 128 logical CPUs.

These are not 128 physical cores. They are 128 hardware execution contexts sharing 64 physical cores.

The second meaning is a software thread. An application can create multiple software threads to divide its work. These threads are scheduled onto the available hardware threads by the operating system.

The distinction matters because requesting more CPUs does not necessarily mean your application will create more software threads. Your application must be designed or configured to use them.

4. How does this work on Intel and AMD systems?

Both Intel Xeon and AMD EPYC processors follow the same basic concepts, although their architectures and features vary by processor generation and model.

Imagine an Intel based server with two Xeon processors. Each processor has 32 physical cores, giving the server 64 physical cores in total.

Now imagine an AMD based server with two EPYC processors. Each processor has 48 physical cores, giving the server 96 physical cores in total.

If both systems support two hardware threads per core, Linux could report 128 logical CPUs on the Intel server and 192 on the AMD server.

These are illustrative examples, not specifications for particular processor models.

Notice that the number of logical CPUs is higher than the number of physical cores. This is because Intel Hyper Threading and AMD SMT allow each physical core to expose two hardware threads to the operating system.

However, more logical CPUs do not automatically mean better HPC performance. The actual results depend on the application, memory bandwidth, processor architecture, and how efficiently the workload uses available resources.

That is why HPC administrators need to understand both the hardware and the workload before deciding how resources should be allocated.

5. What does this mean for an HPC job?

Suppose you submit a simple Slurm job requesting four CPUs.

Your job script might look like this:

#!/bin/bash
#SBATCH --job-name=cpu-test
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:05:00

echo "Running on $(hostname)"
echo "Allocated CPUs: $SLURM_CPUS_PER_TASK"

sleep 60

Submit it using:

sbatch cpu-test.sh

Here is what the important settings mean.

--ntasks=1 requests one task, which is commonly one process.

--cpus-per-task=4 requests four CPUs for that task.

This is useful for an application that uses one process with multiple software threads, such as an appropriately configured OpenMP application.

However, the script above only runs sleep. It does not create four computational threads or perform parallel calculations. It simply demonstrates the resource request.

Slurm allocates resources according to the cluster configuration and scheduling policy. Whether those CPUs correspond to distinct physical cores or hardware threads depends on the node configuration, Slurm settings, and allocation policy.

6. How can you check the actual hardware?

On a Linux compute node, run:

lscpu

Look for these fields:

CPU(s):
Thread(s) per core:
Core(s) per socket:
Socket(s):

For example:

CPU(s):                 128
Thread(s) per core:       2
Core(s) per socket:      32
Socket(s):                2

This indicates 128 logical CPUs, two hardware threads per core, 32 cores per processor package, and two processor packages.

To see how Linux maps logical CPUs to physical cores, use:

lscpu -e

You can also inspect the resources Slurm recognises with:

scontrol show node compute01

Compare the operating system view with Slurm's view. They should make sense together, although Slurm may intentionally allocate only part of the available hardware.

The key takeaway

A physical processor contains cores. A core can expose one or more hardware threads. Applications can create software threads that execute on those hardware resources.

When working with HPC, remember that the number of logical CPUs is not necessarily the number of physical cores, and the number of CPUs requested from Slurm is not necessarily the number of software threads your application will use.

Understanding that difference is one of the first steps towards understanding how HPC jobs actually run.

5 Slurm Configuration Parameters That Shape Your HPC Cluster

Muhammad Zubair Bin Akbar — Fri, 16 Oct 2026 16:15:54 +0000

When building an HPC cluster, we often focus on compute nodes, CPUs, RAM, GPUs, networking and storage.

But there is another important part of the design: how Slurm manages those resources.

A cluster might have hundreds of CPU cores and several GPUs, but how effectively users access those resources depends partly on how Slurm is configured.

Let's explore five important configuration areas in slurm.conf and understand their impact on an HPC environment.

1. `SelectType` and `SelectTypeParameters`

These parameters control how Slurm selects resources and allocates them to jobs.

For example:

SelectType=select/cons_tres
SelectTypeParameters=CR_Core

select/cons_tres allows Slurm to track and allocate consumable resources, such as CPUs, memory and GPUs when configured appropriately.

SelectTypeParameters determines the granularity at which resources are allocated. For example, CR_Core allocates resources at the CPU core level, while CR_CPU allocates at the CPU level as Slurm defines it. CR_Socket allocates at socket granularity, and CR_Memory focuses on memory as a consumable resource. The choice affects how precisely resources can be allocated and shared, potentially influencing resource utilisation and job placement.

Infrastructure impact: This setting affects how compute node resources are shared between jobs. The best option depends on the CPU topology, workload requirements and intended allocation policy.

Technical note: CR_CPU, CR_Core and CR_Socket are not interchangeable on every system. Their effects depend on the CPU topology and Slurm configuration. CR_Memory is a memory focused allocation option, not simply another CPU allocation granularity. Supported combinations also depend on the selected resource plugin and Slurm version, so check the documentation for your installed version before changing production settings.

2. `PriorityWeight` and Job Scheduling Priority

When multiple jobs are waiting in the queue, Slurm needs to determine their scheduling priorities.

With multifactor priority scheduling, administrators can assign weights to different factors that influence job priority.

For example:

PriorityType=priority/multifactor
PriorityWeightAge=1000
PriorityWeightFairshare=10000
PriorityWeightJobSize=100

Here, the weights determine how much the corresponding factors contribute to a job's priority.

PriorityWeightAge gives weight to how long a job has been waiting.
PriorityWeightFairshare gives weight to fairshare usage, helping account for how much cluster resources users or projects have consumed.
PriorityWeightJobSize gives weight to job size.

Why does this matter?

Imagine a research cluster shared by several teams. One team regularly submits large simulations, while another runs many smaller jobs.

Priority weights help administrators influence how waiting time, resource usage and job size affect scheduling decisions.

Infrastructure impact: Priority policies influence how users access shared compute capacity and can help balance competing workloads. They do not create additional resources or guarantee that a particular job runs first. Actual scheduling also depends on available resources, job constraints, reservations, QoS and other policies.

The weights should reflect the cluster's operational goals rather than being chosen arbitrarily.

3. `PartitionName`

A partition is a group of compute nodes that Slurm makes available under a particular set of scheduling policies.

For example:

PartitionName=cpu Nodes=cpu[01-08] \
    Default=YES MaxTime=24:00:00 State=UP

PartitionName=gpu Nodes=gpu[01-04] \
    Default=NO MaxTime=48:00:00 State=UP

In this example, the cluster has two partitions:

cpu: eight CPU compute nodes.
gpu: four GPU compute nodes.

A user can submit a job to the GPU partition with:

sbatch --partition=gpu job.sh

Why not put every node into one partition?

Different workloads have different requirements. A simulation might need many CPU cores, while an AI workload might require GPUs. Some applications need large amounts of RAM, and others have shorter execution times.

Partitions allow administrators to organise resources and apply different scheduling limits to different groups of nodes.

Infrastructure impact: Partition design influences which types of compute nodes are needed, how workloads are separated and how resources are made available to users.

A partition does not automatically guarantee a particular performance level or priority. Those behaviours depend on the wider scheduling configuration and policies.

4. `MaxTime`

Jobs can run for seconds, hours or several days. Slurm needs a way to limit how long a job is allowed to run.

For example:

PartitionName=cpu Nodes=cpu[01-08] \
    Default=YES MaxTime=24:00:00 State=UP

MaxTime=24:00:00 sets the maximum permitted job duration for this partition to 24 hours.

A job requesting a longer duration than the partition allows will not be accepted with that request.

Why does this matter?

Imagine a cluster used by two groups:

Interactive users running short tests.
Researchers running simulations that take several days.

Administrators could create separate partitions with different maximum durations, allowing short jobs to use one pool and longer jobs to use another.

Infrastructure impact: Runtime limits influence how administrators organise compute resources and balance job throughput against long running workloads.

However, a shorter MaxTime does not automatically make jobs finish faster or guarantee shorter queue times. Actual scheduling behaviour depends on resource availability, priority, backfill and other policies.

5. Accounting and Quality of Service (QoS)

An HPC cluster often serves multiple research teams, departments or projects. Administrators need to understand resource usage and control how resources are shared.

Two important concepts are accounting and Quality of Service.

An accounting configuration might include:

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=headnode

These settings tell Slurm to use SlurmDBD for accounting data, with the accounting service hosted at headnode in this example. SlurmDBD must be installed and configured separately.

Accounting records help administrators answer questions such as:

Which users and projects consume the most compute time?
How much CPU or GPU time has a research team used?
Which jobs failed, and how long did they run?

QoS policies can apply limits to jobs and allocations or influence their scheduling priority. For example, a research cluster might define a QoS that limits the number of running jobs or the resources a user can consume.

These policies are configured separately using Slurm's accounting tools, such as sacctmgr, along with the relevant associations and QoS settings.

Infrastructure impact: Accounting and QoS help administrators govern shared capacity, monitor demand and identify whether additional CPU nodes, GPU nodes or high memory nodes may be needed.

Accounting alone does not automatically enforce every usage limit. The relevant limits and policies must be configured explicitly.

Bringing it all together

These configuration areas address different parts of cluster management:

SelectType and SelectTypeParameters determine how resources are selected and allocated.
PriorityWeight settings influence how pending jobs are ranked.
PartitionName organises compute resources into scheduling pools.
MaxTime controls permitted job duration.
Accounting and QoS support usage visibility and resource governance.

Consider an HPC cluster serving both AI researchers and traditional simulation users. The administrator may need GPU nodes, high memory nodes, separate partitions, suitable allocation policies and accounting controls.

Slurm cannot replace good infrastructure design. However, its configuration helps ensure that available resources are allocated according to the cluster's requirements and policies.

The key takeaway: Understanding Slurm configuration is not just about making jobs run. It is about designing an HPC environment that allocates resources effectively, supports different workloads and can grow as demand changes.

Why Memory Bandwidth Matters in HPC

Muhammad Zubair Bin Akbar — Mon, 12 Oct 2026 17:49:12 +0000

Imagine having a powerful CPU with dozens of cores, but those cores spend time waiting for data from memory.

The processor is ready to work, but the data isn't arriving quickly enough.

This is one reason why adding more CPU cores doesn't always make an HPC application faster.

Memory capacity vs. memory bandwidth

These two terms sound similar, but they describe different things.

Memory capacity is how much data can fit in memory. For example, a node might have 256 GB of RAM.

Memory bandwidth is how quickly data can move between memory and the processor. It is commonly measured in GB/s.

Think of memory as a warehouse.

Capacity tells you how much stock the warehouse can hold. Bandwidth tells you how quickly goods can be delivered to the workers.

A large warehouse doesn't automatically mean goods can be delivered quickly.

The same applies to HPC systems.

Why does this matter in HPC?

HPC applications process large amounts of data, often across many CPU cores.

Consider an application that repeatedly reads arrays from memory, performs calculations, and writes the results back.

If the CPU can process data faster than memory can supply it, the processor may spend time waiting rather than performing useful calculations.

This is called a memory bandwidth bottleneck.

Adding more CPU cores might not help much because those cores must share the available memory bandwidth.

A simple example

Imagine a compute node with 32 CPU cores.

Initially, only 8 cores are running an application, and performance is good.

You increase the workload to 16 cores, then 32 cores.

However, the application becomes only slightly faster.

Why?

The additional cores increase demand for data, but the memory subsystem cannot deliver that data proportionally faster.

The CPUs are not necessarily too slow. The application may be limited by how quickly data reaches them.

This is one reason HPC performance doesn't always scale linearly with the number of CPU cores.

How can HPC engineers investigate it?

When an application stops scaling as expected, CPU utilization alone doesn't tell the whole story.

Engineers may investigate:

Memory bandwidth: Is the application reaching the memory subsystem's throughput limits?
NUMA placement: Are processes accessing memory attached to another CPU socket?
Memory access patterns: Is the application accessing data efficiently?
CPU affinity: Are processes and threads placed appropriately on the available cores?

Tools such as numastat, perf, and hardware performance counters can help investigate these issues, depending on the system and available permissions.

The bigger picture

HPC performance depends on more than processor speed.

CPU cores need data from memory. Applications may also depend on storage, network communication, and synchronization between processes.

When one part cannot keep up with the others, it can limit the performance of the entire application.

The key takeaway: More CPU cores are useful only when the application and the rest of the system can keep them productive.

In HPC, understanding how quickly data moves can be just as important as understanding how quickly a processor calculates.

Why More CPU Cores Don't Always Make an HPC Application Faster

Muhammad Zubair Bin Akbar — Sun, 04 Oct 2026 15:39:09 +0000

If 1 CPU can run a program, then 8 CPUs should make it much faster.

Right?

Not necessarily.

This is one of the first things that can be surprising when learning HPC.

We can keep adding CPU cores to a system, but that does not mean an application will keep getting proportionally faster.

The reason comes down to parallelism.

Not every part of a program can run in parallel

Imagine an application that takes 100 seconds to run on one CPU.

Suppose:

80 seconds can be parallelised
20 seconds must run sequentially

The 80 second part can use multiple CPU cores.

But the 20 second part still has to run one step after another.

So even if we give the application 100 CPU cores, that sequential portion doesn't magically disappear.

This is the basic idea behind Amdahl's Law.

What happens as we add more cores?

Imagine the application running on:

1 core   → 100 seconds
2 cores  → 60 seconds
4 cores  → 40 seconds
8 cores  → 30 seconds
16 cores → 25 seconds
32 cores → 23 seconds

We're adding more and more CPU cores.

But the improvement gets smaller.

Why?

Because eventually the parts of the application that cannot be parallelised become the limiting factor.

The application has reached a point where adding more CPUs provides very little benefit.

The problem isn't always the CPU

This is an important idea in HPC.

When an application doesn't scale well, the CPU may not be the problem.

The application might be waiting for:

Memory
Network communication
Storage I/O
Synchronisation between processes
Locks
Other processes

For example, imagine an MPI application running across many compute nodes.

Each process may need to exchange data with other processes.

Adding more nodes gives you more CPUs, but it also means more communication between them.

At some point, the communication overhead can start becoming significant.

You have more computing resources, but you are spending more time coordinating them.

Strong scaling vs weak scaling

This leads to two important HPC concepts.

Strong scaling asks:

How much faster can I solve the same problem by adding more resources?

For example:

Same problem
     ↓
8 cores → 16 cores → 32 cores → 64 cores

Ideally, the runtime keeps decreasing.

But eventually the speedup usually becomes smaller.

Weak scaling asks something different:

Can I solve a larger problem by adding more resources while keeping the workload per resource roughly the same?

For example:

8 cores   → small problem
16 cores  → larger problem
32 cores  → even larger problem

Both are important ways of understanding HPC application performance.

More cores can even make things worse

This can sound strange, but adding resources can sometimes reduce efficiency.

Imagine 16 processes constantly exchanging data.

Now increase that to 128 processes.

There are many more processes that may need to communicate and synchronise.

If the application's communication pattern doesn't scale well, the additional resources can create more overhead than useful computation.

The CPUs aren't necessarily doing more useful work.

They may be spending more time waiting.

This is why benchmarking matters

You cannot determine how well an HPC application will scale simply by looking at the number of CPU cores in a server.

You need to test it.

For example:

1 node   → 100 seconds
2 nodes  → 55 seconds
4 nodes  → 32 seconds
8 nodes  → 25 seconds
16 nodes → 23 seconds

The application is scaling.

But not linearly.

Doubling the resources does not always halve the runtime.

That is normal for many real world HPC applications.

The goal isn't always maximum CPU usage

Another important lesson is that 100% CPU utilisation doesn't automatically mean an application is scaling well.

An application could be using every CPU core while spending significant time waiting for memory, communicating with other processes or performing synchronisation.

This is why HPC performance analysis looks at much more than CPU utilisation.

You may need to investigate:

CPU utilisation
Memory bandwidth
NUMA behaviour
Network traffic
MPI communication
Storage I/O
Process placement
CPU affinity

The bottleneck can move as you add resources.

So how many CPU cores should you use?

There isn't one answer.

The right number depends on the application.

For some workloads:

8 cores → good scaling
16 cores → better
32 cores → better
64 cores → little additional benefit

For another application, 64 or 128 cores might still provide excellent scaling.

This is why HPC systems don't simply try to give every application as many CPUs as possible.

The goal is to use the resources efficiently.

The bigger HPC lesson

HPC isn't simply about having more CPUs.

It's about using parallel resources effectively.

A well designed application can divide a large problem across many CPU cores or nodes and achieve significant speedup.

But parallelism has limits.

Sequential work, communication, synchronisation, memory bandwidth and I/O can all become bottlenecks.

So the next time someone says:

"This server has twice as many CPU cores, so the application should be twice as fast."

The better question is:

"Does the application scale?"

That's one of the fundamental questions in HPC performance.

And sometimes, the answer is:

not as much as you might expect.

Why Does HPC Need InfiniBand?

Muhammad Zubair Bin Akbar — Wed, 30 Sep 2026 18:50:16 +0000

When people first learn about HPC clusters, they often notice something unusual.

The compute nodes don't just have Ethernet.

Many HPC clusters also have InfiniBand.

So a natural question is:

Why?

If Ethernet already connects computers together, why do HPC clusters need another network?

The answer becomes clearer when you look at what HPC applications actually do.

HPC applications communicate a lot

Consider an MPI application running across 32 compute nodes.

The application may constantly exchange data between those nodes.

For example:

Node 1  <---->  Node 2
  ↕              ↕
Node 3  <---->  Node 4

The application isn't just sending an occasional file or SSH connection.

The compute nodes can be communicating continuously while the application is running.

That makes network performance extremely important.

It's not just about bandwidth

A common first thought is:

"Just give the cluster a faster network."

But HPC networking is about more than bandwidth.

Two important factors are:

Bandwidth

How much data can be transferred per second.

Latency

How long it takes for a message to travel between nodes.

For many HPC workloads, both matter.

A network can have high bandwidth but still have latency that isn't ideal for tightly coupled workloads.

Where InfiniBand comes in

InfiniBand was designed for high performance communication.

It provides:

High bandwidth
Low latency
RDMA
Efficient communication between nodes

One particularly important feature is RDMA.

RDMA allows data to be transferred directly between the memory of different systems with minimal involvement from the CPU and operating system.

That can reduce overhead during communication.

Why does that matter?

Imagine an MPI application running across hundreds of nodes.

The application might repeatedly exchange relatively small messages between processes.

If every communication requires significant CPU and operating system involvement, that overhead can become important.

With technologies such as RDMA, the network can handle communication more efficiently.

This is one reason high performance networks are so important in HPC.

InfiniBand isn't the only option

InfiniBand isn't the only technology used for HPC networking.

You can also find:

High speed Ethernet
RoCE
Other specialised interconnects

Modern Ethernet can provide very high performance, and technologies such as RoCE can provide RDMA capabilities over Ethernet.

So the question isn't simply:

"InfiniBand or Ethernet?"

It's really about what the workload and cluster architecture require.

The bigger picture

An HPC cluster is a collection of computers working together.

The CPUs do the computation.

The memory holds the data being processed.

The storage provides the data.

And the network allows the nodes to communicate.

If the application spends a lot of time waiting for data from another node, having powerful CPUs won't solve the problem.

The network becomes part of the application's performance.

That's why HPC networking is such an important part of cluster design.

And that's also why you'll often hear technologies such as:

InfiniBand → RDMA → MPI → UCX

when talking about HPC communication.

Understanding how these pieces fit together is much more useful than simply knowing that "HPC uses InfiniBand."

What Actually Happens After You Run sbatch?

Muhammad Zubair Bin Akbar — Fri, 25 Sep 2026 20:09:44 +0000

If you are learning Slurm, you probably start with a few commands:

sbatch job.sh
squeue
scancel <jobid>

It is easy to think of sbatch as simply "running a job."

But that is not really what happens.

When you run:

sbatch job.sh

you are submitting a request to Slurm.

What happens after that is where things get interesting.

sbatch submits the job

Suppose we have a simple script:

#!/bin/bash

#SBATCH --job-name=zubair-test
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00

./my_application

When we run:

sbatch job.sh

Slurm accepts the job request and gives it a job ID.

For example:

Submitted batch job 12345

At this point, your application has not necessarily started.

The job has simply entered Slurm.

This is one of the most important concepts to understand:

Submitting a job is not the same as running a job.

The job enters the scheduling system

The Slurm controller now knows about your job.

You can see it with:

squeue

You might see:

JOBID   PARTITION   NAME   USER           ST   TIME
12345   compute     test   zubair-test    PD   0:00

The PD state means pending.

The job is waiting for an opportunity to run.

But why does it have to wait?

Because Slurm has to make a scheduling decision.

Slurm looks at the job's request

The scheduler needs to understand what the job is asking for.

In our example, the job requested:

4 CPUs
8 GB memory
1 hour of runtime

But a real HPC job could request much more:

16 CPUs
128 GB memory
2 GPUs
4 hours
specific partition
specific features

Slurm has to determine whether those requirements can be satisfied.

And this is where simply looking at "free CPUs" can become misleading.

A cluster might have idle CPUs but still not be able to run your job because of memory, GPU availability, partition limits, reservations, priorities, dependencies or other scheduling rules.

The scheduler decides when the job can run

Slurm's scheduler considers the jobs waiting in the system and the resources available on the cluster.

It is not simply doing this:

"There are 4 free CPUs, so run the job."

Instead, it has to consider the overall scheduling configuration and policies.

For example:

Which partition can run the job?
Does the job have enough priority?
Are the requested CPUs available?
Is enough memory available?
Does the job require GPUs?
Are there reservations?
Are there account or QoS restrictions?
Are other jobs already using the required resources?

This is why understanding Slurm requires more than memorising commands.

The commands are the interface.

The scheduler is the system underneath.

An allocation is created

Eventually, Slurm determines that the job can run.

It allocates the requested resources.

For example:

Node: compute-05
CPUs: 4
Memory: 8 GB
Time limit: 1 hour

The job changes from:

PD

to:

R

where R means running.

Now the job has an allocation.

This is an important distinction:

The scheduler decides what resources the job gets.

The application then runs inside that allocation.

The job reaches a compute node

The actual application does not normally run on the Slurm controller.

The controller manages the cluster and scheduling decisions.

The job is launched on one or more compute nodes.

For example:

Slurm Controller
       |
       |
   Scheduler
       |
       |
   Allocation
       |
       |
   compute-05
       |
       |
   Your application

Once the job starts on the compute node, a lot more of the HPC stack becomes involved.

Linux manages the processes.

The CPU and memory topology affects performance.

cgroups can control resource usage.

If GPUs are requested, Slurm's GRES configuration and GPU management become relevant.

If the application communicates across nodes, the network becomes part of the picture.

And if it reads or writes large amounts of data, the storage system becomes important.

This is where HPC gets interesting

Consider an MPI application running across 32 nodes.

Slurm has already done its job of allocating resources.

But now the application depends on many other parts of the system working together.

You might have:

CPUs and NUMA topology
memory
GPUs
InfiniBand or Ethernet
RDMA
MPI
UCX
parallel storage
Linux processes and cgroups

So when an HPC application is slow, the problem is not necessarily Slurm.

Slurm may have allocated the resources correctly.

The bottleneck could instead be CPU placement, memory access, network communication, storage I/O, GPU utilisation or the application itself.

That is why HPC administration involves understanding how these layers interact.

From one command to an entire system

This is what makes a simple command like:

sbatch job.sh

much more interesting than it first appears.

You submit a job.

Slurm records the request.

The scheduler evaluates it.

Resources are allocated.

The job is launched on compute nodes.

Then Linux, CPUs, memory, GPUs, networking, storage and the application all become part of the execution.

So the next time you run:

sbatch job.sh

remember that you are not simply "starting a program."

You are asking an entire resource management system to find a place for that program to run.

And understanding that process is one of the foundations of understanding HPC.

Kubernetes and Slurm are both popular for AI and HPC workloads, but they serve different purposes. This article explores their strengths, trade offs, and ideal use cases across scheduling, GPUs, distributed training, networking, and more.

Muhammad Zubair Bin Akbar — Thu, 16 Jul 2026 11:54:49 +0000

Muhammad Zubair Bin Akbar

Jul 16

Slurm vs Kubernetes: Understanding the Right Tool for the Right Workload

#ai #kubernetes #hpc #productivity

4 min read

Slurm vs Kubernetes: Understanding the Right Tool for the Right Workload

Muhammad Zubair Bin Akbar — Thu, 16 Jul 2026 11:48:29 +0000

Why This Comparison Matters

As AI and High Performance Computing (HPC) continue to converge, one question appears in almost every infrastructure discussion:

Should we use Kubernetes or Slurm?

The answer is rarely as simple as choosing one over the other.

Kubernetes dominates cloud native application deployment, while Slurm has been the scheduler of choice for scientific computing and supercomputers for decades. Today, organizations running AI training, machine learning, simulations, computational chemistry, CFD, and financial modeling often need to decide which platform better fits their workloads.

This article explains where each platform excels, where it struggles, and when using both together makes the most sense.

Understanding Their Different Goals

Although both systems schedule workloads across clusters, they were built with completely different objectives.

Kubernetes was designed to orchestrate containers and microservices at massive scale.

Its priorities include:

High availability
Auto scaling
Self healing applications
Rolling updates
Service discovery
Cloud native deployments

Slurm, on the other hand, was built specifically for HPC environments where maximizing compute efficiency is the primary objective.

Its priorities include:

Efficient resource allocation
Fair scheduling
Job queuing
Large MPI jobs
GPU scheduling
High cluster utilization

Both schedule workloads but they optimize for entirely different problems.

Architecture Comparison

Kubernetes	Slurm
Container orchestrator	HPC workload manager
API driven	Job scheduler
Works with pods	Works with batch jobs
Focuses on services	Focuses on compute jobs
Cloud native	HPC native
Long running applications	Batch oriented workloads

Resource Scheduling

This is where the biggest differences appear.

Kubernetes Scheduling

Kubernetes schedules Pods based on:

CPU requests
Memory requests
Labels
Affinity rules
Taints and tolerations
Custom schedulers
Priority classes

It primarily answers:

"Where should this container run?"

The scheduler focuses on cluster health rather than maximizing utilization.

Slurm Scheduling

Slurm considers far more information.

Examples include:

Available CPUs
Memory
GPUs
Licenses
Partitions
QoS
Reservations
Node topology
NUMA layout
Job priority
Fair share
Backfill opportunities

Instead, Slurm answers:

"How can I execute every compute job as efficiently as possible?"

For HPC environments, this distinction is extremely important.

GPU Management

AI workloads live and die by GPU utilization.

Kubernetes

GPU support comes from:

NVIDIA Device Plugin
GPU Operator
MIG support
Device sharing
Dynamic provisioning

Excellent for:

AI inference
Model serving
MLOps pipelines
Kubernetes native AI platforms

Slurm

GPU scheduling has been part of HPC for years.

It supports:

GRES
GPU affinity
Exclusive GPU allocation
Multi node GPU jobs
GPU topology awareness
CUDA aware scheduling

Large distributed training jobs typically integrate naturally with Slurm.

Multi node Distributed Training

Modern LLM training often spans dozens or even thousands of GPUs.

Kubernetes

Possible using:

Kubeflow
Ray
Volcano Scheduler
MPI Operator
Kueue

While these projects have matured significantly, distributed AI training often requires additional components beyond core Kubernetes.

Slurm

Distributed workloads are native.

Launching thousands of MPI processes is as simple as:

srun -N 64 -n 512 ./application

Or GPU training:

srun torchrun ...

MPI, NCCL, UCX, and InfiniBand integrate naturally with Slurm environments.

Networking

High performance networking is another major differentiator.

Kubernetes

Usually designed around:

CNI plugins
Overlay networks
Service meshes
Network policies

Although RDMA and InfiniBand are supported, they often require additional configuration.

Slurm

Designed for clusters using:

InfiniBand
HDR/NDR fabrics
RoCE
UCX
MPI
Low latency communication

Network performance is treated as a first class citizen.

Job Lifecycle

Kubernetes

Applications often run continuously.

Examples:

Web APIs
Databases
AI inference servers
Monitoring services

Pods restart automatically after failure.

Slurm

Jobs are temporary.

Typical lifecycle:

Submit
Queue
Allocate resources
Execute
Finish
Release resources

This matches scientific computing perfectly.

Auto Scaling

Kubernetes

One of Kubernetes' biggest strengths.

Supports:

Horizontal Pod Autoscaler
Vertical Pod Autoscaler
Cluster Autoscaler
Cloud auto provisioning

Ideal for dynamic workloads.

Slurm

Traditionally built for fixed clusters.

However, cloud integrations now allow elastic compute using:

Azure CycleCloud
AWS ParallelCluster
AWS PCS
Google Cloud HPC Toolkit

Nodes can automatically start and stop based on queue demand.

Ecosystem

Kubernetes Ecosystem

The ecosystem is enormous.

Popular AI tools include:

Kubeflow
MLflow
Argo Workflows
Ray
KServe
Prometheus
Grafana
Istio
Helm

Cloud providers offer fully managed Kubernetes services.

Slurm Ecosystem

Focused on HPC.

Common integrations include:

OpenMPI
MPICH
UCX
BeeGFS
Lustre
GPFS
Spack
Lmod
Apptainer (Singularity)

These tools are standard across many research institutions and supercomputing centers.

Performance

For tightly coupled HPC jobs, Slurm generally provides better performance because it understands:

CPU topology
NUMA domains
GPU locality
Network topology
Exclusive allocations

Kubernetes introduces additional abstraction layers that are acceptable for cloud applications but may add complexity for latency sensitive HPC workloads.

For AI inference or microservices, Kubernetes is usually the stronger choice.

Operational Complexity

Kubernetes	Slurm
Steeper learning curve for cloud native concepts	Easier for traditional HPC administrators
Large ecosystem	Focused ecosystem
Many moving parts	Smaller control plane
Excellent API ecosystem	Excellent batch scheduling

Both platforms require expertise, but in different domains.

Can They Work Together?

Absolutely.

Many organizations no longer choose one over the other.

A common architecture looks like this:

Kubernetes for:
- AI inference
- APIs
- MLOps
- Jupyter notebooks
- Web services
- Model deployment
Slurm for:
- AI training
- HPC simulations
- MPI applications
- Large GPU clusters
- Scientific computing
- Batch processing

Some environments even allow Kubernetes to submit workloads into Slurm managed clusters, combining the flexibility of cloud native tooling with the efficiency of HPC scheduling.

Which One Should You Choose?

Workload	Best Choice
Web applications	Kubernetes
AI inference	Kubernetes
MLOps pipelines	Kubernetes
Scientific simulations	Slurm
MPI applications	Slurm
CFD	Slurm
Computational chemistry	Slurm
Large scale distributed AI training	Slurm
Mixed enterprise AI platform	Kubernetes + Slurm

Final Thoughts

Kubernetes and Slurm are not competitors in the traditional sense. They were designed to solve different problems, and both have evolved to support modern AI infrastructure.

If your primary focus is cloud native applications, model serving, and scalable AI services, Kubernetes provides unmatched flexibility and ecosystem support.

If your goal is to maximize utilization of expensive CPUs and GPUs for tightly coupled scientific workloads or large scale distributed AI training, Slurm remains the gold standard.

As AI platforms continue to grow, the most effective architectures increasingly combine the strengths of both technologies rather than forcing a choice between them.

The future of AI infrastructure is not Kubernetes versus Slurm. It's knowing when to use each one.

Why AI Clusters Fail Even When GPUs Are Idle

Muhammad Zubair Bin Akbar — Fri, 26 Jun 2026 22:50:48 +0000

When organizations build AI infrastructure, GPUs usually get all the attention.

Teams invest in the latest accelerators, add high speed networking, and expect training jobs to scale effortlessly. Yet many AI clusters deliver disappointing performance despite having powerful hardware.

The surprising part?

The GPUs are often idle.

GPU monitoring dashboards may show utilization dropping to 20%, 10%, or even 0% between bursts of activity. At first glance, this looks like a GPU problem, but in most cases it isn’t.

The GPUs are simply waiting.

Let’s understand why this happens and how HPC principles can help solve it.

⸻

The GPU Is Only One Part of the Pipeline

Think of an AI training job like an assembly line.

Before a GPU can process a batch, several things must happen:

Data must be read from storage.
Files may need decompression.
Images or text must be preprocessed.
Data is copied into system memory.
Finally, it is transferred to GPU memory.

Only after all these steps can computation begin.

If any stage becomes slow, the GPU has nothing to process and simply waits.

Imagine buying the fastest race car in the world but fueling it with a tiny garden hose.

The car isn’t slow.

The fuel delivery is.

⸻

Common Reasons GPUs Sit Idle

1. Slow Storage Performance

Large AI datasets often consist of millions of small files.

If the storage system cannot deliver data quickly enough, GPUs finish processing one batch before the next is ready.

This is especially common when:

Using network-attached storage
Reading millions of tiny files
Storage bandwidth is shared among many users

The result is expensive GPUs waiting for data.

⸻

2. Data Loading Becomes the Bottleneck

Most deep learning frameworks rely on data loader workers running on CPUs.

These workers:

Read files
Decode images
Tokenize text
Apply augmentations
Prepare training batches

If there are too few workers or the CPUs are overloaded, GPU utilization drops dramatically.

Many people immediately reduce batch size or change GPU settings, when the actual bottleneck is the CPU.

⸻

3. CPUs Cannot Keep Up

Modern GPUs are incredibly fast.

Preparing data fast enough to feed them requires powerful CPUs.

If CPU cores are fully occupied with preprocessing tasks, GPUs repeatedly wait for the next batch.

This becomes more noticeable as GPU performance increases.

Ironically, upgrading GPUs without upgrading CPUs can actually expose new bottlenecks.

⸻

4. Poor Network Performance

Distributed training depends heavily on communication.

Gradients, parameters, and synchronization data constantly move between nodes.

If the network is slow or congested:

GPUs complete computation.
They wait for synchronization.
Training stalls before the next iteration begins.

This is why technologies like InfiniBand, Omni Path, and RDMA are so valuable in AI clusters.

⸻

5. Small Batch Sizes

Sometimes the workload itself is too small.

If each GPU receives only a tiny amount of work:

Computation finishes quickly.
Communication overhead dominates.
GPUs spend more time waiting than computing.

Increasing batch size or improving workload distribution often improves utilization.

⸻

6. Filesystem Contention

In shared HPC environments, dozens or hundreds of users may access the same storage simultaneously.

Even if a single training job performs well during testing, production workloads may compete for:

Storage bandwidth
Metadata operations
Shared filesystem resources

As contention grows, GPUs spend more time waiting for IO.

⸻

The Hidden Cost of Idle GPUs

Imagine an organization with:

64 GPUs
Each GPU costs thousands of dollars
Jobs run continuously

If GPU utilization averages only 40%, then more than half of the available computing power is effectively wasted.

Organizations often respond by purchasing more GPUs.

In reality, fixing storage, networking, scheduling, or data pipelines could provide a much larger performance improvement at a fraction of the cost.

⸻

How HPC Practices Help

Traditional HPC has dealt with resource bottlenecks for decades.

Many of the same principles improve AI workloads.

Optimize Data Locality

Store frequently used datasets close to compute nodes whenever possible.

Reducing unnecessary data movement keeps GPUs busy.

⸻

Improve Storage Performance

Use parallel filesystems, local NVMe storage, or intelligent caching for large datasets.

Faster data access directly translates into higher GPU utilization.

⸻

Tune Data Loaders

Experiment with:

Number of worker processes
Prefetching
Pinned memory
Batch preparation

Small configuration changes can produce significant improvements.

⸻

Balance CPU and GPU Resources

Use High Speed Interconnects

Distributed AI workloads benefit greatly from low latency networking.

Reducing communication delays allows GPUs to spend more time computing.

⸻

Monitor the Entire Pipeline

Instead of monitoring only GPU utilization, observe:

CPU usage
Disk throughput
Network bandwidth
Filesystem latency
Memory utilization
Data loading times

The real bottleneck is often outside the GPU.

⸻

A Real World Example

Consider a cluster with eight GPUs training an image classification model.

During monitoring:

GPU utilization averages only 35%.
CPUs remain close to 100%.
Storage shows heavy read activity.

The instinct might be to upgrade the GPUs.

Instead, the team moves the dataset to local NVMe storage and increases the number of data loader workers.

GPU utilization jumps to over 90%.

No new GPUs were purchased.

The bottleneck was never the accelerators.

⸻

Final Thoughts

AI performance is about far more than GPUs.

A training job is only as fast as its slowest component. Storage, CPUs, networking, filesystems, and data pipelines all contribute to overall performance.

When GPUs appear idle, they’re usually waiting for the rest of the system to catch up.

Understanding the entire infrastructure, rather than focusing solely on accelerators, is what separates a well designed AI cluster from an expensive collection of underutilized hardware.

The next time someone says, “Our GPUs are slow”, take a closer look.

The GPUs may simply be waiting for everyone else.

From HPC Clusters to AI Training: How High Performance Computing Powers Modern AI

Muhammad Zubair Bin Akbar — Mon, 25 May 2026 18:04:02 +0000

Artificial Intelligence and Machine Learning workloads continue to grow rapidly. Training modern AI models now requires massive compute power, large memory capacity, fast storage, and efficient networking. This is where High Performance Computing (HPC) becomes important.

HPC is no longer limited to scientific simulations or academic research. Today, it plays a major role in AI and ML development across industries including healthcare, finance, automotive, cybersecurity, and research.

What is HPC?

High Performance Computing refers to a group of powerful servers connected together to work as a single system. These systems are designed to process extremely large workloads much faster than traditional servers or desktop machines.

An HPC cluster usually includes:

Multiple compute nodes
High core count CPUs
GPUs for acceleration
Fast parallel storage
High speed interconnects like InfiniBand or Omni Path
Resource managers such as Slurm

Instead of running workloads on a single machine, HPC distributes tasks across many nodes to reduce execution time and improve performance.

Why AI and ML Need HPC

AI and ML training workloads are computationally expensive. Modern models process massive datasets and perform billions or even trillions of calculations during training.

A normal server may struggle with:

Large language model training
Deep learning workloads
Distributed GPU training
Processing huge datasets
Real time inference at scale

HPC solves these limitations by providing scalable compute resources and parallel processing capabilities.

GPU Acceleration in AI Training

GPUs are one of the biggest reasons HPC is widely used in AI.

Unlike CPUs, GPUs can process thousands of operations simultaneously. This makes them ideal for matrix calculations and tensor operations used in deep learning frameworks such as:

PyTorch
TensorFlow
JAX

In HPC environments, multiple GPUs can work together across different compute nodes using technologies like:

NVIDIA NCCL
CUDA
MPI
RDMA

This allows AI models to train significantly faster compared to single machine setups.

Distributed Training

Training large AI models on one GPU is often impossible because of memory and compute limitations.

HPC clusters enable distributed training where:

Data is split across multiple GPUs
Workloads are shared between nodes
Training happens in parallel
Results are synchronized efficiently

Distributed training helps reduce training time from weeks to days or even hours.

For example:

A single GPU may take several weeks to train a large model
An HPC cluster with hundreds of GPUs can complete the same training much faster

Importance of High Speed Networking

AI workloads generate constant communication between GPUs and compute nodes.

Slow networking creates bottlenecks and reduces performance. HPC clusters use low latency high bandwidth interconnects such as:

InfiniBand
Omni Path
RoCE

These technologies help accelerate:

GPU to GPU communication
Distributed storage access
MPI communication
Parameter synchronization

Low latency networking is critical for efficient AI scaling.

Storage Requirements for AI

AI training datasets can range from gigabytes to petabytes.

Traditional storage solutions may not provide enough throughput for parallel workloads. HPC environments often use:

Lustre
BeeGFS
GPFS
NVMe based storage

These parallel filesystems allow multiple nodes to read and write data simultaneously without major bottlenecks.

Job Scheduling and Resource Management

In shared HPC environments, multiple users run workloads at the same time.

Schedulers like Slurm help manage resources efficiently by:

Allocating CPUs and GPUs
Managing job queues
Enforcing fair usage
Optimizing cluster utilization

AI researchers can submit training jobs, reserve GPUs, monitor workloads, and scale resources dynamically.

Common AI Workloads Running on HPC

HPC clusters are commonly used for:

Deep Learning Training

Training neural networks for image recognition, NLP, and recommendation systems.

Large Language Models

Training and fine tuning transformer based models.

Computer Vision

Object detection, segmentation, and video analytics.

Scientific AI

Using AI for climate modeling, genomics, and physics research.

Reinforcement Learning

Large scale simulation based training environments.

Benefits of Using HPC for AI

Faster Training

Massive parallelism reduces overall training time.

Scalability

Resources can scale from a few GPUs to thousands.

Better Resource Utilization

Shared infrastructure improves efficiency and reduces hardware waste.

Improved Collaboration

Researchers and teams can share compute infrastructure securely.

Support for Large Models

HPC makes training large modern AI models practical.

Challenges in AI HPC Environments

Despite the advantages, AI on HPC also introduces challenges:

GPU memory limitations
Network bottlenecks
Storage throughput issues
Distributed training complexity
Power and cooling requirements
Software compatibility

Proper cluster design and tuning are important for achieving good AI performance.

The Future of HPC and AI

AI and HPC are becoming increasingly connected. Many modern supercomputers are now designed primarily for AI workloads.

Future HPC systems will continue focusing on:

Larger GPU clusters
Faster interconnects
AI optimized processors
Energy efficient computing
Hybrid cloud HPC environments

As AI models continue growing, HPC will remain one of the core technologies powering innovation.

Final Thoughts

High Performance Computing has become one of the most important foundations for modern AI and Machine Learning workloads.

From distributed GPU training to high speed networking and parallel storage, HPC provides the infrastructure needed to train and run advanced AI models efficiently at scale.

As AI adoption continues increasing across industries, the role of HPC will only become more important in the years ahead.

DEV Community: Muhammad Zubair Bin Akbar

GPU Memory Explained: VRAM, System RAM and GPU Utilisation

1. What is GPU Memory?

2. What is VRAM?

3. What is System RAM?

4. How Do System RAM and VRAM Work Together?

5. What is GPU Utilisation?

6. How to Monitor GPU Memory and Utilisation

7. Why Does This Matter in HPC and AI Infrastructure?

Final Thoughts

What Is an AI Factory? Understanding the Infrastructure Behind AI

1. What Exactly Is an AI Factory?

2. The Main Components of an AI Factory

Compute: CPUs and GPUs

Networking: Connecting the Compute Resources

Storage: Feeding Data to the GPUs

Software: Making Everything Work Together

Power and Cooling

3. How Do These Components Work Together?

4. What Makes an AI Factory Different from a Traditional HPC Cluster?

5. Where Does Slurm Fit Into an AI Factory?

6. Why GPU Utilisation Is Not the Whole Story

7. Why Is the AI Factory Concept Important?

Conclusion

CPU, Cores and Threads in HPC: Understanding What Your Job Actually Uses

1. What is a CPU?

2. What is a core?

3. What is a thread?

4. How does this work on Intel and AMD systems?

5. What does this mean for an HPC job?

6. How can you check the actual hardware?

The key takeaway

5 Slurm Configuration Parameters That Shape Your HPC Cluster

1. SelectType and SelectTypeParameters

2. PriorityWeight and Job Scheduling Priority

3. PartitionName

4. MaxTime

5. Accounting and Quality of Service (QoS)

Bringing it all together

Why Memory Bandwidth Matters in HPC

Memory capacity vs. memory bandwidth

Why does this matter in HPC?

A simple example

How can HPC engineers investigate it?

The bigger picture

Why More CPU Cores Don't Always Make an HPC Application Faster

Not every part of a program can run in parallel

What happens as we add more cores?

The problem isn't always the CPU

Strong scaling vs weak scaling

More cores can even make things worse

This is why benchmarking matters

The goal isn't always maximum CPU usage

So how many CPU cores should you use?

The bigger HPC lesson

Why Does HPC Need InfiniBand?

HPC applications communicate a lot

It's not just about bandwidth

Bandwidth

Latency

Where InfiniBand comes in

Why does that matter?

InfiniBand isn't the only option

The bigger picture

What Actually Happens After You Run sbatch?

sbatch submits the job

The job enters the scheduling system

Slurm looks at the job's request

The scheduler decides when the job can run

An allocation is created

The job reaches a compute node

This is where HPC gets interesting

From one command to an entire system

Kubernetes and Slurm are both popular for AI and HPC workloads, but they serve different purposes. This article explores their strengths, trade offs, and ideal use cases across scheduling, GPUs, distributed training, networking, and more.

Slurm vs Kubernetes: Understanding the Right Tool for the Right Workload

Slurm vs Kubernetes: Understanding the Right Tool for the Right Workload

Why This Comparison Matters

Understanding Their Different Goals

Architecture Comparison

Resource Scheduling

1. `SelectType` and `SelectTypeParameters`

2. `PriorityWeight` and Job Scheduling Priority

3. `PartitionName`

4. `MaxTime`