DEV Community

Cover image for What Actually Happens After You Run sbatch?
Muhammad Zubair Bin Akbar
Muhammad Zubair Bin Akbar

Posted on Originally published at hpcpulse.substack.com

What Actually Happens After You Run sbatch?

#ai #hpc #slurm #parallelcomputing

If you are learning Slurm, you probably start with a few commands:

sbatch job.sh
squeue
scancel <jobid>
Enter fullscreen mode Exit fullscreen mode

It is easy to think of sbatch as simply "running a job."

But that is not really what happens.

When you run:

sbatch job.sh
Enter fullscreen mode Exit fullscreen mode

you are submitting a request to Slurm.

What happens after that is where things get interesting.

sbatch submits the job

Suppose we have a simple script:

#!/bin/bash

#SBATCH --job-name=zubair-test
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00

./my_application
Enter fullscreen mode Exit fullscreen mode

When we run:

sbatch job.sh
Enter fullscreen mode Exit fullscreen mode

Slurm accepts the job request and gives it a job ID.

For example:

Submitted batch job 12345
Enter fullscreen mode Exit fullscreen mode

At this point, your application has not necessarily started.

The job has simply entered Slurm.

This is one of the most important concepts to understand:

Submitting a job is not the same as running a job.

The job enters the scheduling system

The Slurm controller now knows about your job.

You can see it with:

squeue
Enter fullscreen mode Exit fullscreen mode

You might see:

JOBID   PARTITION   NAME   USER           ST   TIME
12345   compute     test   zubair-test    PD   0:00
Enter fullscreen mode Exit fullscreen mode

The PD state means pending.

The job is waiting for an opportunity to run.

But why does it have to wait?

Because Slurm has to make a scheduling decision.

Slurm looks at the job's request

The scheduler needs to understand what the job is asking for.

In our example, the job requested:

  • 4 CPUs
  • 8 GB memory
  • 1 hour of runtime

But a real HPC job could request much more:

16 CPUs
128 GB memory
2 GPUs
4 hours
specific partition
specific features
Enter fullscreen mode Exit fullscreen mode

Slurm has to determine whether those requirements can be satisfied.

And this is where simply looking at "free CPUs" can become misleading.

A cluster might have idle CPUs but still not be able to run your job because of memory, GPU availability, partition limits, reservations, priorities, dependencies or other scheduling rules.

The scheduler decides when the job can run

Slurm's scheduler considers the jobs waiting in the system and the resources available on the cluster.

It is not simply doing this:

"There are 4 free CPUs, so run the job."

Instead, it has to consider the overall scheduling configuration and policies.

For example:

  • Which partition can run the job?
  • Does the job have enough priority?
  • Are the requested CPUs available?
  • Is enough memory available?
  • Does the job require GPUs?
  • Are there reservations?
  • Are there account or QoS restrictions?
  • Are other jobs already using the required resources?

This is why understanding Slurm requires more than memorising commands.

The commands are the interface.

The scheduler is the system underneath.

An allocation is created

Eventually, Slurm determines that the job can run.

It allocates the requested resources.

For example:

Node: compute-05
CPUs: 4
Memory: 8 GB
Time limit: 1 hour
Enter fullscreen mode Exit fullscreen mode

The job changes from:

PD

to:

R

where R means running.

Now the job has an allocation.

This is an important distinction:

The scheduler decides what resources the job gets.

The application then runs inside that allocation.

The job reaches a compute node

The actual application does not normally run on the Slurm controller.

The controller manages the cluster and scheduling decisions.

The job is launched on one or more compute nodes.

For example:

Slurm Controller
       |
       |
   Scheduler
       |
       |
   Allocation
       |
       |
   compute-05
       |
       |
   Your application
Enter fullscreen mode Exit fullscreen mode

Once the job starts on the compute node, a lot more of the HPC stack becomes involved.

Linux manages the processes.

The CPU and memory topology affects performance.

cgroups can control resource usage.

If GPUs are requested, Slurm's GRES configuration and GPU management become relevant.

If the application communicates across nodes, the network becomes part of the picture.

And if it reads or writes large amounts of data, the storage system becomes important.

This is where HPC gets interesting

Consider an MPI application running across 32 nodes.

Slurm has already done its job of allocating resources.

But now the application depends on many other parts of the system working together.

You might have:

  • CPUs and NUMA topology
  • memory
  • GPUs
  • InfiniBand or Ethernet
  • RDMA
  • MPI
  • UCX
  • parallel storage
  • Linux processes and cgroups

So when an HPC application is slow, the problem is not necessarily Slurm.

Slurm may have allocated the resources correctly.

The bottleneck could instead be CPU placement, memory access, network communication, storage I/O, GPU utilisation or the application itself.

That is why HPC administration involves understanding how these layers interact.

From one command to an entire system

This is what makes a simple command like:

sbatch job.sh
Enter fullscreen mode Exit fullscreen mode

much more interesting than it first appears.

You submit a job.

Slurm records the request.

The scheduler evaluates it.

Resources are allocated.

The job is launched on compute nodes.

Then Linux, CPUs, memory, GPUs, networking, storage and the application all become part of the execution.

So the next time you run:

sbatch job.sh
Enter fullscreen mode Exit fullscreen mode

remember that you are not simply "starting a program."

You are asking an entire resource management system to find a place for that program to run.

And understanding that process is one of the foundations of understanding HPC.

Top comments (0)