DEV Community: Muhammad Zubair Bin Akbar

How MPI Works Under the Hood (Without the Jargon)

Muhammad Zubair Bin Akbar — Fri, 01 May 2026 21:39:19 +0000

If you have ever run a job on an HPC cluster, chances are you have used MPI without fully knowing what’s happening behind the scenes. And that’s completely normal. MPI often feels like a black box that just “makes parallel jobs work.”

Let’s open that box a bit, without diving into heavy theory or academic jargon.

⸻

The Basic Idea

MPI (Message Passing Interface) is simply a way for multiple processes to talk to each other while running a program.

Think of it like this:

Instead of one program doing all the work, MPI lets you run many copies of the same program. Each copy handles a portion of the task and communicates with others when needed.

⸻

What Actually Happens When You Run an MPI Job?

When you launch an MPI job using something like:

mpirun -np 4 ./my_app

Here’s what’s going on under the hood:

1. Multiple Processes Are Started

MPI doesn’t create threads. It starts completely separate processes.

Each process:

Has its own memory space
Runs independently
Gets a unique ID called a rank

⸻

2. Each Process Knows Its Role

Every MPI process gets a rank:

Rank 0 → usually the coordinator
Rank 1, 2, 3… → workers

Your code uses these ranks to decide who does what.

⸻

3. Communication Happens via Messages

Processes don’t share memory. Instead, they send and receive messages.

Example:

Process 0 sends data → Process 1 receives it
Process 2 broadcasts something → everyone gets it

This is the core of MPI.

⸻

What Does “Sending a Message” Really Mean?

When one process sends data:

The data is copied into a buffer
MPI hands it to the system (network or shared memory)
It travels to the target process
The receiving process copies it into its memory

If processes are:

On the same node → shared memory is used
On different nodes → network (like InfiniBand or Ethernet)

⸻

How MPI Uses the Hardware

MPI is smarter than it looks. It adapts based on where processes are running:

Same Node

Uses shared memory (fast)
No real “network” involved

Different Nodes

Uses high-speed interconnects
Optimized protocols to reduce latency

Good MPI implementations automatically pick the best method.

⸻

Synchronization (Keeping Everyone in Check)

Sometimes processes need to wait for each other.

MPI provides mechanisms like:

Barriers → everyone pauses until all reach a point
Collective operations → like broadcast, reduce

This ensures coordination across processes.

⸻

A Simple Mental Model

Imagine a group project:

Each person (process) works on their part
They occasionally send updates to others
One person might collect results and combine everything

MPI is just the system that:

Assigns roles
Handles communication
Keeps things in sync

⸻

Why Things Sometimes Go Wrong

MPI issues often come from:

One process waiting for a message that never arrives
Mismatched send/receive calls
Network or node issues
Poor workload distribution

Because everything runs independently, small mistakes can cause hangs or failures.

⸻

Why MPI Is Still So Widely Used

Despite newer technologies, MPI remains dominant in HPC because:

It scales extremely well
Works across thousands of nodes
Gives precise control over communication
Is highly optimized for performance

⸻

Final Thoughts

MPI isn’t magic. It’s just a well-designed system for:

Running multiple processes
Passing messages between them
Coordinating work efficiently

Once you understand that, debugging and optimizing MPI jobs becomes much easier.

Bare Metal vs Virtual Machines vs Containers in HPC

Muhammad Zubair Bin Akbar — Thu, 30 Apr 2026 19:29:11 +0000

High Performance Computing isn’t just about powerful CPUs and fast interconnects. The way workloads are deployed matters just as much. Whether you’re running simulations, AI training, or large-scale data processing, choosing between bare metal, virtual machines, and containers can directly impact performance, flexibility, and efficiency.

Let’s break it down in a practical way.

Bare Metal: Maximum Performance, Minimum Abstraction

Bare metal means running workloads directly on physical hardware without any virtualization layer.

Why HPC loves it:

Full access to CPU, memory, GPUs, and high-speed networks
No virtualization overhead
Best for tightly coupled MPI jobs

Where it shines:

Large-scale simulations
CFD, weather modeling
Latency-sensitive workloads

Trade-offs:

Harder to manage at scale
Less flexible environment control
Software conflicts can become painful

Bare metal is still the gold standard in traditional HPC clusters, especially when every microsecond counts.

Virtual Machines: Isolation with Overhead

Virtual Machines (VMs) add a hypervisor layer, allowing multiple OS instances on the same hardware.

Why they’re used:

Strong isolation between workloads
Easy to snapshot, clone, and migrate
Good for multi-tenant environments

Where they fit in HPC:

Cloud-based HPC setups
Development and testing environments
Workloads that don’t need ultra-low latency

Trade-offs:

Performance overhead (CPU, I/O, networking)
Limited access to specialized hardware (unless using passthrough)

VMs are more common in cloud HPC than in on-prem clusters, where performance loss is less acceptable.

Containers: Lightweight and Portable

Containers package applications with their dependencies, running on the host OS without a full VM.

Why they’re gaining popularity:

Near bare-metal performance
Easy reproducibility
Portable across environments

Popular tools in HPC:

Docker (less common in production HPC)
Singularity / Apptainer (designed for HPC)

Where they shine:

AI/ML workloads
Research reproducibility
Complex software stacks

Trade-offs:

Shared kernel (less isolation than VMs)
Requires proper integration with schedulers like Slurm

Containers strike a strong balance between performance and flexibility, which is why they’re rapidly becoming standard in modern HPC environments.

Choosing the Right Model for Your Use Case

Instead of thinking about which one is “better,” it’s more useful to map each approach to real HPC scenarios.

Go with bare metal when:

You’re running tightly coupled MPI jobs across nodes
Network latency and bandwidth are critical
You need full GPU or accelerator performance
You’re operating a traditional on-prem HPC cluster

This is typical in scientific computing, engineering simulations, and large-scale physics workloads.

Use virtual machines when:

You’re running HPC workloads in the cloud
Multiple users or teams need strict isolation
You want to spin up environments quickly for testing
Performance is important, but not the absolute priority

VMs make sense in hybrid HPC setups or when infrastructure flexibility matters more than squeezing out every bit of performance.

Choose containers when:

You need reproducible environments across clusters
Your workloads depend on complex or conflicting libraries
You’re running AI/ML pipelines or modern data workloads
You want users to bring their own software stack easily

Containers are especially powerful in research environments where portability and consistency are critical.

The Real-World Approach

Most modern HPC environments don’t rely on just one model.

A common pattern looks like this:

Bare metal nodes for raw compute power
Containers for application portability
Virtual machines in cloud or hybrid layers

This hybrid approach gives you the best of all worlds: performance, flexibility, and scalability.

Final Thought

HPC is evolving beyond just hardware. The focus is shifting toward how efficiently workloads can be deployed and reproduced.

Bare metal still dominates performance-critical workloads. Containers are redefining usability and portability. Virtual machines fill the gap where flexibility and isolation are needed.

The right choice depends on what you’re optimizing for, not what’s trending.

What Actually Happens When You Run sbatch in Slurm

Muhammad Zubair Bin Akbar — Tue, 28 Apr 2026 20:40:47 +0000

If you work with HPC clusters, you likely use sbatch every day. You submit a script and expect it to run.

But that single command triggers a full workflow inside Slurm.

Understanding this internal flow helps you debug issues faster, optimize job performance, and better understand how your cluster behaves.

⸻

Step 1: Submitting the Job

When you run:

sbatch job.sh

You are not starting the job. You are submitting a request to Slurm.

The script includes:

Resource requirements such as CPUs, memory, GPUs
Job metadata like name and output paths
The actual commands to execute

At this point, Slurm simply accepts the job.

⸻

Step 2: Communication with slurmctld

The sbatch command sends the job to the Slurm controller daemon, slurmctld.

This daemon:

Assigns a Job ID
Stores the job details
Marks the job as PENDING

Nothing is running yet.

⸻

Step 3: Job Enters the Queue

The job is now placed in the scheduling queue.

evaluates:

Job priority
Fairshare usage
Partition limits
Resource availability

This determines when your job will run.

⸻

Step 4: Scheduling Decision

The scheduler continuously checks:

Free nodes
Resource fragmentation
Backfill opportunities

If your job fits available resources, it gets selected. Otherwise, it stays pending.

⸻

Step 5: Resource Allocation

Once selected, Slurm:

Assigns specific compute nodes
Reserves CPUs, memory, and GPUs
Changes job state to RUNNING

Now your job has allocated resources.

⸻

Step 6: Node-Level Communication

Each compute node runs a daemon called slurmd.

The controller sends job details to these nodes. The nodes prepare the execution environment.

⸻

Step 7: Job Execution via slurmstepd

On the compute node, slurmstepd is launched.

This process:

Starts your application
Manages job steps
Handles output and error streams
Enforces resource limits using cgroups

Your script begins executing here.

⸻

Step 8: Monitoring During Execution

While the job runs:

Slurm tracks resource usage
Logs are written to output files
Accounting data is collected

You can monitor the job using:

squeue
scontrol show job <jobid>

⸻

Step 9: Job Completion

When the job finishes:

slurmstepd exits
Resources are released
Temporary processes are cleaned up

The job state becomes COMPLETED, FAILED, TIMEOUT, or CANCELLED.

⸻

Step 10: Accounting and Logs

Finally:

Job statistics are stored
Output files remain available
Usage data is recorded

You can check this using:

sacct

⸻

Full Flow Summary

Submit job using sbatch
slurmctld receives and queues it
Scheduler evaluates priority
Resources are allocated
slurmd prepares nodes
slurmstepd runs the job
Job completes and resources are released

⸻

Common Misconceptions

“sbatch runs the job immediately”
It only submits the job.

“Pending means failure”
It usually means waiting for resources.

“Slurm just runs scripts”
It manages scheduling, allocation, execution, and cleanup.

⸻

Final Thought

sbatch may look simple, but it triggers a complete orchestration pipeline inside Slurm.

Once you understand this flow, debugging becomes easier, performance tuning improves, and cluster behavior becomes predictable.

⸻

4 Practical Boto3 Scripts for S3 Every DevOps Engineer Should Know

Muhammad Zubair Bin Akbar — Sun, 26 Apr 2026 21:54:05 +0000

Working with AWS S3 through the console is fine until you need automation, repeatability, and control. That’s where Boto3 comes in. In this post, we’ll walk through four practical Python scripts to manage S3 efficiently.

⸻

1. List All S3 Buckets with Creation Dates

A simple script to get visibility into your S3 environment.

import boto3
s3 = boto3.client('s3')
response = s3.list_buckets()
print("S3 Buckets:\n")
for bucket in response['Buckets']:
    print(f"Name: {bucket['Name']} | Created On: {bucket['CreationDate']}")

Why this matters:

Useful for audits, inventory tracking, or quick checks across accounts.

⸻

2. Upload a File to S3 with Error Handling

Uploading files is common but handling failures properly is what makes scripts production-ready.

import boto3
from botocore.exceptions import FileNotFoundError, NoCredentialsError, ClientError
s3 = boto3.client('s3')
file_name = "test.txt"
bucket_name = "your-bucket-name"
object_name = "uploads/test.txt"
try:
    s3.upload_file(file_name, bucket_name, object_name)
    print("File uploaded successfully.")
except FileNotFoundError:
    print("The file was not found.")
except NoCredentialsError:
    print("Credentials not available.")
except ClientError as e:
    print(f"AWS Error: {e}")

Why this matters:

Prevents silent failures and gives clear debugging output.

⸻

3. Download Files from S3 with Progress Tracking

For large files, progress tracking makes a big difference.

import boto3
s3 = boto3.client('s3')
bucket_name = "your-bucket-name"
object_name = "large-file.zip"
file_name = "downloaded.zip"
def progress_callback(bytes_transferred):
    print(f"Transferred: {bytes_transferred} bytes")
s3.download_file(
    bucket_name,
    object_name,
    file_name,
    Callback=progress_callback
)
print("Download complete.")

Why this matters:

Gives visibility into long running downloads especially useful in automation pipelines.

⸻

4. Create and Delete S3 Buckets Programmatically

Automating bucket lifecycle management is useful in testing and dynamic environments.

import boto3
from botocore.exceptions import ClientError
s3 = boto3.client('s3')
bucket_name = "my-unique-bucket-name-12345"
# Create Bucket
try:
    s3.create_bucket(
        Bucket=bucket_name,
        CreateBucketConfiguration={
            'LocationConstraint': 'eu-west-1'
        }
    )
    print("Bucket created successfully.")
except ClientError as e:
    print(f"Error creating bucket: {e}")
# Delete Bucket
try:
    s3.delete_bucket(Bucket=bucket_name)
    print("Bucket deleted successfully.")
except ClientError as e:
    print(f"Error deleting bucket: {e}")

Note:

Make sure the bucket is empty before deleting, otherwise the delete operation will fail.

⸻

Final Thoughts

These four scripts cover the most common S3 operations:

Visibility (listing buckets)
Data movement (upload/download)
Resource lifecycle (create/delete)

They’re simple, but extremely useful when building automation around AWS.

As you scale, you can extend these with:

Logging
Retry mechanisms
Parallel uploads/downloads

This is the kind of practical automation that saves time in real environments.

The Future of Coding in the Age of AI: What Developers Need to Know

Muhammad Zubair Bin Akbar — Sat, 25 Apr 2026 17:14:35 +0000

AI is no longer a “future trend” in software development — it’s already here, integrated into how code is written, reviewed, and deployed.

From auto-completing functions to generating entire applications, AI tools are changing how developers work. The real question is not whether AI will replace developers, but how the role of developers is evolving.

This shift affects everyone — from web developers to game developers to system engineers.

The Shift: From Writing Code to Designing Systems

Traditionally, development meant:

Writing logic line by line
Debugging manually
Managing boilerplate code

Now, AI tools can:

Generate code snippets instantly
Suggest fixes and optimizations
Handle repetitive tasks

This changes the developer’s role from code writer to:

System designer
Problem solver
Architecture thinker

The focus is moving toward what to build and how systems fit together, rather than just how to write code.

AI as a Productivity Multiplier

AI is best understood as a tool that increases output, not replaces skill.

What It Does Well

Boilerplate generation
Documentation assistance
Code suggestions
Basic debugging

What Still Requires Humans

Designing scalable systems
Making trade-offs
Understanding real-world requirements
Debugging complex, non-obvious issues

Developers who use AI effectively can:

Build faster
Experiment more
Focus on higher-value work

Impact Across Different Domains

Web Development

Faster UI scaffolding
Backend APIs generated quickly
Improved testing and documentation

Shift: More focus on architecture, performance, and user experience.

Game Development

Procedural content generation
AI-assisted asset creation
Faster prototyping

Shift: More emphasis on creativity, design, and gameplay mechanics.

Software Engineering / Systems

Infrastructure as code generation
Automation scripts
Faster troubleshooting assistance

Shift: More focus on system reliability, scalability, and integration.

The Skills That Will Matter More

As AI handles repetitive coding, certain skills become more valuable:

1. Problem Understanding

Clearly defining problems becomes critical.
AI is only as good as the input it receives.

2. System Design

Understanding how components interact:

APIs
Databases
Distributed systems

3. Debugging and Validation

AI-generated code is not always correct.

Developers must:

Verify outputs
Identify edge cases
Ensure correctness

4. Performance Optimization

AI can generate working code, but not always efficient code.

Optimization remains a human-driven task.

5. Communication

Explaining systems, writing clear prompts, and collaborating with teams becomes more important.

What Developers Need to Do to Stay Relevant

The shift toward AI-assisted development means adapting your approach, not resisting it.

1. Learn How to Use AI Tools Effectively

AI is becoming part of the development workflow.

Use it for productivity, not dependency
Understand its limitations
Review and refine generated code

2. Strengthen Fundamentals

Core knowledge becomes even more important:

Operating systems
Networking
Data structures and algorithms
System design

These are areas AI cannot replace easily.

3. Focus on Real Problem Solving

Move beyond tutorials and boilerplate projects.

Work on real-world use cases
Build systems, not just scripts
Understand trade-offs

4. Develop Debugging Skills

When AI-generated code fails, you need to fix it.

Read logs
Trace issues
Understand root causes

5. Think in Systems, Not Just Code

Understand how everything connects:

Frontend ↔ Backend
Application ↔ Infrastructure
Code ↔ Performance

6. Keep Learning and Adapting

The pace of change is increasing.

Stay updated with tools and trends
Experiment regularly
Be flexible in your approach

Common Misconceptions

“AI Will Replace Developers”

Unlikely in the near term.

AI lacks:

Context awareness
Deep problem understanding
Accountability

“Coding Skills Will Become Irrelevant”

Coding is still essential, but the nature of coding is changing.

Knowing how systems work under the hood will remain valuable.

“AI Makes Everything Faster Automatically”

Only if used correctly.

Poor usage can lead to:

Bad code
Security issues
Hidden bugs

Realistic Future: What to Expect

Short Term

AI becomes a standard part of developer workflows
Increased productivity
Faster development cycles

Mid Term

Smaller teams building larger systems
More emphasis on architecture and design
AI-assisted debugging and optimization improves

Long Term

Developers act more like system architects
AI handles a larger portion of implementation
Creativity and problem-solving become the main differentiators

Final Thoughts

AI is not removing the need for developers — it is reshaping the role.

The developers who thrive will be those who:

Understand systems deeply
Use AI as a tool, not a crutch
Focus on solving meaningful problems

Coding is not going away.
But the way we approach it is changing — and quickly.

Adapting to this shift is essential for staying relevant in modern software development.

Running Slurm on AWS/Azure: Architecture & Pitfalls

Muhammad Zubair Bin Akbar — Fri, 24 Apr 2026 20:53:48 +0000

Running Slurm in the cloud sounds simple at first: spin up some VMs, install Slurm, and start submitting jobs.

In reality, cloud-based HPC introduces a different set of design decisions and trade-offs compared to on-prem clusters. If the architecture is not planned properly, costs increase quickly and performance can drop.

This guide walks through a typical Slurm architecture on AWS/Azure and highlights the most common pitfalls.

Why Run Slurm in the Cloud?

Common reasons include:

On-demand scaling for peak workloads
No upfront hardware investment
Access to GPU instances when needed
Flexibility for short-term projects

However, cloud HPC is not always cheaper or faster — it depends heavily on how it is configured.

Typical Slurm Architecture in Cloud

A standard setup usually includes:

1. Head Node (Controller)

Runs slurmctld
Manages scheduling and job queues
Typically a small-to-medium VM

Key Point:
This node should be stable and always available.

2. Compute Nodes

Dynamically provisioned instances
Can be CPU or GPU-based
Often scaled up/down based on demand

Common Approach:

Auto-scaling groups (AWS)
Virtual Machine Scale Sets (Azure)

3. Login Node

User access via SSH
Job submission and monitoring

This is often combined with the head node in smaller setups, but separated in production environments.

4. Shared Storage

Required for:

Input/output data
Job scripts
Application binaries

Options:

AWS: EFS, FSx (Lustre)
Azure: Azure NetApp Files, Azure Files, Lustre

5. Networking

Virtual Private Cloud (AWS) / Virtual Network (Azure)
Security groups / NSGs
High-speed networking (placement groups, accelerated networking)

Basic Workflow

User connects to login node
Submits job using sbatch
Slurm provisions compute nodes (if not already running)
Job runs on allocated instances
Nodes are terminated after job completion (optional)

Recommended Architecture Pattern

For most use cases:

Persistent head/login node
Auto-scaling compute nodes
Shared parallel storage
Private network with restricted access

This balances cost, performance, and manageability.

Common Pitfalls (and How to Avoid Them)

1. Ignoring Network Performance

Problem

Using standard cloud networking for MPI workloads.

Impact

High latency
Poor scaling across nodes

Fix

Use placement groups (AWS) or proximity placement groups (Azure)
Enable enhanced/accelerated networking
Choose HPC-optimized instance types

2. Storage Becomes the Bottleneck

Problem

Using basic network storage for high I/O workloads.

Impact

Slow reads/writes
Idle compute nodes

Fix

Use parallel file systems (FSx for Lustre, Azure Lustre)
Match storage throughput with compute scale

3. Poor Auto-Scaling Configuration

Problem

Nodes take too long to start or are over-provisioned.

Impact

Increased wait times
Higher costs

Fix

Tune scaling policies
Keep a small number of warm nodes
Use instance pools where possible

4. Using the Wrong Instance Types

Problem

Choosing general-purpose VMs for HPC workloads.

Impact

Lower performance
Inefficient scaling

Fix

Use compute-optimized or HPC-specific instances
For GPUs, select instances with proper interconnect support

5. Ignoring Cost Management

Problem

Leaving nodes running after jobs finish.

Impact

Unexpected cloud bills

Fix

Enable auto-termination of idle nodes
Use spot/preemptible instances where suitable

6. Not Handling Preemption (Spot Instances)

Problem

Using spot instances without fault tolerance.

Impact

Job failures
Lost progress

Fix

Use checkpointing
Combine on-demand + spot nodes

7. Single Point of Failure (Head Node)

Problem

Head node goes down → entire cluster stops.

Fix

Use backups or snapshots
Consider failover strategies

8. Security Misconfiguration

Problem

Open SSH access or weak network rules.

Impact

Security risks

Fix

Restrict access via VPN or IP whitelisting
Use IAM roles and proper authentication

9. Slow Job Startup Times

Problem

VM provisioning delays job execution.

Impact

Poor user experience

Fix

Pre-scale nodes
Use lightweight images
Optimize bootstrapping scripts

10. Treating Cloud Like On-Prem

Problem

Applying static cluster design to a dynamic environment.

Impact

Inefficiency
Higher costs

Fix

Design for elasticity
Scale based on workload demand

Real-World Example

Initial Setup:

Static compute nodes
Standard storage
No placement group

Issues:

Poor MPI scaling
High costs

Improved Setup:

Auto-scaling compute nodes
FSx for Lustre storage
Placement group enabled

Result:

Better performance
Reduced costs
Faster job turnaround

Final Thoughts

Running Slurm on AWS or Azure can be powerful, but it is not just about lifting and shifting your on-prem setup.

Success depends on:

Choosing the right architecture
Understanding cloud limitations
Avoiding common pitfalls

With the right design, cloud-based Slurm clusters can deliver both flexibility and performance — without unnecessary cost.

Designing HPC Cluster Networking: What Speeds You Actually Need

Muhammad Zubair Bin Akbar — Thu, 23 Apr 2026 21:04:18 +0000

Designing HPC Cluster Networking: What Speeds You Actually Need

When building or scaling an HPC cluster, CPUs and GPUs usually get most of the attention.

But in practice, the network design is just as critical. A poorly designed network can bottleneck even the most powerful compute nodes, while a well designed one can significantly improve performance without changing hardware.

This guide breaks down typical networking components in an HPC cluster and what speeds are generally recommended between them.

⸻

Why Networking Matters in HPC

In HPC environments, nodes rarely work in isolation.

They constantly exchange data for:

MPI communication
Distributed AI/ML training
Accessing shared storage

If the network cannot keep up, nodes spend time waiting instead of computing.

⸻

Key Network Paths in an HPC Cluster

Let’s break the cluster into major communication paths:

Compute Node ↔ Compute Node (Interconnect)
Compute Node ↔ Storage
Login Node ↔ Compute Nodes
External Access (Users ↔ Login Node)

Each of these has different requirements.

⸻

1. Compute Node ↔ Compute Node (Interconnect)

This is the most critical network in HPC.

It handles:

MPI traffic
Synchronization between processes
Distributed workloads

Recommended Speeds

Minimum: 25 Gbps
Common: 100 Gbps
High-end: 200–400 Gbps

Technologies

InfiniBand (very low latency)
Omni-Path
High-speed Ethernet (RoCE, RDMA-enabled)

Key Focus

Low latency is more important than raw bandwidth
RDMA support is highly recommended

Impact

Poor interconnect leads to:

Poor scaling
High communication overhead
Underutilized CPUs/GPUs

⸻

2. Compute Node ↔ Storage

Handles:

Reading input datasets
Writing results
Checkpoints

Recommended Speeds

Minimum: 10–25 Gbps
Typical: 40–100 Gbps
High-performance setups: 100+ Gbps

Storage Types

NFS (basic setups)
Lustre / BeeGFS / GPFS (parallel file systems)

Key Considerations

Throughput matters more than latency
Parallel file systems scale better than NFS

Impact

If storage is slow:

Jobs stall during I/O
GPUs sit idle waiting for data

⸻

3. Login Node ↔ Compute Nodes

Role

Job submission
Monitoring
Light data movement

Recommended Speeds

1–10 Gbps is usually sufficient

Notes

This path is not performance-critical
Should be isolated from high-speed compute traffic

⸻

4. External Access (User ↔ Login Node)

Role

SSH access
File transfers
Development workflows

Recommended Speeds

Depends on environment
Typically 1–10 Gbps uplink

Considerations

Security is more important than speed here
Use firewalls, VPNs, and access controls

⸻

Network Design Approaches

1. Single Network (Simple Setup)

One network for everything
Lower cost
Easier to manage

Downside:
Traffic contention between compute, storage, and users

⸻

2. Dual Network (Recommended)

High-speed network for compute + storage
Separate Ethernet network for management

Benefits:

Better performance
Reduced congestion
More predictable behavior

⸻

3. Dedicated Storage Network (Advanced)

Separate network just for storage traffic

Used in:

Large clusters
Data-intensive workloads

⸻

Latency vs Bandwidth (Important Distinction)

Latency: Time to send a message
Bandwidth: Amount of data transferred

In HPC:

MPI workloads → sensitive to latency
Data-heavy workloads → depend on bandwidth

A fast network with high latency can still perform poorly for MPI jobs.

⸻

Common Mistakes in HPC Networking

Using standard Ethernet without RDMA for MPI workloads
Mixing storage and compute traffic on the same link
Underestimating storage bandwidth needs
Ignoring network topology (oversubscription issues)
Not validating actual performance with benchmarks

⸻

Practical Example

Cluster Setup:

16 compute nodes
GPU workloads + MPI

Network Design:

100 Gbps InfiniBand for inter-node communication
100 Gbps link to parallel storage
1 Gbps management network

Result:

Efficient scaling across nodes
Reduced job runtime
Stable performance under load

⸻

Final Thoughts

HPC networking is not just about choosing the fastest hardware.

It is about:

Matching the network to your workload
Separating traffic intelligently
Avoiding bottlenecks before they appear

In many cases, upgrading or redesigning the network delivers more performance improvement than upgrading CPUs or GPUs.

If your cluster is not scaling as expected, the network is often the first place to look.

Top 10 Slurm Mistakes That Kill Cluster Performance

Muhammad Zubair Bin Akbar — Wed, 22 Apr 2026 18:29:34 +0000

Slurm is designed to make efficient use of cluster resources.
But in practice, a few common mistakes can quietly destroy performance — not just for one user, but for the entire cluster.

The tricky part is that most of these don’t cause failures. Jobs still run… just slower, inefficiently, or at the cost of others.

Here are 10 of the most common Slurm mistakes and how to fix them.

1. Over-Requesting Resources

The Problem

Requesting more CPUs, memory, or GPUs than needed.

#SBATCH --cpus-per-task=32
#SBATCH --mem=128G

When your job only uses a fraction of that.

Impact

Longer queue times
Wasted resources
Lower overall cluster utilization

Fix

Profile your job and request only what you actually need.

2. Under-Requesting Memory

The Problem

Requesting too little memory.

Impact

Job crashes (OOM)
Wasted compute time
Repeated retries

Fix

Monitor memory usage and add a buffer:

#SBATCH --mem=8G

3. Running Jobs on Login Nodes

The Problem

Running heavy workloads directly on login nodes.

Impact

Slows down the entire system
Affects all users

Fix

Always use Slurm:

sbatch job.sh

4. Ignoring CPU Binding

The Problem

Processes are not bound to cores.

Impact

Context switching
Cache inefficiency
Lower CPU utilization

Fix

srun --cpu-bind=cores ./app

5. Poor Parallelization Choices

The Problem

Using too many tasks for a workload that doesn’t scale.

Impact

Communication overhead
Worse performance than fewer cores

Fix

Test scaling before increasing resources blindly.

6. Hardcoding Specific Nodes

The Problem

#SBATCH --nodelist=node01

Impact

Jobs stuck pending
Reduced scheduler flexibility

Fix

Let Slurm decide placement unless absolutely necessary.

7. Not Using Job Arrays

The Problem

Submitting hundreds of similar jobs manually.

Impact

Scheduler overload
Inefficient job handling

Fix

Use job arrays:

#SBATCH --array=1-100

8. Setting Unrealistic Time Limits

The Problem

Too short → jobs get killed
Too long → blocks scheduling

Impact

Wasted compute time
Increased queue delays

Fix

Estimate runtime realistically.

9. Ignoring Job Output and Errors

The Problem

Not checking logs.

#SBATCH --output=output.log
#SBATCH --error=error.log

Impact

Silent failures
Poor debugging

Fix

Always review logs after job completion.

10. Not Monitoring Jobs

The Problem

Submitting jobs and not tracking them.

Impact

Missed failures
Inefficient usage

Fix

Use:

squeue -u $USER
scontrol show job <job_id>

Real-World Scenario

Before:

Over-requested CPUs
No binding
Poor scaling

Result:

50% CPU utilization
Long queue times

After Fixes:

Right-sized resources
Proper binding
Optimized parallelism

Result:

Higher utilization
Faster job completion
Better cluster efficiency

Final Thoughts

Slurm itself is rarely the problem.

Most performance issues come from how jobs are submitted and configured.

Avoiding these common mistakes can:

Improve your job performance
Reduce wait times
Make the entire cluster more efficient

Small changes in job scripts can have a big impact — not just for you, but for everyone using the cluster.

Optimizing MPI Performance (Real Examples)

Muhammad Zubair Bin Akbar — Tue, 21 Apr 2026 20:42:56 +0000

MPI jobs that run are easy.
MPI jobs that run fast and efficiently — that’s where things get interesting.

If your application scales poorly, takes longer than expected, or wastes CPU time, the issue is usually not the code itself… it’s how it’s running.

That said, performance tuning is rarely about a single fix. The examples below highlight common issues and improvements, but in real-world HPC workloads, these are often just one of several factors impacting performance.

Here’s a practical breakdown of MPI performance tuning, with real examples you can apply immediately.

Where MPI Performance Actually Breaks

Most MPI slowdowns come from:

Poor process placement
Network bottlenecks
Imbalanced workloads
Excessive communication
Memory/NUMA issues

The tricky part is that these don’t show up as errors — just slow jobs.

1. CPU Binding & Process Placement

The Problem

MPI processes float across CPUs, leading to cache misses and context switching.

The Fix

Bind processes to cores explicitly.

mpirun --bind-to core --map-by socket -np 32 ./app

On Slurm:

srun --cpu-bind=cores ./app

Real Impact

Before: ~65% CPU efficiency
After: ~90%+ CPU utilization

2. NUMA Awareness (Hidden Performance Killer)

On multi-socket systems, memory is not equal.

The Problem

A process runs on one socket but accesses memory from another, increasing latency.

The Fix

Use NUMA-aware mapping:

mpirun --map-by ppr:1:numa ./app

Check layout:

numactl --hardware

Real Impact

Reduced memory latency
Better scaling across nodes

3. Network Optimization (InfiniBand / Omni-Path)

MPI performance heavily depends on interconnect.

The Problem

MPI falls back to TCP instead of using high-speed fabric.

The Fix

Set the correct transport layer.

For Intel MPI:

export I_MPI_FABRICS=shm:ofi

For OpenMPI:

mpirun --mca pml ucx --mca btl ^tcp ./app

Real Impact

Lower latency
Faster multi-node scaling

4. Load Imbalance Between Processes

The Problem

Some ranks finish early while others continue working, leaving CPUs idle

Detect It

mpirun -np 4 ./app

If one rank consistently lags, there is imbalance.

The Fix

Distribute work evenly
Use dynamic scheduling where possible

Real Impact

Even a small imbalance can reduce performance by 30–50%.

5. Too Much Communication

MPI applications often slow down due to excessive messaging.

The Problem

Frequent small messages create high communication overhead.

The Fix

Batch messages
Use collective operations such as:
- MPI_Bcast
- MPI_Reduce

Real Example

Replacing multiple MPI_Send calls with a single MPI_Bcast significantly improved runtime in real workloads.

6. Benchmark Before You Guess

Avoid blind optimization. Measure first.

Useful Tools

mpirun --report-bindings
Intel MPI Benchmarks: IMB-MPI1
OSU Micro-Benchmarks: osu_latency

What to Look For

Latency
Bandwidth
CPU utilization

7. Slurm-Specific Optimization

If you are using Slurm, your job script plays a critical role.

Example

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=1

srun --cpu-bind=cores ./app

Key Tips

Match ntasks with available cores
Avoid oversubscription
Use --exclusive for consistent performance

Real Scenario (Before vs After)

Before Optimization:

No CPU binding
Default MPI settings
TCP communication
Runtime: 120 minutes

After Optimization:

Core binding enabled
NUMA-aware mapping
High-speed fabric (OFI/UCX)
Reduced communication

Runtime reduced to approximately 70 minutes, without any changes to the application code.

Final Takeaway

MPI performance is rarely about rewriting your application.

It is about:

Running it on the right cores
Using the right network
Avoiding unnecessary overhead

Small configuration changes can lead to significant improvements, but real-world performance is always influenced by multiple factors working together.

If your MPI job feels slower than expected, the limitation is often not the hardware — it is how efficiently it is being used.

Why Your Slurm Jobs Stay Pending (and How to Actually Fix It)

Muhammad Zubair Bin Akbar — Mon, 20 Apr 2026 21:29:20 +0000

If you’ve worked with Slurm long enough, you’ve definitely seen this:

PD (Pending)

You submit a job, everything looks fine… and then nothing happens.

No errors. No logs. Just waiting.

Let’s break down why this happens and, more importantly, how to fix it without guessing.

What “Pending” Actually Means

A Slurm job in PENDING (PD) state simply means:

The scheduler hasn’t found a suitable way to run your job yet.

That could be due to:

Resource shortages
Configuration limits
Priority issues
Or constraints you didn’t even realize you set

The key is: Slurm always tells you why — you just need to ask the right way.

Step 1: Check the Real Reason

Run:

squeue -j <job_id> -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"

Look at the NODELIST(REASON) column.

Common outputs:

(Resources)
(Priority)
(ReqNodeNotAvail)
(QOSMaxJobsPerUserLimit)

This reason is your starting point — not the guesswork.

Most Common Reasons (and Fixes)

1. (Resources) — Not Enough Resources Available

Meaning:
Your job is asking for more than what’s currently free.

Example:

Too many CPUs
Too much memory
GPU request when none are available

Fix:
Reduce your request:

#SBATCH --cpus-per-task=4
#SBATCH --mem=8G

Or wait (if the request is valid but large)

Pro tip: Check cluster usage with:

sinfo

2. (Priority) — Your Job Is in Line

Meaning:
Other jobs have higher priority than yours.

Slurm prioritizes based on:

Fairshare
Job age
Partition rules
QOS

Fix:

Check priority:

sprio -j <job_id>

If possible:
- Use a different partition
- Reduce requested resources (smaller jobs start faster)

3. (ReqNodeNotAvail) — Requested Node Is Not Usable

Meaning:
You requested a node that is:

Down
Drained
Reserved

Fix:

Avoid hardcoding nodes:

#SBATCH --nodelist=node01   ❌

Check node state:

sinfo -R

(QOSMaxJobsPerUserLimit) — You Hit a Limit

Meaning:
You’ve reached the maximum number of jobs allowed.

Fix:

Check your running jobs:

squeue -u $USER

Wait or cancel unnecessary jobs
Talk to your admin if limits are too restrictive

(PartitionLimit) — Partition Constraints

Meaning:
Your job exceeds partition limits (time, memory, nodes).

Fix:

Check partition config:

sinfo -o "%P %l %m %c"

Adjust your script:

#SBATCH --time=01:00:00

Advanced Debugging (Admins & Power Users)

If the reason isn’t obvious:

Check job details:

scontrol show job <job_id>

Look at scheduler decisions:

sdiag

Check Slurm logs:

slurmctld.log
slurmd.log

These often reveal hidden issues like:

Invalid accounts
Association limits
Misconfigured QOS

Real-World Tip: Smaller Jobs Start Faster

Slurm prefers jobs that can fit quickly.

If your job asks for:

2 GPUs → might wait hours
1 GPU → might start immediately

Strategy:
Break large jobs into smaller chunks when possible.

Quick Checklist

Before blaming Slurm, check:

Did I request too many resources?
Am I hitting a user/job limit?
Is my priority too low?
Did I accidentally constrain nodes?
Does my partition allow this job?

Final Thought

Slurm isn’t “stuck” when jobs are pending — it’s being strict and logical.

The difference between a beginner and an experienced HPC user is simple:

Beginners wait. Experts check the reason and fix it.

A Simple OpenClaw Workflow for HPC Job Analysis

Muhammad Zubair Bin Akbar — Thu, 16 Apr 2026 17:51:55 +0000

This is a submission for the OpenClaw Challenge.

What I Built

I built a simple automation workflow using OpenClaw to assist with HPC job troubleshooting.

In HPC environments, users often struggle with failed jobs, unclear error logs, and resource misconfigurations. The idea was to create a small assistant that can take job logs or error messages and provide quick, actionable insights.

How I Used OpenClaw

Demo

I used OpenClaw to create a workflow that:

Accepts Slurm job logs or error outputs
Processes common failure patterns (e.g. out-of-memory, pending jobs, GPU issues)
Returns simplified explanations along with possible fixes

The setup included:

A basic input interface for logs
Prompt-based logic to classify issues
Structured responses focused on real-world HPC troubleshooting

The goal was not to replace debugging, but to speed up the initial analysis.

What I Learned

A few interesting takeaways:

Many HPC issues follow repeatable patterns, which makes them ideal for automation
Translating complex system errors into simple explanations is surprisingly valuable
Even a lightweight workflow can save time for both users and admins

It also showed how tools like OpenClaw can fit into infrastructure and operations workflows, not just typical app use cases.

ClawCon Michigan

Did not attend, but following the OpenClaw ecosystem and exploring how it can be applied to real-world infrastructure use cases like HPC.

What Makes HPC Different from Cloud or Traditional Servers

Muhammad Zubair Bin Akbar — Thu, 16 Apr 2026 16:55:23 +0000

At a glance, High Performance Computing (HPC), cloud platforms, and traditional servers might seem similar. After all, they all involve running workloads on machines.

But in practice, they are built for different purposes.

Also, an important point: HPC is not tied to on-prem environments anymore. Cloud providers like AWS and Azure now offer HPC solutions through tools like ParallelCluster and CycleCloud.

So the real difference is not where HPC runs, but how the workloads behave and how the systems are designed.

Let’s break it down.

What Traditional Servers Are Designed For

Traditional servers are built to handle:

Web applications
Databases
File storage
Enterprise applications

These workloads are usually:

Long-running
Service-based (always on)
Independent from each other

Each server handles its own tasks, and communication between servers is limited.

What Cloud Platforms Focus On

Cloud platforms like AWS or Azure are designed for:

Cloud platforms focus on:

Scalability
Flexibility
On-demand infrastructure

You can:

Launch instances anytime
Scale resources quickly
Pay based on usage

Most cloud workloads are:

Loosely coupled
Stateless or microservice-based

Where HPC Fits In

HPC is designed for a different goal:

*Solving large, compute-intensive problems as fast as possible.
*
This changes how systems are built and used.

And importantly:

HPC can run on-prem OR in the cloud

On-prem → dedicated clusters
Cloud → managed HPC environments (e.g., AWS ParallelCluster, Azure CycleCloud)

So HPC is more about architecture and workload type, not location.

1. Workloads Are Parallel and Tightly Coupled

In HPC, a single job is often split across multiple nodes.

These nodes:

Work on the same problem
Exchange data continuously
Depend on each other

If communication is slow, the entire job slows down.

This is very different from cloud or traditional systems where tasks are mostly independent.

2. The Network Is Part of the Compute

In typical systems:

Network = data transfer

In HPC:

Network = part of computation

High-speed interconnects (like InfiniBand or optimized cloud networking) enable:

Low latency
High bandwidth
Efficient data exchange

Even in cloud HPC setups, networking configuration plays a huge role in performance.

3. Job Scheduling Instead of Always-On Services

In HPC, workloads are submitted as jobs:

Jobs enter a queue
Scheduler (like Slurm) assigns resources
Jobs run when resources are available

In contrast:

Traditional servers → always running services
Cloud → on-demand instances

Even in cloud HPC (ParallelCluster, CycleCloud), this job-based model remains the same.

4. Resource Allocation Is Explicit

In HPC, you must define:

CPUs
Memory
GPUs
Runtime

Example:

#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=02:00:00

This ensures fair usage across shared environments.

This model applies whether the cluster is:

On-prem
Or deployed in the cloud

5. Performance Over Flexibility

Cloud (general purpose):

Flexible
Easy to scale

HPC:

Performance-focused
Optimized for efficiency

Even in cloud HPC setups:

Instances are carefully chosen
Networking is tuned
Storage is optimized

It is not just “spin up and run”.

6. Storage Is Built for Throughput

Traditional storage:

Optimized for transactions

HPC storage:

Optimized for parallel access

Parallel file systems allow:

Multiple nodes to read/write simultaneously
High throughput for large datasets

Cloud HPC often replicates this using:

High-performance shared storage
Parallel file system integrations

7. Cost Model Is Different

The confusion often comes from mixing platform and workload type.

Here’s the clearer view:

Traditional servers → run services
Cloud → provides flexible infrastructure
HPC → defines how compute-heavy workloads are executed

And today:

HPC can run on both on-prem clusters AND cloud platforms

Final Thoughts

HPC is not just “more powerful servers” or “a type of cloud”.

It is a different computing model where:

Workloads are parallel
Communication is critical
Performance is the priority

Cloud platforms now make HPC more accessible, but they do not change its core principles.

So when comparing HPC with cloud or traditional systems, the real question is not where it runs, but what kind of workload you are solving.