DEV Community: hosni1982

Why High Load Does Not Always Mean High CPU

hosni1982 — Sat, 05 Sep 2026 13:07:37 +0000

Why High Load Does Not Always Mean High CPU

One of the first things I check when someone says "the Linux server is slow" is the load average.

And one of the most common mistakes is to immediately translate high load into high CPU usage.

These two things are related, but they are not the same.

I have seen production servers with a load average above 30 while the CPUs were mostly idle. In cases like this, adding CPU or killing the process at the top of top would not have solved anything.

The first question should be simpler:

What are the processes actually waiting for?

Start with the symptom

Suppose we connect to a server and get this:

$ uptime
14:32:18 up 47 days, 3:21, 4 users, load average: 31.42, 28.77, 21.06

A load of 31 immediately gets attention.

So we check CPU usage:

$ top
%Cpu(s): 6.2 us, 2.8 sy, 0.0 ni, 84.1 id, 6.7 wa, 0.2 hi

More than 80% of the CPU is idle.

That should change the direction of the investigation.

The machine has a high load, but it is not CPU saturated.

What load average is actually telling us

On Linux, load average is not simply a measure of CPU consumption.

It includes tasks that are runnable, but also tasks stuck in uninterruptible sleep — typically processes waiting for some kernel operation or I/O to complete.

This is why looking only at:

top

or:

ps aux --sort=-%cpu

can send the investigation in the wrong direction.

I usually check the process states next.

$ ps -eo state,pid,ppid,comm,wchan:32 | head -30

Or, for a quick count:

$ ps -eo state= | sort | uniq -c

Imagine we get:

      1 R
    146 S
     27 D

The interesting number here is not R.

It is the 27 processes in D state.

D state changes the investigation

D means uninterruptible sleep.

A process in this state is usually waiting inside the kernel for something to finish. Storage is a common cause, although it is not the only possible one.

At this point, I would not restart the application yet.

I would first identify the affected processes:

$ ps -eo pid,ppid,state,wchan:32,comm | awk '$3=="D"'

Then look for a pattern.

Are they all application processes?

Are they accessing the same filesystem?

Did they all become blocked around the same time?

For example:

PID    PPID S WCHAN                            COMMAND
18421     1 D xfs_file_buffered_aio_write     java
18476     1 D xfs_file_buffered_aio_write     java
18503     1 D wait_on_page_bit_common         java

Now we have something much more useful than "load is high."

The application threads are waiting on filesystem or storage operations.

Follow the dependency chain

This is where production troubleshooting becomes more interesting.

The visible symptom may be:

Application is slow

but the actual chain could be:

application
    ↓
filesystem
    ↓
logical volume
    ↓
multipath device
    ↓
SAN path
    ↓
storage array

If the application is waiting because the storage layer is degraded, restarting the application only removes the symptom temporarily — and sometimes makes the incident worse.

I would continue with basic I/O checks:

iostat -xz 1

and:

vmstat 1

For SAN-backed storage:

multipath -ll

I would also check the kernel messages:

journalctl -k --since "-30 min"

or:

dmesg -T | tail -100

Messages about path failures, I/O timeouts, SCSI errors or device resets can quickly move the investigation toward the real failure layer.

Don't trust one metric

This is the main lesson.

A high load average is an observation. It is not a root cause.

I try to separate these three questions:

What do I see?
        ↓
What does the evidence prove?
        ↓
Which layer is actually failing?

For high load, that means correlating at least:

uptime
top
vmstat 1
ps -eo state,pid,ppid,comm,wchan:32
iostat -xz 1
journalctl -k

Not every incident needs every command. The important part is knowing what question each command is answering.

What I would avoid

During an incident, there is often pressure to do something quickly.

Typical reactions are:

kill -9 <pid>
systemctl restart <service>
reboot

Sometimes one of those actions is eventually necessary.

But doing it before collecting evidence can destroy the information that tells us why the server became slow in the first place.

If 30 processes are blocked waiting for the same storage device, killing one of them does not fix the storage device.

And rebooting the server may turn a degraded-storage incident into a boot incident if the required LUNs are not available when the machine comes back.

My preference is:

Observe. Prove. Fix. Verify.

First understand the failure layer. Then make the smallest justified change.

I use this same approach throughout Linux Production Troubleshooting, where I collected 100 production-style Linux scenarios covering boot problems, systemd, CPU, memory, filesystems, LVM, Multipath, networking, DNS, authentication, and performance.

The idea behind the book is the same as this article: not just which command to run, but why you are running it and what the result proves.

If this way of troubleshooting matches how you work — or how you want to approach production incidents — you can find the book here: Linux Production Troubleshooting

RHEL: /data Disappeared After Reboot? Troubleshoot LVM and Multipath Before Touching pvcreate

hosni1982 — Sat, 29 Aug 2026 22:44:42 +0000

A filesystem disappearing after a reboot is one of those incidents where a quick "fix" can make the situation much worse.

Imagine this:

/data is missing
Application cannot start
systemd reports a mount failure
Server may even enter emergency mode

The first reaction might be:

pvcreate
vgcreate
lvcreate
mkfs.xfs

On an existing production LUN, that can be exactly the wrong thing to do.

The problem may not be LVM at all.

Start with the failure layers

In an enterprise RHEL environment, /data can depend on several layers:

/data
  ↓
Filesystem
  ↓
Logical Volume
  ↓
Volume Group
  ↓
Physical Volume
  ↓
Multipath
  ↓
SAN LUN
  ↓
Storage paths

The goal is simple:

Find the first broken layer before changing anything.

1. Is `/data` actually expected to be mounted?

Start with:

findmnt /data
grep -w '/data' /etc/fstab
lsblk -f

Then inspect the boot errors:

journalctl -b -p err..alert --no-pager

You may see something like:

Failed to mount /data
Dependency failed for Local File Systems

That only proves the mount failed.

It does not prove that the filesystem is corrupted.

2. Does the Logical Volume exist?

Check LVM:

lvs -a -o lv_name,vg_name,lv_attr,devices

A healthy configuration might show:

LV       VG       Attr       Devices
lv_data  vg_data  -wi-a----- /dev/mapper/mpatha(0)

If the LV is missing, move down one layer.

Do not recreate it yet.

3. Does the Volume Group exist?

vgs

Expected example:

VG       #PV #LV #SN Attr   VSize    VFree
vg_data    1   1   0 wz--n- 500.00g  20.00g

If vg_data is missing, resist the temptation to run vgcreate.

Check the PV first.

4. Is the Physical Volume visible?

pvs -o pv_name,vg_name,pv_size,pv_free

Expected:

PV                  VG       PSize    PFree
/dev/mapper/mpatha  vg_data  500.00g  20.00g

If /dev/mapper/mpatha is missing, the problem is probably below LVM.

Now the investigation changes direction.

5. Check Multipath

multipath -ll

Example of a healthy map:

mpatha (3600508b400105df70000e00000ac0000) dm-3
size=500G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio=50 status=active
| `- 2:0:0:1 sdb 8:16 active ready running

`-+- policy='service-time 0' prio=10 status=enabled
  `- 3:0:0:1 sdc 8:32 active ready running

You can inspect the individual paths with:

multipathd show paths

and:

lsscsi

If the expected WWID is absent completely, recreating the VG will not fix the root cause.

The SAN LUN may simply not be visible to the server.

A typical failure chain

Suppose your checks give this result:

findmnt /data
    → not mounted

lvs
    → lv_data missing

vgs
    → vg_data missing

pvs
    → expected PV missing

multipath -ll
    → expected WWID missing

Now the picture is very different:

SAN LUN not visible
        ↓
Multipath map missing
        ↓
PV missing
        ↓
VG missing
        ↓
LV missing
        ↓
Filesystem unavailable
        ↓
/data not mounted

The missing filesystem was only the visible symptom.

The real failure occurred much lower in the stack.

Commands I would avoid at this stage

Unless you have proven that the storage is new and intentionally being initialized, avoid commands such as:

pvcreate /dev/mapper/mpatha
vgcreate vg_data /dev/mapper/mpatha
lvcreate ...
mkfs.xfs ...

These are creation commands, not troubleshooting commands.

On an existing production disk, they may overwrite metadata you are trying to recover.

Evidence to collect before escalation

Before calling the storage, SAN, VMware or infrastructure team, collect evidence:

date
hostnamectl
uname -r
lsblk -f
findmnt
pvs
vgs
lvs -a -o +devices
multipath -ll
multipathd show paths
journalctl -b -p err..alert --no-pager

In complex cases, and if permitted by your environment:

sos report

This gives the next support level something concrete to work with.

The troubleshooting habit that matters

When /data disappears, don't immediately ask:

Which command can recreate /data?

Ask:

Which command proves why /data disappeared?

That small change in mindset can prevent a recoverable infrastructure incident from becoming a data-loss incident.

Free Linux troubleshooting resources

I maintain a small public project called Linux Ops Rescue Kit with free RHEL troubleshooting examples and read-only diagnostic tools.

The GitHub repository currently includes:

High load but low CPU troubleshooting
LVM / Multipath missing filesystem troubleshooting
A read-only Linux snapshot script
An evidence-first troubleshooting methodology

GitHub:

https://github.com/hosni1982/linux-ops-rescue-kit

The free release is also available here:

https://github.com/hosni1982/linux-ops-rescue-kit/releases/tag/v1.0-free

For engineers who need a larger on-call runbook, the complete Linux Ops Rescue Kit contains 30 production incident workflows, verification, rollback and Stop & Escalate guidance.

If you work on RHEL production systems, I'd be interested to know:

What is the most dangerous "quick fix" you've seen during a storage incident?

DEV Community: hosni1982

Why High Load Does Not Always Mean High CPU

Why High Load Does Not Always Mean High CPU

Start with the symptom

What load average is actually telling us

D state changes the investigation

Follow the dependency chain

Don't trust one metric

What I would avoid

RHEL: /data Disappeared After Reboot? Troubleshoot LVM and Multipath Before Touching pvcreate

Start with the failure layers

1. Is /data actually expected to be mounted?

2. Does the Logical Volume exist?

3. Does the Volume Group exist?

4. Is the Physical Volume visible?

5. Check Multipath

A typical failure chain

Commands I would avoid at this stage

Evidence to collect before escalation

The troubleshooting habit that matters

Free Linux troubleshooting resources

1. Is `/data` actually expected to be mounted?