Why High Load Does Not Always Mean High CPU

One of the first things I check when someone says "the Linux server is slow" is the load average.

And one of the most common mistakes is to immediately translate high load into high CPU usage.

These two things are related, but they are not the same.

I have seen production servers with a load average above 30 while the CPUs were mostly idle. In cases like this, adding CPU or killing the process at the top of top would not have solved anything.

The first question should be simpler:

What are the processes actually waiting for?

Start with the symptom

Suppose we connect to a server and get this:

$ uptime
14:32:18 up 47 days, 3:21, 4 users, load average: 31.42, 28.77, 21.06

A load of 31 immediately gets attention.

So we check CPU usage:

$ top
%Cpu(s): 6.2 us, 2.8 sy, 0.0 ni, 84.1 id, 6.7 wa, 0.2 hi

More than 80% of the CPU is idle.

That should change the direction of the investigation.

The machine has a high load, but it is not CPU saturated.

What load average is actually telling us

On Linux, load average is not simply a measure of CPU consumption.

It includes tasks that are runnable, but also tasks stuck in uninterruptible sleep — typically processes waiting for some kernel operation or I/O to complete.

This is why looking only at:

top

or:

ps aux --sort=-%cpu

can send the investigation in the wrong direction.

I usually check the process states next.

$ ps -eo state,pid,ppid,comm,wchan:32 | head -30

Or, for a quick count:

$ ps -eo state= | sort | uniq -c

Imagine we get:

      1 R
    146 S
     27 D

The interesting number here is not R.

It is the 27 processes in D state.

D state changes the investigation

D means uninterruptible sleep.

A process in this state is usually waiting inside the kernel for something to finish. Storage is a common cause, although it is not the only possible one.

At this point, I would not restart the application yet.

I would first identify the affected processes:

$ ps -eo pid,ppid,state,wchan:32,comm | awk '$3=="D"'

Then look for a pattern.

Are they all application processes?

Are they accessing the same filesystem?

Did they all become blocked around the same time?

For example:

PID    PPID S WCHAN                            COMMAND
18421     1 D xfs_file_buffered_aio_write     java
18476     1 D xfs_file_buffered_aio_write     java
18503     1 D wait_on_page_bit_common         java

Now we have something much more useful than "load is high."

The application threads are waiting on filesystem or storage operations.

Follow the dependency chain

This is where production troubleshooting becomes more interesting.

The visible symptom may be:

Application is slow

but the actual chain could be:

application
    ↓
filesystem
    ↓
logical volume
    ↓
multipath device
    ↓
SAN path
    ↓
storage array

If the application is waiting because the storage layer is degraded, restarting the application only removes the symptom temporarily — and sometimes makes the incident worse.

I would continue with basic I/O checks:

iostat -xz 1

and:

vmstat 1

For SAN-backed storage:

multipath -ll

I would also check the kernel messages:

journalctl -k --since "-30 min"

or:

dmesg -T | tail -100

Messages about path failures, I/O timeouts, SCSI errors or device resets can quickly move the investigation toward the real failure layer.

Don't trust one metric

This is the main lesson.

A high load average is an observation. It is not a root cause.

I try to separate these three questions:

What do I see?
        ↓
What does the evidence prove?
        ↓
Which layer is actually failing?

For high load, that means correlating at least:

uptime
top
vmstat 1
ps -eo state,pid,ppid,comm,wchan:32
iostat -xz 1
journalctl -k

Not every incident needs every command. The important part is knowing what question each command is answering.

What I would avoid

During an incident, there is often pressure to do something quickly.

Typical reactions are:

kill -9 <pid>
systemctl restart <service>
reboot

Sometimes one of those actions is eventually necessary.

But doing it before collecting evidence can destroy the information that tells us why the server became slow in the first place.

If 30 processes are blocked waiting for the same storage device, killing one of them does not fix the storage device.

And rebooting the server may turn a degraded-storage incident into a boot incident if the required LUNs are not available when the machine comes back.

My preference is:

Observe. Prove. Fix. Verify.

First understand the failure layer. Then make the smallest justified change.

---

I use this same approach throughout Linux Production Troubleshooting, where I collected 100 production-style Linux scenarios covering boot problems, systemd, CPU, memory, filesystems, LVM, Multipath, networking, DNS, authentication, and performance.

The idea behind the book is the same as this article: not just which command to run, but why you are running it and what the result proves.

If this way of troubleshooting matches how you work — or how you want to approach production incidents — you can find the book here: Linux Production Troubleshooting