DEV Community

Opservo
Opservo

Posted on Originally published at getopservo.com

What to Do When a Linux Server Load Average Is High

High load average on your Linux server? Learn how to diagnose the real cause and fix it fast — with practical commands and clear steps.
You glance at your monitoring dashboard or run a quick top and see a load average of 14 on a 4-core server. Something is wrong — but load average alone doesn't tell you what. It's one of those metrics that looks alarming and yet gives you almost no actionable information on its own. Here's how to actually diagnose what's happening and get your server back to normal.

What Load Average Actually Means

Linux load average is a 1-, 5-, and 15-minute rolling average of the number of processes either running or waiting for CPU or I/O. The key word is 'waiting.' A load of 4.0 on a 4-core machine means your cores are fully saturated — but it doesn't tell you whether processes are waiting for the CPU or waiting on a slow disk. Those two problems have completely different fixes.

A quick rule of thumb: divide the load average by your CPU count. Run nproc to get your core count. If load / cores is above 1.0 for a sustained period, something needs attention. If it's above 2.0, something is actively broken.

Step 1 — Figure Out Whether It's CPU or I/O

Run top and immediately press the key 1 to expand per-CPU stats. Look at the %wa (wait) column. If CPU wait is consistently above 10–20%, your bottleneck is I/O, not CPU. If wait is low but %us (user) or %sy (system) is pegged, you have a CPU problem.

For a clearer picture of I/O, run iostat -xz 2 from the sysstat package. Watch the %util column for each disk. A disk sitting at 90–100% util is saturated. The await column shows average request wait time in milliseconds — anything above 20–30ms for a spinning disk or above a few milliseconds for SSD suggests a bottleneck.

  • Install sysstat if needed: sudo apt install sysstat or sudo yum install sysstat
  • Run: iostat -xz 2 5 — this samples every 2 seconds, 5 times
  • Run: vmstat 2 5 — check the b column (blocked processes) and wa (I/O wait)
  • High b count with high wa confirms I/O saturation

Step 2 — Find the Processes Responsible

Once you know whether it's CPU or I/O, find the culprits. For CPU, top sorted by CPU (default) shows you the worst offenders. For I/O, the picture is harder — top doesn't sort by I/O by default.

Use iotop -o to show only processes with active I/O. This is the fastest way to catch a runaway database vacuum, a backup job gone wrong, or a log rotation task chewing through disk. You'll need root: sudo iotop -o.

Another powerful option is pidstat -u -d 2 — it shows both CPU and I/O stats per process in one view, refreshing every 2 seconds. If you spot a PID causing trouble, get more detail with cat /proc/<PID>/status and ls -l /proc/<PID>/fd to see what files it has open.

  • CPU culprits: top (default), or ps aux --sort=-%cpu | head -10
  • I/O culprits: sudo iotop -o or pidstat -d 2
  • Identify what a process is doing: strace -p <PID> -e trace=read,write -c
  • Check open files: lsof -p <PID>

Step 3 — Common Causes and Their Fixes

Once you've identified the process, the fix depends on the pattern. Here are the most common scenarios:

  • Runaway cron job or backup: check crontab -l and /etc/cron.d/. If a backup is hitting disk hard, consider scheduling it during off-peak hours or using ionice -c3 -p <PID> to lower its I/O priority without killing it.
  • Database doing heavy work: could be a missing index causing a full table scan, autovacuum (PostgreSQL), or a slow query. Check database slow query logs first before restarting anything.
  • Log file explosion: a bug can cause an app to write millions of log lines per second. Check with du -sh /var/log/* and watch for files growing in real time using watch -n1 ls -lh /var/log/myapp/.
  • Memory pressure causing swap: run free -h. If swap usage is high and growing, the kernel is paging — which causes I/O wait even when your disk is healthy. The real fix is reducing memory usage or adding RAM, not tuning disk.
  • Fork bomb or process leak: ps aux | wc -l to count processes. If it's in the thousands, you may have a process spawning out of control. Set limits in /etc/security/limits.conf (nproc) to prevent recurrence.

Step 4 — After You Fix It, Prevent It

A one-time spike is forgivable. A recurring spike that takes down production is a monitoring problem. Set up alerts on load average — but also on the underlying signals: CPU utilization, disk I/O wait, and memory usage. Alert on the combination, not just one number in isolation.

If you're running a small team without dedicated infrastructure staff, tools like Opservo can help by continuously watching these signals together and explaining in plain English what's happening — so you're not deciphering iostat output at 2am when something breaks. It can also flag patterns like a cron job that spikes load every hour before your users notice.

High load average is a symptom, not a diagnosis. The commands above — iostat, iotop, vmstat, pidstat — are your actual diagnostic tools. Get comfortable running them the moment you see a spike, and you'll cut your resolution time from hours to minutes.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Top comments (0)