New Relic - CPU usage (%) and Load Average

At AWS re:Invent 2025, I had the opportunity to give a brief demonstration at the New Relic booth that opened the stage to some AWS Heroes. In Linux environments, administrators often rely on CPU usage and load average metrics to determine whether an instance is appropriately sized. An oversized instance that sits idle wastes resources and drives up the cloud bill, whereas an undersized instance pushed to its limits can degrade application performance—ultimately impacting a company’s revenue.

To set the stage for my demo, I began with the Summary tab’s first graph, which displays CPU usage as a percentage. I started a simple CPU-bound task using the yes command, which continuously outputs the character "y". Then, I moved on to a more complex workload: fio with 32 threads performing synchronous, direct I/O disk writes:

fio --rw=write --ioengine=psync --direct=1 --bs=1M --numjobs=32 --name=test --filename=/tmp/x --size=10G --thread

One could expect the yes command to saturate the CPU, and this fio workload to be I/O-bound, but surprisingly, both tasks show 100% CPU usage:

The load average graph doesn't help and shows the 32 threads on the CPU:

We don't have more details on the load average. The Process tab shows yes, but with only low CPU usage, and not fio:

What could help detail the load average is the process state, not gathered here. The R state is real CPU usage, for running tasks, like the 12.5% of yes. The fio processes are mostly doing disk I/O in the D state, which doesn't utilize CPU, despite what is displayed here.

Back to the CPU usage graph, I can get more details. I open the query and instead of cpuPercent I display the details:

SELECT --average(cpuPercent),
 average(cpuUserPercent), average(cpuSystemPercent),  -- running
 average(cpuStealPercent),                            -- hypervisor
 average(cpuIdlePercent), average(cpuIOWaitPercent)   -- idle (incl. wait I/O)
FROM SystemSample WHERE (entityGuid = 'NzM2NzA3MXxJTkZSQXxOQXw0MDAyNTY2MTYyODgwNDkyMzM0') 
TIMESERIES AUTO SINCE 5 minutes ago UNTIL now

The 100% CPU usage was in "Steal" for the running yes command because I've run that on an overprovisioned virtual machine where the hypervisor gives only 1/4th of the CPU cycles, and was in "IO Wait" for fio when it was waiting for IO completion rather than running in CPU:

To explain this "IO Wait" and that it is just like "Idle", I've started yes again while fio was running and the "IO Wait" disappeared:

The reason is that "IO Wait" is accounted when a CPU is idle, because the process that was running waits on an IO call, and no other process had to run on CPU. Then, the process stays scheduled on the CPU with this state. But if another process comes to run in CPU then the "IO Wait" is not accounted anymore. The CPU usage (%) is not the state of the processes, but the state of the processor:

• when a process is running on a processor's threads, it is in R state and counts in cpuUserPercent or cpuSystemPercent depending if it is running in userspace (the application) or kernel (system call). If the hypervisor preempted the CPU cycles, it is reported as "Steal"
  • when a process is on an uninterruptible call and not scheduled out of the processor's thread, it is in D state and counts as "IO Wait"
  • when the process in the processor's thread it is waiting on something else, it is accounted as "Idle"

Back to the load average, the reason why the D state ("IO Wait") is accounted in addition to the R state ("User", "Sys" and "Steal") is visible in the loadavg.c code:

 * The global load average is an exponentially decaying average of nr_running +
 * nr_uninterruptible.
 *
 * Once every LOAD_FREQ:
 *
 *   nr_active = 0;
 *   for_each_possible_cpu(cpu)
 *  nr_active += cpu_of(cpu)->nr_running + cpu_of(cpu)->nr_uninterruptible;
 *
 *   avenrun[n] = avenrun[0] * exp_n + nr_active * (1 - exp_n)

However, the most important comment is the one that explains that it is a silly number:

/*
 * kernel/sched/loadavg.c
 *
 * This file contains the magic bits required to compute the global loadavg
 * figure. Its a silly number but people think its important. We go through
 * great pains to make it work on big machines and tickless kernels.
 */

The load average metric traces back to early Unix systems (1970s), where it reflected how busy a system was by counting processes in the run queue. At that time, computers used periodic scheduling ticks—even while idle—to keep track of time and processes. Sysadmins would see a load average of 1 on a single CPU as one process running or waiting, thus equating it to CPU usage. Today, with multi-core processors, sophisticated scheduling, and tickless kernels, the load average is a far less reliable indicator of real‑time CPU usage and is often misunderstood without considering the real state of the processes.

The uninterruptible state D is not necessarily linked to disk activity. For example, asynchronous disk I/O operations that collect completed IO do not enter the D state. I demonstrated this by changing the fio job from psync to async, observing identical I/O throughput and rate but less "IO Wait" and a lower load average. Furthermore, some system calls can appear as "IO Wait" and increase the load average, even when they are idle and harmless. I also showed this by launching a thousand processes with {vfork();sleep(300);}.

The first step when using the New Relic dashboard for Linux is to replace 'cpuPercent' in the CPU usage charts with more detailed metrics:

• cpuUserPercent and cpuSystemPercent for tasks running in CPU, respectively, for user (application) and system (kernel) code.
• cpuStealPercent, cpuIOWaitPercent, cpuIdlePercent for idle CPU, because the hypervisor didn't allow tasks to run (and steals CPU cycles), or no tasks has something to run (with or without a task waiting on uninterruptible call)

Remember, the load average is not a reliable metric because it does not exclusively reflect actively running processes. It may also include various types of waits, but not all. The 'IO wait' percentage does not indicate high IO activity. Instead, it shows the CPU is idle while many processes are waiting for IO operations. In cloud environments where minimizing costs by reducing idle CPU time is crucial, you should focus on CPU usage for user and system processes, excluding waiting tasks, when sizing an instance. An inexperienced user might misinterpret the 100% usage shown in my fio demo as a sign the instance is too small, while in fact, it's the opposite.