DEV Community

Cover image for What top Actually Shows You (And What It Doesnt)
Schiff Heimlich
Schiff Heimlich

Posted on

What top Actually Shows You (And What It Doesnt)

#linux #performance #debugging #sysadmin

You run top or htop. You see CPU at 80%. You think "my app is using 80% CPU." That is not what that number means.

The Breakdown

top shows you four main categories:

  • us (user) — your application code executing
  • sy (system) — kernel work on behalf of your process (syscalls, page faults, scheduler)
  • ni (nice) — low-priority user work
  • id (idle) — CPU doing nothing

There are also wa (iowait), hi/si (hardware/software interrupts), st (steal — time the hypervisor gave to another VM).

The problem: us + sy is not "your application's CPU usage." It is "CPU time accounted to your process context." A process spinning in a futex loop, a process blocked on a slow filesystem, a process constantly context-switching — all show up as CPU time.

What Actually Matters for Performance

Brendan Gregg has been saying this for years: CPU utilization is not a performance metric. It is a capacity metric.

What you actually want to know:

  1. IPC (Instructions Per Cycle) — are you getting work done per cycle? Low IPC = memory-bound, cache misses, branch mispredicts.
  2. Stalled cycles — frontend stalls (instruction fetch), backend stalls (data dependencies, cache misses).
  3. Run-queue latency — how long tasks wait for CPU. This is what pidstat -w or perf sched shows you.
  4. Context switch rate — high voluntary switches = locking/contention. High involuntary = time slice exhaustion.

The Practical Commands 

# Per-process scheduler stats (voluntary/involuntary switches)
pidstat -w 1

# System-wide run-queue latency
perf sched latency

# CPU-wide IPC and stall cycles (requires PMU access)
perf stat -e cycles,instructions,cache-references,cache-misses -a sleep 10

# What top *doesnt* show you: steal time in VMs
# If st is high, your hypervisor is overcommitted — your process isnt slow, it is not running
top -1  # press 1 to see per-CPU breakdown
Enter fullscreen mode Exit fullscreen mode

A Real Example

Go service, CPU at 90% in top. pidstat -w shows 50k voluntary context switches/sec. perf stat shows IPC of 0.3 (terrible). The culprit: a mutex hot path in a logging library. Every log call locked a global mutex. Threads spent cycles spinning, not working.

top said "90% CPU." The reality: "90% CPU, 10% useful work."

The Fix Is Not In top

You do not fix this by staring at top. You fix it by:

  • perf record -g to find the hot path
  • perf report to see the call graph
  • Actually reading the flame graph

top is for "is the box busy?" Not "why is my app slow?"

Schiff Heimlich | Sysadmin who checks perf before blaming the scheduler

Top comments (0)