DEV Community

Cover image for Debugging Linux Performance with /proc, PSI, and Cgroups
Anuj Kumar Tiwari
Anuj Kumar Tiwari

Posted on

Debugging Linux Performance with /proc, PSI, and Cgroups

Debugging Linux Performance with /proc, PSI, and Cgroups

When a Linux application becomes slow, the first instinct is often to inspect application code, check CPU usage, or restart the affected process.

But Linux performance problems are not always caused by application-level code.

A container may be experiencing CPU throttling even when the host has available CPU capacity. A process may have low CPU utilization but spend time waiting for I/O. A system may show apparently sufficient memory while applications are experiencing memory pressure.

The important part of performance debugging is understanding which resource is constrained, which process or container is affected, and whether the available measurements actually support the suspected cause.

Here is a practical workflow for investigating these problems.

1. Start by identifying the scope

Before collecting metrics, identify where the problem is occurring.

There are three useful levels to investigate:

  • Host level: Overall CPU, memory, pressure, disk, and network activity.
  • Container level: Resource limits, CPU throttling, and memory constraints.
  • Process level: CPU consumption, memory usage, I/O activity, and scheduling behavior.

These levels are related, but they are not interchangeable.

For example, high memory usage across the host does not automatically mean that one particular process is responsible.

Likewise, low host CPU utilization does not prove that a container has enough CPU capacity. Its cgroup may impose a much lower limit.

Always establish the scope before drawing conclusions.

2. Understand CPU measurements over time

Linux exposes CPU statistics through /proc/stat.

These counters are cumulative, so one isolated reading is not enough to understand current CPU activity.

A useful approach is to collect measurements over a consistent time interval and compare the changes.

Look at:

  • User CPU time
  • System CPU time
  • Idle CPU time
  • I/O wait
  • Steal time
  • Context switches

For a process, CPU time can be inspected through /proc/PID/stat and summarized using tools such as ps.

For example:

ps -p 1234 -o pid,comm,%cpu,%mem,nlwp,state
Enter fullscreen mode Exit fullscreen mode

The important distinction is between a cumulative counter and a rate calculated over an observation window.

A process that accumulated significant CPU time over several hours may currently be idle.

3. Investigate memory pressure, not just memory usage

Looking at free memory alone can be misleading.

Linux uses available memory for caching, and the amount of memory currently in use does not independently establish that the system is under pressure.

Start with:

grep -E 'MemTotal|MemAvailable|SwapTotal|SwapFree' /proc/meminfo
Enter fullscreen mode Exit fullscreen mode

Then inspect memory-related counters:

grep -E 'pgfault|pgmajfault|oom_kill' /proc/vmstat
Enter fullscreen mode Exit fullscreen mode

For additional context, Linux Pressure Stall Information provides visibility into resource stalls.

Check:

cat /proc/pressure/memory
Enter fullscreen mode Exit fullscreen mode

PSI helps identify whether tasks are experiencing delays because memory resources are constrained.

The useful approach is to correlate memory availability, page faults, memory pressure, and OOM-related events instead of relying on one number.

4. Inspect cgroups when containers are involved

Container resource limits introduce another layer of investigation.

A host can have available memory and CPU capacity while a container is constrained by its own limits.

For cgroup v2, inspect the relevant workload's cgroup directory.

Common files include:

cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.events
Enter fullscreen mode Exit fullscreen mode

The workload's actual cgroup directory may be nested, so do not assume the root directory contains the correct values. Its membership can be inspected through /proc/PID/cgroup.

Pay particular attention to:

  • CPU quota and available capacity
  • nr_throttled
  • throttled_usec
  • Memory usage relative to the configured limit
  • OOM and other memory events

Cgroup v1 uses different file layouts and counter names, so collectors need to account for the version in use.

5. Investigate the target process

Once the affected process is identified, inspect its individual resource usage.

For example:

PID=1234

grep -E '^(Name|State|VmRSS|VmSwap|voluntary_ctxt_switches|nonvoluntary_ctxt_switches):' "/proc/$PID/status"
Enter fullscreen mode Exit fullscreen mode

Process-level information can help answer questions such as:

  • Is the process using more CPU than expected?
  • Is resident memory increasing?
  • Is swap usage changing?
  • Are major page faults increasing?
  • Are context switches increasing?
  • Is the process generating substantial I/O?

Where permissions allow, /proc/PID/io and /proc/PID/fd can provide additional information about I/O activity and open file descriptors.

Again, compare values across time where appropriate. Cumulative context-switch and I/O counters are more useful as changes or rates than as isolated totals.

6. Check disk and network activity

Resource bottlenecks are not always CPU or memory problems.

For block-device statistics, Linux exposes:

cat /proc/diskstats
Enter fullscreen mode Exit fullscreen mode

For network connections, inspect socket states and queue information:

ss -tan
Enter fullscreen mode Exit fullscreen mode

Pay attention to:

  • Disk latency and accumulated I/O activity
  • Socket states such as CLOSE_WAIT, TIME_WAIT, and SYN_RECV
  • Non-zero Recv-Q and Send-Q values
  • TCP retransmission trends

A non-zero queue or an individual retransmission counter is not enough to prove a bottleneck. Correlate the observations with application latency and changes over time.

7. Use active profiling only when necessary

Passive telemetry should generally be the first step.

When the evidence indicates that more detailed investigation is needed, tools such as strace and perf can provide additional information.

They are useful for investigating system calls, scheduling, and process-level activity.

However, active profiling can introduce overhead and may require additional permissions or capabilities.

Keep profiling targeted to the affected process, use bounded durations, and understand the impact before enabling it.

8. Separate measured facts from hypotheses

This is one of the most important parts of a useful diagnostic workflow.

Consider these two statements:

Measured observation:

The process reported 30% CPU utilization during the observation interval.

Diagnostic hypothesis:

The process may be experiencing scheduling contention.

The first statement is a measurement. The second requires supporting evidence.

A good diagnostic report should distinguish between:

  • Observed measurements
  • Relevant counter changes
  • Possible causes
  • Confidence in a finding
  • Recommended next diagnostic actions

This prevents an initial suspicion from being reported as a confirmed root cause.

9. A small example from a controlled workload

During a local test, I ran a controlled Python process and collected a process-level snapshot.

The sanitized output looked like this:

{
  "process": "python3",
  "threads": 1,
  "cpu_percent": 30.0,
  "memory_rss_mib": 9.65,
  "swap_mib": 0,
  "context_switches": {
    "voluntary": 71,
    "nonvoluntary": 165
  }
}
Enter fullscreen mode Exit fullscreen mode

These values demonstrate the structure of a process-level diagnostic snapshot. They are from one controlled run, not a benchmark or a performance guarantee.

The next step is not to conclude that the process has a problem. It is to compare the measurements with the expected workload and investigate any relevant changes or pressure signals.

10. A practical debugging checklist

When investigating a Linux performance issue, I would follow this order:

  1. Identify whether the issue is host-wide, container-specific, or process-specific.
  2. Collect system CPU and memory measurements over consistent intervals.
  3. Inspect CPU, memory, and I/O pressure.
  4. Check cgroup limits and throttling when containers are involved.
  5. Inspect target-process CPU, memory, context switches, and I/O.
  6. Correlate disk and network observations with application behavior.
  7. Use active profiling only when passive evidence indicates that it is necessary.
  8. Keep measured facts separate from diagnostic hypotheses.

The main lesson is simple: measure first, correlate second, and diagnose third.

Linux exposes a substantial amount of useful performance information through /proc, /sys, cgroups, and PSI. The challenge is collecting the correct signals and interpreting them in the right context.

A disciplined evidence-first workflow makes Linux performance investigations more reproducible and helps avoid unnecessary changes based on assumptions.

Top comments (0)