<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Anuj Kumar Tiwari</title>
    <description>The latest articles on DEV Community by Anuj Kumar Tiwari (@anujtiwaristudent).</description>
    <link>https://dev.to/anujtiwaristudent</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254108%2F10ec94b2-1ed3-4fc3-a378-5160023240a1.jpg</url>
      <title>DEV Community: Anuj Kumar Tiwari</title>
      <link>https://dev.to/anujtiwaristudent</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/anujtiwaristudent"/>
    <language>en</language>
    <item>
      <title>Debugging Linux Performance with /proc, PSI, and Cgroups</title>
      <dc:creator>Anuj Kumar Tiwari</dc:creator>
      <pubDate>Thu, 19 Nov 2026 15:39:49 +0000</pubDate>
      <link>https://dev.to/anujtiwaristudent/debugging-linux-performance-with-proc-psi-and-cgroups-2bfi</link>
      <guid>https://dev.to/anujtiwaristudent/debugging-linux-performance-with-proc-psi-and-cgroups-2bfi</guid>
      <description>&lt;h1&gt;
  
  
  Debugging Linux Performance with &lt;code&gt;/proc&lt;/code&gt;, PSI, and Cgroups
&lt;/h1&gt;

&lt;p&gt;When a Linux application becomes slow, the first instinct is often to inspect application code, check CPU usage, or restart the affected process.&lt;/p&gt;

&lt;p&gt;But Linux performance problems are not always caused by application-level code.&lt;/p&gt;

&lt;p&gt;A container may be experiencing CPU throttling even when the host has available CPU capacity. A process may have low CPU utilization but spend time waiting for I/O. A system may show apparently sufficient memory while applications are experiencing memory pressure.&lt;/p&gt;

&lt;p&gt;The important part of performance debugging is understanding which resource is constrained, which process or container is affected, and whether the available measurements actually support the suspected cause.&lt;/p&gt;

&lt;p&gt;Here is a practical workflow for investigating these problems.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Start by identifying the scope
&lt;/h2&gt;

&lt;p&gt;Before collecting metrics, identify where the problem is occurring.&lt;/p&gt;

&lt;p&gt;There are three useful levels to investigate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Host level:&lt;/strong&gt; Overall CPU, memory, pressure, disk, and network activity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Container level:&lt;/strong&gt; Resource limits, CPU throttling, and memory constraints.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Process level:&lt;/strong&gt; CPU consumption, memory usage, I/O activity, and scheduling behavior.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These levels are related, but they are not interchangeable.&lt;/p&gt;

&lt;p&gt;For example, high memory usage across the host does not automatically mean that one particular process is responsible.&lt;/p&gt;

&lt;p&gt;Likewise, low host CPU utilization does not prove that a container has enough CPU capacity. Its cgroup may impose a much lower limit.&lt;/p&gt;

&lt;p&gt;Always establish the scope before drawing conclusions.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Understand CPU measurements over time
&lt;/h2&gt;

&lt;p&gt;Linux exposes CPU statistics through &lt;code&gt;/proc/stat&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;These counters are cumulative, so one isolated reading is not enough to understand current CPU activity.&lt;/p&gt;

&lt;p&gt;A useful approach is to collect measurements over a consistent time interval and compare the changes.&lt;/p&gt;

&lt;p&gt;Look at:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;User CPU time&lt;/li&gt;
&lt;li&gt;System CPU time&lt;/li&gt;
&lt;li&gt;Idle CPU time&lt;/li&gt;
&lt;li&gt;I/O wait&lt;/li&gt;
&lt;li&gt;Steal time&lt;/li&gt;
&lt;li&gt;Context switches&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a process, CPU time can be inspected through &lt;code&gt;/proc/PID/stat&lt;/code&gt; and summarized using tools such as &lt;code&gt;ps&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ps -p 1234 -o pid,comm,%cpu,%mem,nlwp,state
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important distinction is between a cumulative counter and a rate calculated over an observation window.&lt;/p&gt;

&lt;p&gt;A process that accumulated significant CPU time over several hours may currently be idle.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Investigate memory pressure, not just memory usage
&lt;/h2&gt;

&lt;p&gt;Looking at free memory alone can be misleading.&lt;/p&gt;

&lt;p&gt;Linux uses available memory for caching, and the amount of memory currently in use does not independently establish that the system is under pressure.&lt;/p&gt;

&lt;p&gt;Start with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -E &lt;span class="s1"&gt;'MemTotal|MemAvailable|SwapTotal|SwapFree'&lt;/span&gt; /proc/meminfo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then inspect memory-related counters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -E &lt;span class="s1"&gt;'pgfault|pgmajfault|oom_kill'&lt;/span&gt; /proc/vmstat
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For additional context, Linux Pressure Stall Information provides visibility into resource stalls.&lt;/p&gt;

&lt;p&gt;Check:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /proc/pressure/memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;PSI helps identify whether tasks are experiencing delays because memory resources are constrained.&lt;/p&gt;

&lt;p&gt;The useful approach is to correlate memory availability, page faults, memory pressure, and OOM-related events instead of relying on one number.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Inspect cgroups when containers are involved
&lt;/h2&gt;

&lt;p&gt;Container resource limits introduce another layer of investigation.&lt;/p&gt;

&lt;p&gt;A host can have available memory and CPU capacity while a container is constrained by its own limits.&lt;/p&gt;

&lt;p&gt;For cgroup v2, inspect the relevant workload's cgroup directory.&lt;/p&gt;

&lt;p&gt;Common files include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.events
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The workload's actual cgroup directory may be nested, so do not assume the root directory contains the correct values. Its membership can be inspected through &lt;code&gt;/proc/PID/cgroup&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Pay particular attention to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU quota and available capacity&lt;/li&gt;
&lt;li&gt;&lt;code&gt;nr_throttled&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;throttled_usec&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Memory usage relative to the configured limit&lt;/li&gt;
&lt;li&gt;OOM and other memory events&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cgroup v1 uses different file layouts and counter names, so collectors need to account for the version in use.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Investigate the target process
&lt;/h2&gt;

&lt;p&gt;Once the affected process is identified, inspect its individual resource usage.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;PID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1234

grep -E &lt;span class="s1"&gt;'^(Name|State|VmRSS|VmSwap|voluntary_ctxt_switches|nonvoluntary_ctxt_switches):'&lt;/span&gt; &lt;span class="s2"&gt;"/proc/$PID/status"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Process-level information can help answer questions such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is the process using more CPU than expected?&lt;/li&gt;
&lt;li&gt;Is resident memory increasing?&lt;/li&gt;
&lt;li&gt;Is swap usage changing?&lt;/li&gt;
&lt;li&gt;Are major page faults increasing?&lt;/li&gt;
&lt;li&gt;Are context switches increasing?&lt;/li&gt;
&lt;li&gt;Is the process generating substantial I/O?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Where permissions allow, &lt;code&gt;/proc/PID/io&lt;/code&gt; and &lt;code&gt;/proc/PID/fd&lt;/code&gt; can provide additional information about I/O activity and open file descriptors.&lt;/p&gt;

&lt;p&gt;Again, compare values across time where appropriate. Cumulative context-switch and I/O counters are more useful as changes or rates than as isolated totals.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Check disk and network activity
&lt;/h2&gt;

&lt;p&gt;Resource bottlenecks are not always CPU or memory problems.&lt;/p&gt;

&lt;p&gt;For block-device statistics, Linux exposes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /proc/diskstats
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For network connections, inspect socket states and queue information:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ss -tan
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pay attention to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Disk latency and accumulated I/O activity&lt;/li&gt;
&lt;li&gt;Socket states such as &lt;code&gt;CLOSE_WAIT&lt;/code&gt;, &lt;code&gt;TIME_WAIT&lt;/code&gt;, and &lt;code&gt;SYN_RECV&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Non-zero &lt;code&gt;Recv-Q&lt;/code&gt; and &lt;code&gt;Send-Q&lt;/code&gt; values&lt;/li&gt;
&lt;li&gt;TCP retransmission trends&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A non-zero queue or an individual retransmission counter is not enough to prove a bottleneck. Correlate the observations with application latency and changes over time.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Use active profiling only when necessary
&lt;/h2&gt;

&lt;p&gt;Passive telemetry should generally be the first step.&lt;/p&gt;

&lt;p&gt;When the evidence indicates that more detailed investigation is needed, tools such as &lt;code&gt;strace&lt;/code&gt; and &lt;code&gt;perf&lt;/code&gt; can provide additional information.&lt;/p&gt;

&lt;p&gt;They are useful for investigating system calls, scheduling, and process-level activity.&lt;/p&gt;

&lt;p&gt;However, active profiling can introduce overhead and may require additional permissions or capabilities.&lt;/p&gt;

&lt;p&gt;Keep profiling targeted to the affected process, use bounded durations, and understand the impact before enabling it.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Separate measured facts from hypotheses
&lt;/h2&gt;

&lt;p&gt;This is one of the most important parts of a useful diagnostic workflow.&lt;/p&gt;

&lt;p&gt;Consider these two statements:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Measured observation:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The process reported 30% CPU utilization during the observation interval.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Diagnostic hypothesis:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The process may be experiencing scheduling contention.&lt;/p&gt;

&lt;p&gt;The first statement is a measurement. The second requires supporting evidence.&lt;/p&gt;

&lt;p&gt;A good diagnostic report should distinguish between:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Observed measurements&lt;/li&gt;
&lt;li&gt;Relevant counter changes&lt;/li&gt;
&lt;li&gt;Possible causes&lt;/li&gt;
&lt;li&gt;Confidence in a finding&lt;/li&gt;
&lt;li&gt;Recommended next diagnostic actions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This prevents an initial suspicion from being reported as a confirmed root cause.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. A small example from a controlled workload
&lt;/h2&gt;

&lt;p&gt;During a local test, I ran a controlled Python process and collected a process-level snapshot.&lt;/p&gt;

&lt;p&gt;The sanitized output looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"process"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"python3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"threads"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"cpu_percent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;30.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"memory_rss_mib"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;9.65&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"swap_mib"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"context_switches"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"voluntary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;71&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"nonvoluntary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;165&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These values demonstrate the structure of a process-level diagnostic snapshot. They are from one controlled run, not a benchmark or a performance guarantee.&lt;/p&gt;

&lt;p&gt;The next step is not to conclude that the process has a problem. It is to compare the measurements with the expected workload and investigate any relevant changes or pressure signals.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. A practical debugging checklist
&lt;/h2&gt;

&lt;p&gt;When investigating a Linux performance issue, I would follow this order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Identify whether the issue is host-wide, container-specific, or process-specific.&lt;/li&gt;
&lt;li&gt;Collect system CPU and memory measurements over consistent intervals.&lt;/li&gt;
&lt;li&gt;Inspect CPU, memory, and I/O pressure.&lt;/li&gt;
&lt;li&gt;Check cgroup limits and throttling when containers are involved.&lt;/li&gt;
&lt;li&gt;Inspect target-process CPU, memory, context switches, and I/O.&lt;/li&gt;
&lt;li&gt;Correlate disk and network observations with application behavior.&lt;/li&gt;
&lt;li&gt;Use active profiling only when passive evidence indicates that it is necessary.&lt;/li&gt;
&lt;li&gt;Keep measured facts separate from diagnostic hypotheses.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The main lesson is simple: &lt;strong&gt;measure first, correlate second, and diagnose third.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Linux exposes a substantial amount of useful performance information through &lt;code&gt;/proc&lt;/code&gt;, &lt;code&gt;/sys&lt;/code&gt;, cgroups, and PSI. The challenge is collecting the correct signals and interpreting them in the right context.&lt;/p&gt;

&lt;p&gt;A disciplined evidence-first workflow makes Linux performance investigations more reproducible and helps avoid unnecessary changes based on assumptions.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>obsevability</category>
      <category>devops</category>
      <category>performance</category>
    </item>
  </channel>
</rss>
