DEV Community

Mo Rizal
Mo Rizal

Posted on

Why Is Your Linux Server Slow? A Practical Guide to Performance Troubleshooting

A Linux server can be running without any obvious errors and still feel painfully slow.

Applications may take longer to respond, SSH sessions may become sluggish, users may suddenly experience high latency. The difficult part is that the root cause is not always obvious.

A slow Linux server does not necessarily mean that the CPU is overloaded.

It could be memory pressure, excessive disk I/O, network congestion, too many open file descriptors, or simply a process consuming resources unexpectedly.

The challenge for a DevOps Engineer is not just finding that the server is slow, but identifying why it is slow.

Instead of immediately restarting services or killing processes, we need to investigate the system, layer by layer:

  • Is the CPU under pressure?

  • Is the system running out of memory?

  • Is disk I/O becoming a bottleneck?

  • Are processes exhausting file descriptors?

  • Which process is actually responsible for the problem?

This article is a practical introduction to Linux performance troubleshooting using real reproducible scenarios.

Rather than looking at isolated commands, we will build a troubleshooting mindset: observe the symptom, collect evidence, identify the bottleneck, apply the appropriate solution, and verify the result.

To make the investigation reproducible, I created a companion lab repository containing scenarios that intentionally introduce problems:

github repository

High CPU

High CPU usage is one of the most common Linux performance problems.

In this scenario, we intentionally create a process that continuously executes an infinite loop.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./high-cpu/reproduce.sh
./high-cpu/reproduce.sh
Enter fullscreen mode Exit fullscreen mode

The script will display a warning before starting the scenario.

While it is running, open another terminal to investigate the system.

Identify High CPU Usage

The first step is to confirm whether the system is actually experiencing high CPU utilization.

Use top:

top
Enter fullscreen mode Exit fullscreen mode

Look at the CPU summary:

%Cpu(s)
Enter fullscreen mode Exit fullscreen mode

Example result:

%Cpu(s): 95.2 us,  1.8 sy,  0.0 ni,  0.0 id,  2.5 wa, ...
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • us — CPU time spent running user-space processes

  • sy — CPU time spent running kernel-space processes

  • id — percentage of CPU time that is idle

  • wa — CPU time waiting for I/O

If id is consistently very low while us is high, the CPU is heavily utilized by processes.

Identify the Process

The next step is to find the process that consume the most CPU.

Use:

ps aux --sort=-%cpu | head
Enter fullscreen mode Exit fullscreen mode

Example result:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       38690 90.5  0.0   7740  3512 pts/4   R+   02:39   0:16 bash ./reproduce.sh
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • PID — Process ID

  • %CPU — CPU consumption

  • %MEM — Memory consumption

  • STAT — Current process state

  • COMMAND — Command used to start the process

Here bash ./reproduce.sh process is clearly consuming a significant amount of CPU.

We now have a candidate process, but before terminating it, we should inspect it further.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 38690
Enter fullscreen mode Exit fullscreen mode

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       38690   37782 96 02:39 pts/4    00:00:45 bash ./reproduce.sh
Enter fullscreen mode Exit fullscreen mode

This provides additional information about the process.

The most important fields are:

  • PID — Process ID

  • PPID — Parent Process ID

  • STIME — Process start time

  • TIME — Total CPU time consumed

  • CMD — Command used to start the process

Understand the Cause

Inspect the process script:

cat reproduce.sh
Enter fullscreen mode Exit fullscreen mode

The important part is:

while true; do
    :
done
Enter fullscreen mode Exit fullscreen mode

This creates an infinite loop.

The loop continuously executes without any sleep, blocking operation, or other mechanism that would allow the process to stop consuming CPU therefore keeps the CPU busy.

This is the root cause of the high CPU usage in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 38690
Enter fullscreen mode Exit fullscreen mode

Then check the CPU usage again:

top
Enter fullscreen mode Exit fullscreen mode

The process should no longer appear in the process list, and CPU utilization should return to its previous level.

Disk I/O

High disk I/O is another common Linux performance problem.

In this scenario, we intentionally create a process that continuously writes data to disk.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./disk/reproduce.sh
./disk/reproduce.sh
Enter fullscreen mode Exit fullscreen mode

Identify Disk I/O

The first step is to confirm whether the system is actually experiencing high Disk I/O activity.

Use iostat:

iostat -xz 1
Enter fullscreen mode Exit fullscreen mode

Example result:

Device            r/s     w/s   rkB/s   wkB/s  %util
sda              0.00  125.00    0.00  51200.00  98.50
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • r/s — Number of read requests per second

  • w/s — Number of write requests per second

  • rkB/s — Data read per second

  • wkB/s — Data written per second

  • %util — Percentage of time the device was busy

If %util is consistently close to 100%, the storage device is heavily utilized.

Identify the Process

The next step is to find which process is generating the disk activity.

Use:

pidstat -d 1
Enter fullscreen mode Exit fullscreen mode

Example result:

UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
0     41043      0.00 259548.51      0.00       0  bash
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • PID — Process ID

  • COMMAND — Command generating the I/O

Here the PID 41043 process is clearly generating significant disk write activity.

Inspect the Process

Use the PID identified from the previous step:

ps -fp 41043
Enter fullscreen mode Exit fullscreen mode

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       41043   37782  0 03:03 pts/4    00:00:00 bash ./reproduce.sh
Enter fullscreen mode Exit fullscreen mode

This provides additional information about the process.

The most important fields are:

  • PID — Process ID

  • PPID — Parent Process ID

  • STIME — Process start time

  • TIME — Total CPU time consumed

  • CMD — Command used to start the process

The command shows that the process is continuously writing data to testfile.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh
Enter fullscreen mode Exit fullscreen mode

The important part is:

while true; do
    dd if=/dev/zero of=testfile bs=1M count=1024 conv=fsync
    rm -f testfile
done
Enter fullscreen mode Exit fullscreen mode

The script continuously creates a file and writes data to it.

The dd command reads zero-filled data from /dev/zero and writes it to testfile.

The conv=fsync option forces dd to flush the written data to the storage device before completing the operation. This makes the scenario generate actual disk write activity instead of relying entirely on the filesystem page cache.

The file is then removed and the process repeats the operation.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 41043
Enter fullscreen mode Exit fullscreen mode

Then check the disk activity again:

iostat -xz 1
Enter fullscreen mode Exit fullscreen mode

You can also verify the processes performing I/O:

iotop
Enter fullscreen mode Exit fullscreen mode

The dd process should no longer appear, disk write throughput should drop significantly, and %util should return to its previous level.

Finally, verify that the test file has been removed:

ls -lh testfile
Enter fullscreen mode Exit fullscreen mode

Memory Pressure

High memory usage is another common Linux performance problem.

In this scenario, we create a process that allocates a large amount of memory and keeps it allocated.

The reproduction script is available in the github repository repository.

Run the reproduction script with a memory target:

chmod +x ./memory-pressure/reproduce.sh
./memory-pressure/reproduce.sh 2G
Enter fullscreen mode Exit fullscreen mode

Replace 2G with the amount of memory you want to allocate.

Identify Memory Pressure

The first step is to confirm whether the system is actually experiencing high memory utilization.

Use:

free -h
Enter fullscreen mode Exit fullscreen mode

Example result:

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       3.1Gi       141Mi       1.0Mi       822Mi       692Mi
Swap:             0B          0B          0B
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • total — Total physical memory available

  • used — Memory currently used by the system

  • free — Completely unused memory

  • buff/cache — Memory used by the kernel for buffers and filesystem cache

  • available — Memory available for new applications

The available value is particularly important when diagnosing memory pressure. If available memory is consistently very low while used memory is high, the system is under significant memory pressure.

If the system has swap enabled, also pay attention to swap usage. Significant swap usage can indicate that the system does not have enough physical memory to satisfy current workloads.

Identify the Process

The next step is to find which process is consuming the most memory.

Use:

ps aux --sort=-%mem | head
Enter fullscreen mode Exit fullscreen mode

Example result:

USER         PID %CPU %MEM    VSZ    RSS TTY      STAT START   TIME COMMAND
root       39937  0.1 52.7 2123808 2116080 pts/4 S+   02:56   0:03 python3 ./memory_hog.py 2G
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • PID — Process ID

  • %MEM — Percentage of physical memory used by the process

  • VSZ — Virtual memory size

  • RSS — Resident memory currently held in RAM

  • STAT — Current process state

  • COMMAND — Command used to start the process

Here the PID 39937 process is clearly consuming a significant amount of memory.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 39937
Enter fullscreen mode Exit fullscreen mode

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       39937   39910  0 02:56 pts/4    00:00:03 python3 ./memory_hog.py 2G
Enter fullscreen mode Exit fullscreen mode

This provides additional information about the process.

The most important fields are:

  • PID — Process ID

  • PPID — Parent Process ID

  • STIME — Process start time

  • TIME — Total CPU time consumed

  • CMD — Command used to start the process

The command shows that the process is running memory_hog.py with a target allocation of 2G.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh
Enter fullscreen mode Exit fullscreen mode

The shell script starts the Python memory allocation program:

python3 "$(dirname "$0")/memory_hog.py" "$MEMORY"
Enter fullscreen mode Exit fullscreen mode

The Python script can then be inspected:

cat memory_hog.py
Enter fullscreen mode Exit fullscreen mode

The important part is:

while allocated_bytes < amount:
    remaining = amount - allocated_bytes
    size = min(chunk_size, remaining)

    chunk = bytearray(size)

    chunk[0] = 1

    allocated.append(chunk)
    allocated_bytes += size
Enter fullscreen mode Exit fullscreen mode

The script allocates memory in 1 MiB chunks until the requested memory amount is reached.

Each allocated chunk is stored in the allocated list:

allocated.append(chunk)
Enter fullscreen mode Exit fullscreen mode

This keeps references to the allocated memory, preventing Python from releasing those objects.

The script then remains running:

while True:
    time.sleep(1)
Enter fullscreen mode Exit fullscreen mode

This large memory allocation is the root cause of the memory pressure in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 39937
Enter fullscreen mode Exit fullscreen mode

Then check the memory usage again:

free -h
Enter fullscreen mode Exit fullscreen mode

You can also verify the processes sorted by memory usage:

ps aux --sort=-%mem | head
Enter fullscreen mode Exit fullscreen mode

The memory_hog.py process should no longer appear in the process list.

The available memory should increase, and overall memory utilization should return closer to its previous level.

File Descriptor Exhaustion

In this scenario, we intentionally create a process that continuously opens file descriptors without closing them.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./file-descriptor/reproduce.sh
./file-descriptor/reproduce.sh
Enter fullscreen mode Exit fullscreen mode

Identify File Descriptor Pressure

The first step is to check the system-wide file descriptor usage.

Use:

cat /proc/sys/fs/file-nr
Enter fullscreen mode Exit fullscreen mode

Example result:

2656    0    9223372036854775807
Enter fullscreen mode Exit fullscreen mode

The values represent:

  • allocated — Number of allocated file handles

  • unused — Number of unused allocated file handles

  • maximum — Maximum number of file handles allowed system-wide

The allocated value represents file handles currently allocated by the kernel.

A high number of allocated file handles can indicate file descriptor pressure, especially when applications are approaching their per-process or system-wide limits.

Identify the Process

The next step is to find the process running fd_exhaustion.py.

Use:

ps aux | grep '[f]d_exhaustion'
Enter fullscreen mode Exit fullscreen mode

Example result:

root       42297  0.2  0.2  18488 10504 pts/4    S+   03:11   0:00 python3 ./fd_exhaustion.py
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • PID — Process ID

  • %CPU — CPU usage

  • %MEM — Memory usage

  • COMMAND — Command used to start the process

Here the python3 ./fd_exhaustion.py process is the candidate responsible for the file descriptor exhaustion.

Use the identified PID for the next step.

Inspect the Process

The next step is to determine how many file descriptors the process currently has open.

Use:

ls /proc/42297/fd | wc -l
Enter fullscreen mode Exit fullscreen mode

Example result:

1024
Enter fullscreen mode Exit fullscreen mode

This indicates that the process currently has 1024 file descriptors open.

You can inspect the actual descriptors with:

ls -l /proc/<PID>/fd
Enter fullscreen mode Exit fullscreen mode

Example:

lr-x------ 1 root root 64 Aug 21 03:11 0 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 1 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 2 -> /dev/pts/4
lr-x------ 1 root root 64 Aug 21 03:11 3 -> /dev/null
lr-x------ 1 root root 64 Aug 21 03:11 4 -> /dev/null
Enter fullscreen mode Exit fullscreen mode

The large number of /dev/null entries indicates that the process is continuously opening new file descriptors.

Next, check the process limits:

cat /proc/42297/limits | grep "open files"
Enter fullscreen mode Exit fullscreen mode

Output:

Max open files            1024                 1048576                files
Enter fullscreen mode Exit fullscreen mode

The first value is the soft limit, while the second value is the hard limit.

The soft limit is the limit currently enforced for the process.

Once the process reaches this limit, attempts to open additional file descriptors will fail.

Understand the Cause

Inspect the reproduction script:

cat fd_exhaustion.py
Enter fullscreen mode Exit fullscreen mode

The important part is:

while True:
    try:
        fd = os.open(
            "/dev/null",
            os.O_RDONLY
        )

        FILES.append(fd)

        counter += 1
Enter fullscreen mode Exit fullscreen mode

The script continuously opens /dev/null using os.open().

Every returned file descriptor is stored in the FILES list:

FILES.append(fd)
Enter fullscreen mode Exit fullscreen mode

The descriptors are intentionally not closed while the scenario is running. As a result, the number of open file descriptors continuously increases

Eventually, os.open() fails because the process has reached its file descriptor limit.

The script then reports the error:

Failed to open new file descriptor
[Errno 24] Too many open files
Enter fullscreen mode Exit fullscreen mode

This is the root cause of the file descriptor exhaustion in this scenario.

File descriptor exhaustion can have broader consequences for real applications. Processes may fail to open files, accept new connections, create sockets, or perform other operations that require file descriptors.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 42297
Enter fullscreen mode Exit fullscreen mode

Verify that the process has stopped:

ps -p 42297
Enter fullscreen mode Exit fullscreen mode

If the process has stopped, the command should return no process entry.

You can also verify that the process no longer exists:

test -d /proc/<PID> && echo "Process still exists" || echo "Process stopped"
Enter fullscreen mode Exit fullscreen mode

The file descriptors belonging to the process are automatically released by the kernel when the process terminates.

Therefore, /proc/<PID>/fd will no longer be available after the process has exited.

Finally, verify the system-wide file descriptor state again:

cat /proc/sys/fs/file-nr
Enter fullscreen mode Exit fullscreen mode

The allocated file handle count should return closer to its previous level after the exhausted process has been terminated.

Network Connection Exhaustion

In this scenario, we intentionally create a process that continuously opens TCP connections to a local server and keeps those connections open.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./network-connection/reproduce.sh
./network-connection/reproduce.sh
Enter fullscreen mode Exit fullscreen mode

The script will display a warning before starting the scenario.

The reproduction uses 127.0.0.1:9000 as the target, so all connections remain on the local machine.

Identify Abnormal Connections

The first step is to confirm whether the system is experiencing an unusually large number of TCP connections.

Use ss:

ss -s
Enter fullscreen mode Exit fullscreen mode

Example result:

Total: 1252
TCP:   2051 (estab 1023, closed 1021, orphaned 1, timewait 1020)

Transport Total     IP        IPv6
RAW       1         0         1
UDP       3         3         0
TCP       1030      1028      2
INET      1034      1031      3
FRAG      0         0         0
Enter fullscreen mode Exit fullscreen mode

The most important fields are:

  • estab — Established TCP connections

  • orphaned — Orphaned TCP connections

  • timewait — Connections waiting to be fully closed

  • TCP — Total TCP sockets

A significant increase in established connections can indicate abnormal connection usage.

In this scenario, the large number of estab connections is the main indicator that something is continuously creating and maintaining TCP connections.

Identify the Connection and Process

The next step is to identify which process owns the connections.

Use:

ss -tanp
Enter fullscreen mode Exit fullscreen mode

Example result:

CLOSE-WAIT  1  0  127.0.0.1:46374  127.0.0.1:9000  users:(("python3",pid=46938,fd=880))
CLOSE-WAIT  1  0  127.0.0.1:39120  127.0.0.1:9000  users:(("python3",pid=46938,fd=86))
CLOSE-WAIT  1  0  127.0.0.1:44192  127.0.0.1:9000  users:(("python3",pid=46938,fd=634))
Enter fullscreen mode Exit fullscreen mode

The most important information is:

  • State — Current TCP connection state

  • Local Address:Port — Local endpoint

  • Peer Address:Port — Remote endpoint

  • users — Process owning the socket

  • pid — Process ID

  • fd — File descriptor associated with the socket

Here we can identify the process responsible for the connections:

users:(("python3",pid=46938,fd=880))
Enter fullscreen mode Exit fullscreen mode

The PID is:

46938
Enter fullscreen mode Exit fullscreen mode

The same PID appearing across many connections is a strong indication that one process is responsible for creating the connection buildup.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 46938
Enter fullscreen mode Exit fullscreen mode

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       46938   46922  0 03:42 pts/1    00:00:00 python3 ./connection_exhaustion.py 127.0.0.1 9000
Enter fullscreen mode Exit fullscreen mode

This provides additional information about the process.

The most important fields are:

  • PID — Process ID

  • PPID — Parent Process ID

  • STIME — Process start time

  • TIME — Total CPU time consumed

  • CMD — Command used to start the process

The command shows that the process is running connection_exhaustion.py and connecting to 127.0.0.1:9000.

Next, check how many file descriptors the process currently has open:

ls /proc/46938/fd | wc -l
Enter fullscreen mode Exit fullscreen mode

A large number of file descriptors is expected because every TCP socket consumes a file descriptor.

You can inspect the descriptors directly:

ls -l /proc/46938/fd | head
Enter fullscreen mode Exit fullscreen mode

You should see entries pointing to sockets, for example:

lrwx------ 1 root root 64 Aug 21 03:42 86 -> 'socket:[123456]'
lrwx------ 1 root root 64 Aug 21 03:42 87 -> 'socket:[123457]'
lrwx------ 1 root root 64 Aug 21 03:42 88 -> 'socket:[123458]'
Enter fullscreen mode Exit fullscreen mode

This confirms that the process is holding a large number of socket file descriptors.

Understand the Cause

Inspect the reproduction script:

cat connection_exhaustion.py
Enter fullscreen mode Exit fullscreen mode

The important part is:

while True:
    try:
        sock = socket.socket(
            socket.AF_INET,
            socket.SOCK_STREAM
        )

        sock.connect((host, port))

        CONNECTIONS.append(sock)

        counter += 1
Enter fullscreen mode Exit fullscreen mode

The script continuously creates TCP sockets and connects them to the target server.

Every successful connection is stored in the CONNECTIONS list:

CONNECTIONS.append(sock)
Enter fullscreen mode Exit fullscreen mode

The sockets are intentionally not closed while the scenario is running. As a result, the number of active connections continuously increases:

The TCP server is also intentionally designed to keep accepted connections open:

connection, address = server.accept()
connections.append(connection)
Enter fullscreen mode Exit fullscreen mode

This means the server does not immediately close the connections created by the client. Eventually, the client process can reach its file descriptor limit and fail to create additional sockets.

This pattern can occur in real applications when connections are not properly closed, connection pools are misconfigured, or an application continuously creates new connections instead of reusing existing ones.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 46938
Enter fullscreen mode Exit fullscreen mode

Then check the network connection summary again:

ss -s
Enter fullscreen mode Exit fullscreen mode

You can also check the connections to port 9000 directly:

ss -tan | grep ':9000'
Enter fullscreen mode Exit fullscreen mode

The number of established TCP connections should return to its previous level.

Conclusion

Linux performance troubleshooting is not about finding a single command that tells you why a server is slow.

It is about building a structured investigation from the symptoms you observe.

In the scenarios covered in this article, we investigated several different types of Linux performance problems:

  • High CPU caused by an infinite loop

  • High disk I/O caused by continuous writes

  • Memory pressure caused by excessive memory allocation

  • File descriptor exhaustion caused by descriptors that are never closed

  • Network connection exhaustion caused by continuously opened TCP connections

Although the symptoms are different, the troubleshooting approach remains consistent:

The important lesson is to avoid making assumptions based on a single metric.

High CPU does not immediately tell you which process is responsible. High memory usage does not necessarily mean the system is running out of usable memory. A large number of TCP connections does not automatically mean the network is slow.

Each symptom needs to be investigated with the right evidence.

When a production server becomes slow, the most valuable skill is not knowing how to restart it.

It is being able to answer:

What is happening, why is it happening, and how can I prove that my fix actually worked?

The scenarios in this article are intentionally simplified, but the troubleshooting methodology can be applied to much more complex production incidents.

You can reproduce all of these scenarios and investigate yourself by cloning my github repository bellow:

https://github.com/muhammadyulasfipahrizal/linux-performance-lab.git

Top comments (0)