Troubleshooting High CPU and Memory Usage on Linux

Are you experiencing sluggish performance on your Linux server, with processes eating up all your CPU and RAM? High CPU and memory usage can cripple your applications and leave users frustrated. This article will equip you with practical tools and techniques to diagnose and resolve these common Linux performance bottlenecks.

Understanding the Culprits: CPU and Memory

Before we dive into troubleshooting, let's clarify what CPU and memory are in the context of a server.

CPU (Central Processing Unit) is the brain of your computer. It executes instructions from programs. When your CPU usage is high, it means your processor is working very hard, which can slow down all operations.

Memory (RAM - Random Access Memory) is your server's short-term workspace. Programs load their data into RAM for quick access by the CPU. If your server runs out of RAM, it starts using a slower storage space called swap, significantly impacting performance.

Initial Assessment: Where's the Bottleneck?

The first step in troubleshooting is to get a general overview of your system's resource consumption. Several command-line tools can provide this information.

top and htop: Your Real-Time Performance Dashboards

The top command is a classic utility that displays a dynamic, real-time view of running processes. It shows CPU usage, memory usage, and other vital system information.

top

When you run top, you'll see a list of processes sorted by CPU usage by default. Look for processes consistently at the top with high %CPU and %MEM (percentage of memory used) values.

For a more user-friendly and visually appealing experience, htop is highly recommended. If you don't have it installed, you can typically install it using your distribution's package manager (e.g., sudo apt install htop on Debian/Ubuntu, sudo yum install htop on CentOS/RHEL).

htop

htop offers color-coded displays, easier process navigation, and the ability to interact with processes (like killing them) directly from the interface. Pay attention to the CPU and memory bars at the top, as well as individual process usage.

Analogy: Think of top and htop as the dashboard of your car. They show you how much fuel (CPU) and engine power (Memory) is being used, and which components are consuming the most.

free: A Snapshot of Memory Usage

While top and htop show memory usage per process, the free command gives you an overall picture of your system's RAM and swap usage.

free -h

The -h flag makes the output human-readable, showing values in gigabytes (G) or megabytes (M). You'll see total, used, free, shared, buff/cache (memory used for buffers and caching, which is generally good) and available (memory that can be immediately used by new applications). A low available or consistently high swap usage indicates a memory shortage.

Deep Dive: Identifying Resource-Hungry Processes

Once you have an idea of which resources are strained, you need to pinpoint the exact processes responsible.

ps: Process Status in Detail

The ps command shows information about currently running processes. Combined with other flags, it can be very powerful.

To list all processes and sort them by CPU usage:

ps aux --sort=-%cpu | head -n 10

• a: Show processes for all users.
• u: Display user-oriented format.
• x: Show processes not attached to a terminal.
• --sort=-%cpu: Sort by CPU usage in descending order.
• head -n 10: Display only the top 10.

Similarly, to sort by memory usage:

ps aux --sort=-%mem | head -n 10

This will give you a static list, unlike the dynamic output of top or htop.

iotop: Monitoring Disk I/O

Sometimes, high CPU or memory usage isn't directly caused by a process's computation but by excessive disk input/output (I/O). If your server is constantly reading from or writing to the disk, it can overwhelm the system. iotop helps you identify processes causing high disk activity.

You might need to install it: sudo apt install iotop or sudo yum install iotop.

sudo iotop

Look for processes with high IO> values. This indicates how much of the disk bandwidth they are consuming.

vmstat: Virtual Memory Statistics

vmstat reports on processes, memory, paging, block IO, traps, and CPU activity. It's particularly useful for observing trends over time.

vmstat 5

This command will output statistics every 5 seconds. Key columns to watch are:

• r: The number of runnable processes (waiting for CPU time). A consistently high number here indicates CPU contention.
• b: Processes in uninterruptible sleep (often waiting for I/O).
• swpd: Amount of virtual memory used.
• free: Amount of idle memory.
• buff: Memory used as buffers.
• cache: Memory used as cache.
• si / so: Swap in / Swap out. High values here mean the system is heavily using swap, indicating a RAM shortage.
• bi / bo: Blocks received from / sent to the block device (disk). High values can suggest I/O bottlenecks.
• us / sy / id: User CPU time, system CPU time, and idle CPU time. If us or sy are consistently high and id is low, your CPU is busy.

Common Culprits and How to Address Them

Now that we know how to identify problematic processes, let's look at some common scenarios.

Greedy Applications

Some applications, especially databases, web servers, or custom applications, can sometimes enter a state where they consume excessive resources.

Scenario: You notice a specific web application process (e.g., apache2, nginx, php-fpm, or a Node.js application) is hogging CPU and memory.

Troubleshooting Steps:

1. Check Application Logs: Application-specific logs are your best friend. Look for error messages, repeated requests, or unusual patterns.
2. Review Configuration: Ensure your application's configuration is optimized. For web servers, this might involve tuning worker processes, connection limits, or caching. For databases, it could be buffer pool sizes or query optimization.
3. Update and Patch: Ensure your application and its dependencies are up to date. Bugs in older versions can lead to performance issues.
4. Profile the Application: For custom applications, you might need to use profiling tools specific to the programming language (e.g., pprof for Go, cProfile for Python) to find inefficient code sections.

You might find that your current hosting plan is simply insufficient for the application's demands. In such cases, upgrading your server resources is necessary. Providers like PowerVPS and Immers Cloud offer scalable solutions that can accommodate growing resource needs.

Background Services and Daemons

Many services run in the background on a Linux system, designed to perform specific tasks. Sometimes, these can misbehave.

Scenario: A system service like mysqld, redis, cron, or even a less common daemon is consuming significant resources.

Troubleshooting Steps:

1. Check Service Status: Use systemctl status <service_name> (for systemd-based systems) to see if the service is running correctly and check its logs.
2. Restart the Service: A simple restart can sometimes resolve temporary glitches: sudo systemctl restart <service_name>.
3. Examine Service Configuration: Review the configuration files for the service. Misconfigurations can lead to inefficient operation.
4. Disable Temporarily: If you suspect a specific service is the sole cause, you can try disabling it temporarily to see if performance improves: sudo systemctl stop <service_name> and sudo systemctl disable <service_name> (to prevent it from starting on boot).

System Processes and Kernel Issues

While less common, sometimes core system processes or even the Linux kernel itself can be the source of high resource usage.

Scenario: You see processes like systemd, kworker, or kernel threads consuming a lot of CPU.

Troubleshooting Steps:

1. Kernel Messages: Check kernel logs for errors: dmesg.
2. Systemd Journal: For systemd systems, check the journal: journalctl -xe.
3. Hardware Issues: High kernel-level CPU usage can sometimes indicate underlying hardware problems, such as a faulty disk or network card.
4. Driver Problems: Outdated or buggy hardware drivers can also cause kernel-level issues.

If you suspect a hardware issue, it might be time to consider a new server. Resources like the Server Rental Guide can help you compare different providers and configurations.

Memory Leaks

A memory leak occurs when a program allocates memory but fails to release it when it's no longer needed. Over time, this can consume all available RAM, forcing the system into heavy swap usage.

Scenario: You observe that free memory consistently decreases over time, and swap usage increases, even though no single process appears to be using an exorbitant amount of RAM at any given moment.

Troubleshooting Steps:

1. Monitor Over Time: Use vmstat or sar (System Activity Reporter) to track memory usage trends over hours or days.
2. Application-Specific Tools: Many programming languages and frameworks have tools to detect memory leaks within your application code.
3. Restarting is a Band-Aid: While restarting the offending application or service will temporarily free up memory, it doesn't fix the underlying leak. You need to identify and fix the code responsible.
4. Consider System-Wide Monitoring: Tools like Prometheus with node_exporter can help you track memory usage trends historically, making it easier to spot gradual increases.

Proactive Measures: Preventing Future Issues

Troubleshooting is reactive; prevention is proactive. Here are some ways to avoid hitting these performance walls:

Regular Monitoring

Implement a robust monitoring system. Tools like Prometheus, Grafana, Nagios, or Zabbix can alert you to rising CPU and memory usage before it becomes a critical problem. This allows you to investigate and address issues during off-peak hours.

Resource Limits

For containerized environments (Docker, Kubernetes) or even within your applications, setting resource limits (CPU and memory) can prevent a single runaway process from affecting the entire system.

Performance Testing

Before deploying applications to production, conduct performance and load testing. This helps you understand how your application behaves under stress and identify potential bottlenecks early.

Infrastructure Scaling

As your application grows, so will its resource demands. Regularly review your server's performance metrics and be prepared to scale up your resources or optimize your application. Reliable hosting providers like PowerVPS offer easy scaling options.

Code Optimization

Encourage best practices in code development, focusing on efficient algorithms, proper memory management, and avoiding unnecessary computations.

Conclusion

Diagnosing high CPU and memory usage on Linux involves a systematic approach, moving from broad system overviews to pinpointing specific processes. Tools like top, htop, free, ps, iotop, and vmstat are your primary arsenal. By understanding their output and common causes like greedy applications, misconfigured services, or memory leaks, you can effectively restore your server's performance. Remember that proactive monitoring and optimization are key to maintaining a healthy and responsive system.

---

Disclosure: This article contains affiliate links to PowerVPS and Immers Cloud. If you click through and make a purchase, I may receive a commission at no extra cost to you.