DEV Community

Cover image for System Monitoring and Performance Tuning in Linux -DevOps Prerequisite 5
Aaditya Kediyal
Aaditya Kediyal

Posted on

System Monitoring and Performance Tuning in Linux -DevOps Prerequisite 5

#linux #devops #systems #performance

System Monitoring and Performance Tuning in Linux

System monitoring and performance tuning are essential tasks for ensuring that your Linux environment runs efficiently and effectively. This article will cover a range of tools and techniques for monitoring system performance and tuning various aspects of a Linux system. We will delve into CPU, memory, disk I/O, and network monitoring, as well as provide strategies for optimizing system performance.

Table of Contents

  1. Introduction to System Monitoring and Performance Tuning
  2. Monitoring Tools
    • top
    • htop
    • vmstat
    • iostat
    • dstat
    • netstat and ss
    • iftop
  3. CPU Monitoring and Tuning
    • Monitoring CPU Usage
    • Tuning CPU Performance
  4. Memory Monitoring and Tuning
    • Monitoring Memory Usage
    • Tuning Memory Performance
  5. Disk I/O Monitoring and Tuning
    • Monitoring Disk I/O
    • Tuning Disk Performance
  6. Network Monitoring and Tuning
    • Monitoring Network Traffic
    • Tuning Network Performance
  7. Best Practices for System Monitoring and Performance Tuning
  8. Conclusion

1. Introduction to System Monitoring and Performance Tuning

System monitoring involves continuously checking various system metrics to ensure that your system is running smoothly. Performance tuning involves making adjustments to system parameters and configurations to improve performance. Effective system monitoring and performance tuning can help prevent bottlenecks, reduce downtime, and ensure optimal resource utilization.

2. Monitoring Tools

Linux offers a variety of tools for monitoring system performance. Here are some of the most commonly used tools:

top

The top command provides a dynamic real-time view of the system's processes, showing CPU and memory usage.

top
Enter fullscreen mode Exit fullscreen mode

htop

htop is an enhanced version of top, providing a more user-friendly interface and additional features such as mouse support and visual indicators.

sudo apt install htop
htop
Enter fullscreen mode Exit fullscreen mode

vmstat

vmstat reports information about processes, memory, paging, block I/O, traps, and CPU activity.

vmstat 2
Enter fullscreen mode Exit fullscreen mode

This command updates every 2 seconds.

iostat

iostat provides statistics on CPU and I/O usage.

sudo apt install sysstat
iostat -xz 2
Enter fullscreen mode Exit fullscreen mode

This command shows extended statistics (-x) with device utilization (-z) every 2 seconds.

dstat

dstat combines the functionality of vmstat, iostat, netstat, and ifstat.

sudo apt install dstat
dstat
Enter fullscreen mode Exit fullscreen mode

netstat and ss

netstat and ss are used for network statistics.

netstat -tuln
ss -tuln
Enter fullscreen mode Exit fullscreen mode

iftop

iftop displays bandwidth usage on an interface by host.

sudo apt install iftop
sudo iftop -i eth0
Enter fullscreen mode Exit fullscreen mode

3. CPU Monitoring and Tuning

Monitoring CPU Usage

Use top, htop, vmstat, and iostat to monitor CPU usage.

top -d 1
Enter fullscreen mode Exit fullscreen mode

The -d option sets the delay between updates to 1 second.

Tuning CPU Performance

  • Adjust CPU Scheduling: Use chrt to set real-time scheduling policies. 
  sudo chrt -f -p 99 $(pgrep your_process)
Enter fullscreen mode Exit fullscreen mode
  • Set CPU Affinity: Use taskset to bind processes to specific CPUs. 
  sudo taskset -c 0,1 your_process
Enter fullscreen mode Exit fullscreen mode
  • Enable/Disable Hyper-Threading: Modify the BIOS/UEFI settings to enable or disable hyper-threading.

4. Memory Monitoring and Tuning

Monitoring Memory Usage

Use free, vmstat, and htop to monitor memory usage.

free -h
Enter fullscreen mode Exit fullscreen mode

The -h option displays the output in human-readable format.

Tuning Memory Performance

  • Adjust Swappiness: The swappiness parameter controls the tendency of the kernel to move processes out of physical memory and onto the swap disk. 
  sudo sysctl vm.swappiness=10
Enter fullscreen mode Exit fullscreen mode
  • Cache Pressure: The vfs_cache_pressure parameter controls the tendency of the kernel to reclaim memory used for caching. 
  sudo sysctl vm.vfs_cache_pressure=50
Enter fullscreen mode Exit fullscreen mode
  • Use HugePages: HugePages can improve performance for applications with large memory requirements. 
  sudo sysctl vm.nr_hugepages=128
Enter fullscreen mode Exit fullscreen mode

5. Disk I/O Monitoring and Tuning

Monitoring Disk I/O

Use iostat, dstat, and iotop to monitor disk I/O.

sudo iotop -o
Enter fullscreen mode Exit fullscreen mode

The -o option shows only processes or threads actually doing I/O.

Tuning Disk Performance

  • Use the Correct I/O Scheduler: The I/O scheduler can be changed using sysfs. 
  echo noop | sudo tee /sys/block/sda/queue/scheduler
Enter fullscreen mode Exit fullscreen mode
  • Enable Write Caching: Write caching can improve performance but at the risk of data loss in case of power failure. 
  sudo hdparm -W1 /dev/sda
Enter fullscreen mode Exit fullscreen mode
  • Tune Filesystem Parameters: Mount options such as noatime can reduce I/O operations. 
  sudo mount -o remount,noatime /dev/sda1
Enter fullscreen mode Exit fullscreen mode

6. Network Monitoring and Tuning

Monitoring Network Traffic

Use iftop, netstat, ss, and nload to monitor network traffic.

sudo apt install nload
sudo nload
Enter fullscreen mode Exit fullscreen mode

Tuning Network Performance

  • Adjust TCP Settings: Tune various TCP parameters using sysctl. 
  sudo sysctl -w net.ipv4.tcp_fin_timeout=30
  sudo sysctl -w net.ipv4.tcp_window_scaling=1
Enter fullscreen mode Exit fullscreen mode
  • Use NIC Offloading: Enable or disable NIC offloading features such as TCP segmentation offload (TSO). 
  sudo ethtool -K eth0 tso on
Enter fullscreen mode Exit fullscreen mode
  • Optimize Network Buffers: Increase the size of network buffers to handle more data. 
  sudo sysctl -w net.core.rmem_max=16777216
  sudo sysctl -w net.core.wmem_max=16777216
Enter fullscreen mode Exit fullscreen mode

7. Best Practices for System Monitoring and Performance Tuning

  • Regular Monitoring: Set up regular monitoring to catch issues before they become critical.
  • Automate Tasks: Use tools like cron and monitoring software to automate regular checks and alerts.
  • Document Changes: Keep a log of all tuning changes to understand their impact.
  • Start Small: Make small, incremental changes and monitor their effects before making further adjustments.
  • Balance Performance and Stability: Ensure that performance improvements do not compromise system stability.

8. Conclusion

System monitoring and performance tuning are critical skills for maintaining a healthy and efficient Linux environment. By using the right tools and techniques, you can monitor CPU, memory, disk I/O, and network performance, and make informed adjustments to optimize your system. Regular monitoring and tuning not only improve performance but also help in proactive problem detection and resolution.

Mastering these skills will ensure that your Linux systems run smoothly, providing a reliable platform for your applications and services. Whether you're managing a single server or a fleet of machines, effective system monitoring and performance tuning are indispensable for any Linux administrator.

Code Snippets Recap 

# top command
top

# htop command
sudo apt install htop
htop

# vmstat command
vmstat 2

# iostat command
sudo apt install sysstat
iostat -xz 2

# dstat command
sudo apt install dstat
dstat

# netstat and ss commands
netstat -tuln
ss -tuln

# iftop command
sudo apt install iftop
sudo iftop -i eth0

# Adjust CPU Scheduling
sudo chrt -f -p 99 $(pgrep your_process)

# Set CPU Affinity
sudo taskset -c 0,1 your_process

# Monitoring Memory Usage
free -h

# Adjust Swappiness
sudo sysctl vm.swappiness=10

# Cache Pressure
sudo sysctl vm.vfs_cache_pressure=50

# Use HugePages
sudo sysctl vm.nr_hugepages=128

# Monitoring Disk I/O
sudo iotop -o

# Use the Correct I/O Scheduler
echo noop | sudo tee /sys/block/sda/queue/scheduler

# Enable Write Caching
sudo hdparm -W1 /dev/sda

# Tune Filesystem Parameters
sudo mount -o remount,noatime /dev/sda1

# Monitoring Network Traffic
sudo apt install nload
sudo nload

# Adjust TCP Settings
sudo sysctl -w net.ipv4.tcp_fin_timeout=30
sudo sysctl -w net.ipv4.tcp_window_scaling=1

# Use NIC Offloading
sudo ethtool -K eth0 tso on

# Optimize Network Buffers
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=167
Enter fullscreen mode Exit fullscreen mode

By implementing these practices and utilizing these tools, you can maintain a robust and high-performing Linux environment. Happy monitoring and tuning!

Top comments (0)