DEV Community

Cover image for 🧰 Linux Troubleshooting Guide – Real-World Problems & Solutions 💻🛠️
Satyam Ahirrao
Satyam Ahirrao

Posted on

🧰 Linux Troubleshooting Guide – Real-World Problems & Solutions 💻🛠️

#linux #devops #troubleshoot

📌 Author's Note:

After spending 2 years managing Linux systems in production, I’ve collected a set of real-world issues and battle-tested solutions.

This is not just a command dump — it’s a thinking framework + practical toolkit to troubleshoot smarter and faster.

Let’s dive into 14+ issues ranging from network outages and disk failures to memory leaks and corrupted filesystems.

❌ Issue 1: Server Not Reachable 🌐

🧠 Objective:

Determine why a server can't be reached from the network or another system.

✅ Step-by-Step Guide:

1. 🔍 Check Network Reachability: 

ping server_ip
ping server_hostname
Enter fullscreen mode Exit fullscreen mode
  • 🟢 If both hostname & IP respond:
    • The server is reachable; the issue may lie on the client side or higher-layer services (e.g., SSH, app).

  • ❌ If hostname fails but IP works:

    • Likely a DNS resolution problem.
    • Check these files: 
    cat /etc/hosts
cat /etc/resolv.conf
cat /etc/nsswitch.conf
    • Validate DNS server entries and name resolution order.

  • ❌ If both hostname & IP fail:

    • Check whether the issue is isolated or widespread. 
    ping another_server_in_same_network

2. 🖥️ Access Server via Console (if available):

Check network interface status:

ip addr
nmcli device status
ifconfig
ip link show
Enter fullscreen mode Exit fullscreen mode

Check if default gateway is reachable:

ip route
ping default_gateway
Enter fullscreen mode Exit fullscreen mode

3. 🔐 Check Security Settings:

  • SELinux: 
  getenforce
Enter fullscreen mode Exit fullscreen mode
  • Firewalls: 
  iptables -L
  firewall-cmd --list-all
  ufw status
Enter fullscreen mode Exit fullscreen mode

4. 🔌 Physical Layer (if bare metal):

  • Cable connected?
  • NIC link lights blinking?
  • Hardware errors in dmesg

🌐 Issue 2: Cannot Connect to Website or App

🧠 Objective:

You can reach the server, but the web app or service is unreachable.

✅ Troubleshooting Steps:

1. Ping Server: 

ping app_server_ip
Enter fullscreen mode Exit fullscreen mode

If fails, follow Issue #1.

2. Check Port Reachability: 

telnet server_ip 80       # For HTTP
nc -zv server_ip 443      # For HTTPS
Enter fullscreen mode Exit fullscreen mode

3. If Port is Closed:

  • Is the app/service running? 
  systemctl status nginx
  service apache2 status
Enter fullscreen mode Exit fullscreen mode
  • Restart service: 
  systemctl restart nginx
Enter fullscreen mode Exit fullscreen mode

4. Check App Logs: 

journalctl -u nginx
tail -f /var/log/nginx/error.log
Enter fullscreen mode Exit fullscreen mode

5. Confirm Listening Ports: 

ss -tulnp
netstat -tulnp
Enter fullscreen mode Exit fullscreen mode

6. Check Firewall/SELinux:

As shown in Issue #1

🔐 Issue 3: SSH Fails (Root or User Login)

🧠 Objective:

Fix issues preventing SSH login for root or users.

✅ Checklist:

1. Network Connectivity: 

ping server_ip
Enter fullscreen mode Exit fullscreen mode

If no ping, go to Issue #1.

2. Check SSH Port: 

nc -zv server_ip 22
Enter fullscreen mode Exit fullscreen mode

3. If Port is Open:

  • Check SSH daemon: 
  systemctl status sshd
  systemctl restart sshd
Enter fullscreen mode Exit fullscreen mode
  • Check SSH Config: 
  cat /etc/ssh/sshd_config | grep PermitRootLogin
Enter fullscreen mode Exit fullscreen mode
  • Verify user shell: 
  cat /etc/passwd | grep username
Enter fullscreen mode Exit fullscreen mode

Shell must not be /sbin/nologin or /bin/false.

  • Logs: 
  tail -f /var/log/secure
  tail -f /var/log/auth.log
Enter fullscreen mode Exit fullscreen mode

💽 Issue 4: Disk Full or Add Storage

🧠 Symptoms:

  • Apps crash, logs fail to write, or server feels sluggish.

✅ Steps:

1. Check Usage: 

df -h
Enter fullscreen mode Exit fullscreen mode

2. Find Large Files: 

du -sh /*
du -sh /var/*
Enter fullscreen mode Exit fullscreen mode

3. Cleanup Suggestions:

  • Log rotation: 
  logrotate -f /etc/logrotate.conf
Enter fullscreen mode Exit fullscreen mode
  • Delete unused logs or archives
  • Move files to other mounts

4. Disk Health: 

badblocks -v /dev/sda
Enter fullscreen mode Exit fullscreen mode

5. Monitor I/O: 

iostat -xz 1
iotop
dstat
Enter fullscreen mode Exit fullscreen mode

🧱 Issue 5: Filesystem Corrupted

🧠 Symptoms:

System won’t boot or throws mount errors.

✅ Fix:

  1. Boot with Live CD or ISO into rescue mode.
  2. Mount root FS: 
   chroot /mnt/sysimage
Enter fullscreen mode Exit fullscreen mode
  1. Check Logs: 
   dmesg | grep -i error
   tail /var/log/messages
Enter fullscreen mode Exit fullscreen mode
  1. Run fsck: 
   fsck /dev/sdX1
Enter fullscreen mode Exit fullscreen mode

🧾 Issue 6: /etc/fstab Missing or Invalid

✅ Recovery:

  1. Boot into rescue mode.
  2. Mount root: 
   chroot /mnt/sysimage
Enter fullscreen mode Exit fullscreen mode
  1. View device UUIDs: 
   blkid
Enter fullscreen mode Exit fullscreen mode
  1. Rebuild /etc/fstab: Example: 
   UUID=xxxxx / ext4 defaults 0 1
Enter fullscreen mode Exit fullscreen mode

🚫 Issue 7: Cannot cd to Directory (Even as Root)

🔍 Common Causes:

  • Path doesn't exist: 
  ls -ld /path/to/dir
Enter fullscreen mode Exit fullscreen mode
  • Missing execute bit: 
  chmod +x /dir
Enter fullscreen mode Exit fullscreen mode
  • Ownership issues: 
  chown user:group /dir
Enter fullscreen mode Exit fullscreen mode

🔗 Issue 8: Cannot Create Symlink or Hard Link

🔍 Reasons:

  • Target doesn't exist
  • You’re linking across filesystems (hard links require same device)
  • Permission issues

✅ Examples: 

ln -s /actual/path /shortcut/path
ln /file1 /file2   # Hard link
Enter fullscreen mode Exit fullscreen mode

🧠 Issue 9: Server Running Out of Memory

🧪 Check Usage: 

free -h
top
htop
ps aux --sort=-%mem
Enter fullscreen mode Exit fullscreen mode

🩺 Analyze /proc/meminfo: 

cat /proc/meminfo | grep -i active
Enter fullscreen mode Exit fullscreen mode

✅ Actions:

  • Kill high memory consumers
  • Adjust priority: 
  renice -n 10 -p <pid>
Enter fullscreen mode Exit fullscreen mode
  • Add or extend swap (see next)

💾 Issue 10: Add or Extend Swap Space

➕ Add Swap File: 

dd if=/dev/zero of=/swapfile bs=1G count=4
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
Enter fullscreen mode Exit fullscreen mode

📝 Persistent Config: 

echo '/swapfile none swap sw 0 0' >> /etc/fstab
Enter fullscreen mode Exit fullscreen mode

❓ Issue 11: Can't Run Certain Commands

✅ Check:

  • Is command installed? 
  which command
Enter fullscreen mode Exit fullscreen mode
  • In $PATH? 
  echo $PATH
Enter fullscreen mode Exit fullscreen mode
  • File executable? 
  ls -l /usr/bin/command
  chmod +x
Enter fullscreen mode Exit fullscreen mode
  • Shared libraries: 
  ldd /usr/bin/command
Enter fullscreen mode Exit fullscreen mode

🔁 Issue 12: Unexpected Reboot or Crashes

🧪 Root Causes:

  • Overheating
  • Kernel panic
  • Power/hardware issue
  • Out Of Memory (OOM) killer

✅ Logs: 

journalctl -xe
dmesg | less
uptime
Enter fullscreen mode Exit fullscreen mode

🌐 Issue 13: Server Has No IP Address

✅ Checklist:

  • View interfaces: 
  ip addr
Enter fullscreen mode Exit fullscreen mode
  • NIC status: 
  nmcli device
Enter fullscreen mode Exit fullscreen mode
  • Restart networking: 
  systemctl restart NetworkManager
Enter fullscreen mode Exit fullscreen mode

🗂️ Issue 14: Backup & Restore File Permissions

🧷 Backup Permissions: 

getfacl -R /var/www > www.acl
Enter fullscreen mode Exit fullscreen mode

🔄 Restore: 

setfacl --restore=www.acl
Enter fullscreen mode Exit fullscreen mode

🧠 Pro Tip: Take VM snapshots before major permission changes!

💡 Bonus: Useful Disk Partitioning Tips

  • 🔄 Detect new disk: 
  echo 1 > /sys/block/sdX/device/rescan
Enter fullscreen mode Exit fullscreen mode
  • 📏 Extend LVM: 
  pvcreate /dev/sdX
  vgextend my_vg /dev/sdX
  lvextend -l +100%FREE /dev/my_vg/my_lv
  resize2fs /dev/my_vg/my_lv
Enter fullscreen mode Exit fullscreen mode

🙌 Final Thoughts

🎯 Every Linux problem has a root cause — don’t just reboot, investigate!

Use this guide like a runbook: step-by-step, methodical, and confident.

Keep calm, troubleshoot smart, and script your way to stability. 🧙‍♂️🐧

Top comments (0)