Server Suddenly Stopped! The Cause Was "Log Bloat" – Prevention Measures You Can Take Right Now

📝 What You’ll Learn in This Article

✅ Why did the server suddenly return 502 Bad Gateway?

✅ The risk of "log bloat" that can happen to any server

✅ Immediate actions you can take! How to manage and prevent log issues

---

One Day, the Server Suddenly Went Down…

One day, our production server started returning 502 Bad Gateway errors.

"It broke even though we didn’t do anything…?" – That terrifying experience happened again.

The Cause Was "Log Bloat"!

The server’s journal logs filled up the disk, reaching 100% usage.

As a result, the application went into an endless restart loop, rendering the server inoperable.

---

Immediate Actions: Log Management Checks & Countermeasures

Step 1: Check Disk Usage

• Is your disk running out of space?
  

  df -h
  

  • If the disk usage of / is close to 100%, it’s a red flag!
  • Which logs are taking up the most space?

  sudo du -sh /var/log/* | sort -h -r | head -10
  

  • journal or httpd logs are likely culprits!

Step 2: Log Countermeasures (Journal Logs)

1. Edit the configuration file
   

   sudo vi /etc/systemd/journal.conf
   

• Set a maximum log usage limit like SystemMaxUse=50MB
• If it’s commented out, uncomment it

1. Restart the journal daemon to apply changes
   

   sudo systemctl status systemd-journald
   

• If it says active (running), you’re good!

1. Delete already bloated logs
   

   sudo journalctl --vacuum-size=500M
   

• This will delete journal logs exceeding 500MB

1. Check if the size has been reduced
   

   journalctl --disk-usage
   

---

Step 3: Log Countermeasures (httpd Logs)

Apache (httpd) logs also tend to bloat over time.

To manage them effectively, configure logrotate based on Red Hat’s official documentation.

(Check Red Hat's guide)

(Sorry for keeping this section brief)

---

Summary: Neglecting Log Management Can Lead to Major Failures!

• Logs continuously grow → If ignored, they can cause server crashes
• Make it a habit to check disk usage (df -h / du -sh /var/log/*)
• Set log limits and enable automatic deletion (SystemMaxUse, logrotate)