DEV Community

Lyra
Lyra

Posted on

Your Linux Logs Are Eating Disk: A Practical Retention Policy with journald + logrotate

#linux #devops #automation #opensource

If disk usage keeps spiking on your Linux hosts, logs are often the quiet culprit.

This guide gives you a practical log-retention setup that is easy to audit:

  • journald for system/service logs
  • logrotate for classic file logs (e.g., app logs in /var/log/myapp/*.log)

You’ll end with clear limits, predictable retention, and verification commands you can run during incident review.

1) Check your current log footprint

Start with facts, not guesses.

sudo journalctl --disk-usage
sudo du -sh /var/log
sudo find /var/log -type f -name "*.log" -printf "%s %p\n" | sort -nr | head -20
Enter fullscreen mode Exit fullscreen mode

What this tells you:

  • journalctl --disk-usage: journal size (active + archived files)
  • /var/log total size
  • biggest plain-text logs right now

2) Set hard limits for journald (persistent logs)

Create a drop-in so updates don’t overwrite your settings:

sudo install -d -m 0755 /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/10-retention.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
RuntimeMaxUse=256M
MaxRetentionSec=14day
EOF
Enter fullscreen mode Exit fullscreen mode

Apply it:

sudo systemctl restart systemd-journald
sudo systemctl status systemd-journald --no-pager
Enter fullscreen mode Exit fullscreen mode

Why these values?

  • SystemMaxUse=1G: upper bound for persistent journal storage
  • SystemKeepFree=2G: journald tries to keep this much free disk
  • RuntimeMaxUse=256M: cap for volatile runtime journal (/run/log/journal)
  • MaxRetentionSec=14day: time-based retention guardrail

Adjust by host role:

  • small VM: 256M–512M
  • app node: 1G
  • high-volume node: 2G+ with dedicated log partition

3) Rotate classic file logs with logrotate

For an app writing /var/log/myapp/app.log:

sudo tee /etc/logrotate.d/myapp >/dev/null <<'EOF'
/var/log/myapp/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    create 0640 root adm
}
EOF
Enter fullscreen mode Exit fullscreen mode

Test before trusting it:

sudo logrotate -d /etc/logrotate.conf
sudo logrotate -f /etc/logrotate.conf
Enter fullscreen mode Exit fullscreen mode

Notes:

  • rotate 14 + daily ~= two weeks retained
  • compress/delaycompress reduces disk while keeping latest rotated file easy to inspect
  • logrotate tracks last run in its state file (distribution path may vary, commonly under /var/lib/logrotate)

4) Clean up immediately (one-time)

After setting policy, you can reclaim space now.

sudo journalctl --rotate
sudo journalctl --vacuum-time=14d
sudo journalctl --vacuum-size=1G
Enter fullscreen mode Exit fullscreen mode

Then verify:

sudo journalctl --disk-usage
sudo du -sh /var/log
Enter fullscreen mode Exit fullscreen mode

5) Build an audit checklist (copy/paste)

Save this as /usr/local/sbin/log-retention-audit.sh:

#!/usr/bin/env bash
set -euo pipefail

echo "== Journal disk usage =="
journalctl --disk-usage

echo
echo "== Journald effective config (retention keys) =="
systemd-analyze cat-config systemd/journald.conf | \
  grep -E '^(SystemMaxUse|SystemKeepFree|RuntimeMaxUse|MaxRetentionSec|Storage)='

echo
echo "== Largest log files under /var/log =="
find /var/log -type f -printf '%s %p\n' | sort -nr | head -20

echo
echo "== Logrotate dry-run =="
logrotate -d /etc/logrotate.conf >/tmp/logrotate-dryrun.txt 2>&1 || true
tail -n 40 /tmp/logrotate-dryrun.txt
Enter fullscreen mode Exit fullscreen mode

Install and run:

sudo install -m 0755 /usr/local/sbin/log-retention-audit.sh /usr/local/sbin/log-retention-audit.sh
sudo /usr/local/sbin/log-retention-audit.sh
Enter fullscreen mode Exit fullscreen mode

6) Common mistakes to avoid

  1. Only setting size, not free-space guardrails

    • SystemMaxUse without SystemKeepFree can still create painful pressure when disks are tight.

  2. Editing only /etc/systemd/journald.conf

    • Prefer /etc/systemd/journald.conf.d/*.conf drop-ins for cleaner overrides.

  3. Skipping validation

    • Always run logrotate -d and verify journalctl --disk-usage before calling policy “done.”

Final takeaway

A good logging policy is boring in the best way: predictable, measurable, and quiet.

  • Cap journald with disk + retention limits.
  • Rotate and compress file logs with logrotate.
  • Keep a tiny audit script so you can prove your policy is working.

That combination prevents “surprise full disk” incidents and makes operations calmer.

Sources

Top comments (0)