DEV Community

Russell Jones
Russell Jones

Posted on • Originally published at jonesrussell.github.io

Automated Patching and Server Maintenance

#linux #security

Ahnii!

This is part 8 of the Production Linux series. Previous: Secrets, Certificates, and Credential Rotation.

Security patches mean nothing if they're not applied. This post covers automated patching, detecting stale services, and the maintenance tasks that keep your server from slowly filling its disk.

Unattended Upgrades

Install the package:

apt install unattended-upgrades
Enter fullscreen mode Exit fullscreen mode

This installs the daemon that applies package updates on a schedule without manual intervention.

Edit /etc/apt/apt.conf.d/50unattended-upgrades to control what gets upgraded:

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
    // "${distro_id}:${distro_codename}-updates";
};

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";

Unattended-Upgrade::Mail "you@example.com";
Enter fullscreen mode Exit fullscreen mode

The Allowed-Origins block limits upgrades to the security pocket only. The commented-out -updates line is intentional — applying all updates on production carries more risk than applying security fixes alone. The automatic reboot window at 04:00 ensures the server restarts after kernel patches during low-traffic hours.

Verify your configuration without making changes:

unattended-upgrades --dry-run --debug
Enter fullscreen mode Exit fullscreen mode

This prints the packages that would be upgraded and any configuration errors, so you can confirm your settings before relying on them.

Needrestart: Detect Stale Services

needrestart detects services that are still running against old shared library versions after an upgrade — a common source of "I updated the package but the vulnerability is still exploitable" situations.

apt install needrestart
Enter fullscreen mode Exit fullscreen mode

Configure it to restart services automatically rather than prompting. Edit /etc/needrestart/needrestart.conf:

$nrconf{restart} = 'a';
Enter fullscreen mode Exit fullscreen mode

The 'a' mode restarts services automatically. The default 'i' mode is interactive, which blocks unattended upgrade runs. Setting 'a' ensures services pick up updated libraries without manual intervention.

Run needrestart in batch mode to check current status:

needrestart -b
Enter fullscreen mode Exit fullscreen mode

This outputs a machine-readable list of services that need restarting, suitable for log review or scripting.

Log Rotation

Without rotation, application logs accumulate until they fill the disk. logrotate handles this automatically.

Create /etc/logrotate.d/caddy:

/var/log/caddy/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    sharedscripts
    postrotate
        systemctl reload caddy > /dev/null 2>&1 || true
    endscript
}
Enter fullscreen mode Exit fullscreen mode

rotate 14 keeps 14 days of logs. compress gzip-compresses rotated files; delaycompress skips compression on the most recent rotated file so Caddy can finish writing to it before gzip runs. The postrotate script signals Caddy to reopen its log file handles.

Test the configuration without rotating anything:

logrotate --debug /etc/logrotate.d/caddy
Enter fullscreen mode Exit fullscreen mode

The debug output shows what logrotate would do, including which files it would rotate and compress.

Docker Cleanup

Unused Docker images, stopped containers, and dangling build cache accumulate quickly on a busy server.

Remove stopped containers, unused networks, and dangling images:

docker system prune -f
Enter fullscreen mode Exit fullscreen mode

Remove images that haven't been used in 30 days:

docker image prune -a --filter "until=720h"
Enter fullscreen mode Exit fullscreen mode

The 720h filter targets images older than 30 days. This is conservative enough to avoid removing images you're actively using across deployments, while reclaiming space from old build artifacts.

Automate cleanup with a weekly cron job. Add to /etc/cron.weekly/docker-cleanup or via crontab:

0 3 * * 0 root docker system prune -f && docker image prune -a --filter "until=720h" -f
Enter fullscreen mode Exit fullscreen mode

This runs at 03:00 every Sunday. Check current Docker disk usage at any time:

docker system df
Enter fullscreen mode Exit fullscreen mode

The output breaks down space used by images, containers, volumes, and build cache.

Disk Monitoring

Check current partition usage:

df -h
Enter fullscreen mode Exit fullscreen mode

For automated alerting, a simple script handles the common case:

#!/bin/bash
THRESHOLD=80
df -h | awk 'NR>1 {gsub(/%/,"",$5); if ($5 > '"$THRESHOLD"') print "WARN: "$6" at "$5"%"}'
Enter fullscreen mode Exit fullscreen mode

Save this to /usr/local/bin/check-disk and make it executable. Run it from cron or your monitoring system to get warnings before a full disk causes an outage.

Post 9 covers monitoring and alerting in depth — including structured log shipping and uptime checks.

Baamaapii

Top comments (0)