DEV Community

Cover image for Stop Scattered Logs: Centralize Linux Journald with systemd-journal-remote
Lyra
Lyra

Posted on

Stop Scattered Logs: Centralize Linux Journald with systemd-journal-remote

#linux #systemd #opensource #devops

Stop Scattered Logs: Centralize Linux Journald with systemd-journal-remote

If you manage more than a couple of Linux servers, you have probably experienced the pain of SSHing into each one to chase a single event across journalctl outputs. Local logs are great for a single machine, but they become a liability the moment you need correlation, retention, or audit trails across a fleet.

systemd ships with two small, purpose-built tools that solve this without pulling in a full ELK stack: systemd-journal-upload on clients and systemd-journal-remote on the receiver. The beauty is that they preserve the structured binary journal format, support TLS, and handle reconnection state automatically.

Why native journal remote beats most alternatives

  • No parsing or loss of structured fields (_SYSTEMD_UNIT, PRIORITY, MESSAGE, custom fields, etc.)
  • Resumable uploads via a simple state file
  • Socket-activated receiver (low idle resource use)
  • Works over plain HTTP for trusted networks or HTTPS with certificate auth
  • Querying on the server feels exactly like local journalctl

This is not a replacement for long-term search or alerting platforms, but it is the simplest way to get centralized, queryable logs with almost zero operational overhead.

Server setup (the receiver)

On your central log host (Debian 12 / Ubuntu 24.04 or newer):

sudo apt update
sudo apt install systemd-journal-remote
Enter fullscreen mode Exit fullscreen mode

Create the storage directory and set ownership:

sudo mkdir -p /var/log/journal/remote
sudo chown systemd-journal-remote:systemd-journal-remote /var/log/journal/remote
Enter fullscreen mode Exit fullscreen mode

Enable the socket (recommended over the service for on-demand behavior):

sudo systemctl enable --now systemd-journal-remote.socket
Enter fullscreen mode Exit fullscreen mode

For production, add a drop-in to control listening mode and output path:

sudo systemctl edit systemd-journal-remote.service
Enter fullscreen mode Exit fullscreen mode

Add:

[Service]
ExecStart=
ExecStart=/lib/systemd/systemd-journal-remote \
  --listen-http=-3 \
  --output=/var/log/journal/remote
Enter fullscreen mode Exit fullscreen mode

If you want HTTPS (strongly recommended across untrusted networks), switch to --listen-https and supply certificate paths owned by the systemd-journal-remote user.

Firewall example (UFW):

sudo ufw allow 19532/tcp comment "systemd-journal-remote"
Enter fullscreen mode Exit fullscreen mode

Received journals appear as remote-<hostname>.journal files. Query them with:

journalctl --directory=/var/log/journal/remote --since "1 hour ago"
journalctl --file=/var/log/journal/remote/remote-web-01.journal -u nginx
Enter fullscreen mode Exit fullscreen mode

Client setup (the uploader)

On every machine you want to forward logs:

sudo apt install systemd-journal-remote
Enter fullscreen mode Exit fullscreen mode

Edit the upload configuration:

sudo systemctl edit systemd-journal-upload.service --full --force
Enter fullscreen mode Exit fullscreen mode

Or simply edit /etc/systemd/journal-upload.conf:

[Upload]
URL=https://logs.example.com:19532
# For TLS client certificate auth (recommended)
# ServerKeyFile=/etc/ssl/journal/client.key
# ServerCertificateFile=/etc/ssl/journal/client.crt
# TrustedCertificateFile=/etc/ssl/journal/ca.crt
Enter fullscreen mode Exit fullscreen mode

Enable and start:

sudo systemctl enable --now systemd-journal-upload.service
Enter fullscreen mode Exit fullscreen mode

The service maintains /var/lib/systemd/journal-upload/state. If you ever need a full resend, delete that file.

TLS best practices

  • Use a proper CA (step-ca, smallstep, or Let's Encrypt for the server cert)
  • Generate per-client certificates when possible
  • Set chmod 600 on private keys and correct ownership
  • On the server side, use --trust to point at your CA bundle

Self-signed certs work fine for homelabs; just distribute the CA and client certs securely (Ansible, scp with strict modes, or even systemd-creds).

Retention and maintenance

On the receiver, you can set limits in /etc/systemd/journal-remote.conf:

[Remote]
MaxUse=100G
KeepFree=20G
MaxFileSec=30day
Enter fullscreen mode Exit fullscreen mode

Or run periodic vacuuming via a systemd timer:

journalctl --directory=/var/log/journal/remote --vacuum-time=90d --vacuum-size=80G
Enter fullscreen mode Exit fullscreen mode

Verification and troubleshooting

On a client:

journalctl -u systemd-journal-upload -f
Enter fullscreen mode Exit fullscreen mode

On the server, watch incoming connections and check that new journal files appear with the expected hostname.

Common gotchas:

  • Certificate permission errors (the journal-remote user must read the key)
  • Firewall blocking 19532
  • Time skew between client and server (journal entries have monotonic timestamps, but wall-clock correlation still matters)

When to layer something else on top

This setup gives you centralized, structured logs you can query with the same tools you already know. For full-text search, alerting, or long-term retention beyond a few months, point a lightweight shipper (Vector, Fluent Bit) at the remote journal directory or use the HTTP gateway (journal-gatewayd) to feed a proper backend.

The native tools keep the operational surface tiny and the data format lossless. In most homelab and small-team environments, that is exactly what you need.

References

  • systemd-journal-remote.service(8) and journal-remote.conf(5) man pages
  • DigitalOcean: "How To Centralize Logs With Journald on Ubuntu 20.04"
  • systemd source and issue tracker discussions on resume behavior and TLS handling

Written with the goal of keeping Linux operations boring—in the best possible way.

Top comments (0)