Forem

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

3 1

Linux: persistent logs in systemd-journald

We have an AWS EC2 which suddenly rebooted.

The issue is that its /var/log/messages incomplete and the latest record is about rsyslog service killed – but nothing about who killed it and why.

Then I went to the journald but "surprise" – it stores logs only for the current boot so there was no way to find the reboot’s root cause.:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
0 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:48:44 EEST
Enter fullscreen mode Exit fullscreen mode

So, we need to configure journald to store logs after each reboot.

Its config file /etc/systemd/journald.conf.

To make logs to be stored always – update the Storage parameter to the auto or persistent.

If auto – then the /var/run/journal directory must be created manually, with the persistentsystemd will create it.

Update config, set persistent, restart systemd-journald to apply:

root@bttrm-stage-console:/home/admin# systemctl restart systemd-journald
Enter fullscreen mode Exit fullscreen mode

Check the logs directory:

root@bttrm-stage-console:/home/admin# ll /var/log/journal/
total 4
drwxr-xr-x 2 root root 4096 Sep 24 17:54 7066493f616c4a6285c19cfaa2884b09

root@bttrm-stage-console:/home/admin# ll /var/log/journal/7066493f616c4a6285c19cfaa2884b09/
total 57348
-rw-r----- 1 root root 58720256 Sep 24 17:54 system.journal
Enter fullscreen mode Exit fullscreen mode

Reboot the EC2 and check logs again:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
-1 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:56:19 EEST

0 591136b94276490580f4e39e576e639e Tue 2019-09-24 17:56:20 EEST—Tue 2019-09-24 17:56:41 EEST
Enter fullscreen mode Exit fullscreen mode

Now you are able to check the previous reboot log:

root@bttrm-stage-console:/home/admin# journalctl -b -1

-- Logs begin at Tue 2019-09-24 12:13:02 EEST, end at Tue 2019-09-24 17:57:36 EEST. --

Sep 24 12:13:02 bttrm-stage-console CRON[19407]: pam\_unix(cron:session): session closed for user me-v3
Enter fullscreen mode Exit fullscreen mode

Size can be limited by setting the SystemMaxUse, see the documentation.

Done.

Similar posts

Top comments (0)

Image of Docusign

🛠️ Bring your solution into Docusign. Reach over 1.6M customers.

Docusign is now extensible. Overcome challenges with disconnected products and inaccessible data by bringing your solutions into Docusign and publishing to 1.6M customers in the App Center.

Learn more