DEV Community

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

3 1

Linux: persistent logs in systemd-journald

We have an AWS EC2 which suddenly rebooted.

The issue is that its /var/log/messages incomplete and the latest record is about rsyslog service killed – but nothing about who killed it and why.

Then I went to the journald but "surprise" – it stores logs only for the current boot so there was no way to find the reboot’s root cause.:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
0 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:48:44 EEST
Enter fullscreen mode Exit fullscreen mode

So, we need to configure journald to store logs after each reboot.

Its config file /etc/systemd/journald.conf.

To make logs to be stored always – update the Storage parameter to the auto or persistent.

If auto – then the /var/run/journal directory must be created manually, with the persistentsystemd will create it.

Update config, set persistent, restart systemd-journald to apply:

root@bttrm-stage-console:/home/admin# systemctl restart systemd-journald
Enter fullscreen mode Exit fullscreen mode

Check the logs directory:

root@bttrm-stage-console:/home/admin# ll /var/log/journal/
total 4
drwxr-xr-x 2 root root 4096 Sep 24 17:54 7066493f616c4a6285c19cfaa2884b09

root@bttrm-stage-console:/home/admin# ll /var/log/journal/7066493f616c4a6285c19cfaa2884b09/
total 57348
-rw-r----- 1 root root 58720256 Sep 24 17:54 system.journal
Enter fullscreen mode Exit fullscreen mode

Reboot the EC2 and check logs again:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
-1 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:56:19 EEST

0 591136b94276490580f4e39e576e639e Tue 2019-09-24 17:56:20 EEST—Tue 2019-09-24 17:56:41 EEST
Enter fullscreen mode Exit fullscreen mode

Now you are able to check the previous reboot log:

root@bttrm-stage-console:/home/admin# journalctl -b -1

-- Logs begin at Tue 2019-09-24 12:13:02 EEST, end at Tue 2019-09-24 17:57:36 EEST. --

Sep 24 12:13:02 bttrm-stage-console CRON[19407]: pam\_unix(cron:session): session closed for user me-v3
Enter fullscreen mode Exit fullscreen mode

Size can be limited by setting the SystemMaxUse, see the documentation.

Done.

Similar posts

Speedy emails, satisfied customers

Postmark Image

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay