We have an AWS EC2 which suddenly rebooted.
The issue is that its /var/log/messages incomplete and the latest record is about rsyslog service killed – but nothing about who killed it and why.
Then I went to the journald but "surprise" – it stores logs only for the current boot so there was no way to find the reboot’s root cause.:
root@bttrm-stage-console:/home/admin# journalctl --list-boots
0 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:48:44 EEST
So, we need to configure journald to store logs after each reboot.
Its config file /etc/systemd/journald.conf.
To make logs to be stored always – update the Storage parameter to the auto or persistent.
If auto – then the /var/run/journal directory must be created manually, with the persistent – systemd will create it.
Update config, set persistent, restart systemd-journald to apply:
root@bttrm-stage-console:/home/admin# systemctl restart systemd-journald
Check the logs directory:
root@bttrm-stage-console:/home/admin# ll /var/log/journal/
total 4
drwxr-xr-x 2 root root 4096 Sep 24 17:54 7066493f616c4a6285c19cfaa2884b09
root@bttrm-stage-console:/home/admin# ll /var/log/journal/7066493f616c4a6285c19cfaa2884b09/
total 57348
-rw-r----- 1 root root 58720256 Sep 24 17:54 system.journal
Reboot the EC2 and check logs again:
root@bttrm-stage-console:/home/admin# journalctl --list-boots
-1 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:56:19 EEST
0 591136b94276490580f4e39e576e639e Tue 2019-09-24 17:56:20 EEST—Tue 2019-09-24 17:56:41 EEST
Now you are able to check the previous reboot log:
root@bttrm-stage-console:/home/admin# journalctl -b -1
-- Logs begin at Tue 2019-09-24 12:13:02 EEST, end at Tue 2019-09-24 17:57:36 EEST. --
Sep 24 12:13:02 bttrm-stage-console CRON[19407]: pam\_unix(cron:session): session closed for user me-v3
Size can be limited by setting the SystemMaxUse, see the documentation.
Done.
Top comments (0)