koh-sh

Posted on Nov 8, 2019 • Originally published at koh-sh.hatenablog.com

Finally I am using journalctl

#linux

I know it is too late but now I found out journalctl is convenient.

What is journalctl

From the output of man journalctl

NAME
       journalctl - Query the systemd journal

SYNOPSIS
       journalctl [OPTIONS...] [MATCHES...]

DESCRIPTION
       journalctl may be used to query the contents of the systemd(1) journal as written by
       systemd-journald.service(8).

OK, then what is journald?
Let's see man systemd-journald

DESCRIPTION
       systemd-journald is a system service that collects and stores logging data. It creates and
       maintains structured, indexed journals based on logging information that is received from a
       variety of sources:

To put it simply, journald corrects and manages many kinds of logs.
It can correct the below logs. Pretty much every log.

Kernel log messages, via kmsg
Simple system log messages, via the libc syslog(3) call
Structured system log messages via the native Journal API, see sd_journal_print(3)
Standard output and standard error of service units. For further details see below.
Audit records, originating from the kernel audit subsystem

Work with journalctl

So let's play with journalctl.
Environment: CentOS 7.5.1804 on Vagrant.

Without any options, it prints all logs stated above.
Older entries come top and operated with a pager like less command.

% sudo journalctl

-- Logs begin at Sat 2019-04-13 14:05:21 JST, end at Sat 2019-04-13 14:26:56 JST. --
Apr 13 14:05:21 localhost.localdomain systemd-journal[84]: Runtime journal is using 6.1M (max allowed 49.5M, trying to leave 74.3M free of 489.6M available <E2><86><92> current limit 49.5M).
Apr 13 14:05:21 localhost.localdomain kernel: Initializing cgroup subsys cpuset
Apr 13 14:05:21 localhost.localdomain kernel: Initializing cgroup subsys cpu
<ellipsis>
Apr 13 14:20:01 Vag2 systemd[1]: Stopping User Slice of root.
Apr 13 14:20:32 Vag2 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 13 14:20:32 Vag2 systemd[1]: Started Cleanup of Temporary Directories.
Apr 13 14:23:18 Vag2 sudo[2428]:  vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/journalctl

Options can be used to filter logs.
Here are some useful options.

Kernel messages

-k prints kernel messages like dmesg command.

% sudo journalctl -k

-- Logs begin at Sat 2019-04-13 14:05:21 JST, end at Sat 2019-04-13 14:34:48 JST. --
Apr 13 14:05:21 localhost.localdomain kernel: Initializing cgroup subsys cpuset
Apr 13 14:05:21 localhost.localdomain kernel: Initializing cgroup subsys cpu
Apr 13 14:05:21 localhost.localdomain kernel: Initializing cgroup subsys cpuacct
Apr 13 14:05:21 localhost.localdomain kernel: Linux version 3.10.0-862.11.6.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Tue Aug 14 21:49:04 UTC 2018
Apr 13 14:05:21 localhost.localdomain kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=/dev/mapper/centos-root ro net.ifnames=0 biosdevname=0 crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8

Units

-u prints specific unit logs.
Printing HAproxy's log for example.

% sudo journalctl -u haproxy

-- Logs begin at Sat 2019-04-13 14:05:21 JST, end at Sat 2019-04-13 15:00:48 JST. --
Apr 13 14:05:31 Vag2 systemd[1]: Started HAProxy Load Balancer.
Apr 13 14:05:31 Vag2 systemd[1]: Starting HAProxy Load Balancer...
Apr 13 14:05:31 Vag2 haproxy-systemd-wrapper[1019]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Apr 13 15:00:45 Vag2 systemd[1]: Reloaded HAProxy Load Balancer.
Apr 13 15:00:45 Vag2 haproxy-systemd-wrapper[1019]: haproxy-systemd-wrapper: re-executing on SIGUSR2.
Apr 13 15:00:45 Vag2 haproxy-systemd-wrapper[1019]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 1036

Priority

-p prints a specific priority level of logs.
Specified with below numbers or strings.

emerg(0)
alert(1)
crit(2)
err(3)
warning(4)
notice(5)
info(6)
debug(7)

% sudo journalctl -p err

-- Logs begin at Sat 2019-04-13 14:05:21 JST, end at Sat 2019-04-13 14:48:50 JST. --
Apr 13 14:05:31 Vag2 systemd[1]: Failed to start Crash recovery kernel arming.

Priorities can be multiple.

% sudo journalctl -p err -p warning

-- Logs begin at Sat 2019-04-13 14:05:21 JST, end at Sat 2019-04-13 14:49:58 JST. --
<ellipsis>
Apr 13 05:05:29 Vag2 kernel: 00:00:00.000268 main     Executable: /opt/VBoxGuestAdditions-5.2.18/sbin/VBoxService
                             00:00:00.000268 main     Process ID: 906
                             00:00:00.000269 main     Package type: LINUX_64BITS_GENERIC
Apr 13 05:05:29 Vag2 kernel: 00:00:00.003419 main     5.2.18 r124319 started. Verbose level = 0
Apr 13 14:05:31 Vag2 chronyd[647]: Forward time jump detected!
Apr 13 14:05:31 Vag2 systemd[1]: Failed to start Crash recovery kernel arming.
Apr 13 14:05:31 Vag2 systemd[1]: kdump.service failed.
Apr 13 14:05:41 Vag2 kernel: 00:00:10.017183 timesync vgsvcTimeSyncWorker: Radical guest time change: 32 411 887 294 000ns (GuestNow=1 555 131 941 050 133 000 ns GuestLast=1

Other than filtering

There are some options which change how to print.

Reverse order

-r prints newer logs first.

% sudo journalctl -r

-- Logs begin at Sat 2019-04-13 14:05:21 JST, end at Sat 2019-04-13 14:56:44 JST. --
Apr 13 14:56:44 Vag2 sudo[2609]:  vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/journalctl -r
Apr 13 14:56:24 Vag2 sudo[2605]:  vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/journalctl -r
Apr 13 14:50:01 Vag2 systemd[1]: Stopping User Slice of root.
Apr 13 14:50:01 Vag2 systemd[1]: Removed slice User Slice of root.
Apr 13 14:50:01 Vag2 CROND[2596]: (root) CMD (/usr/lib64/sa/sa1 1 1)

Follow

-f keep printing new entries while using it. Like tail -f command.

% sudo journalctl -f

-- Logs begin at Sat 2019-04-13 14:05:21 JST. --
Apr 13 14:50:01 Vag2 systemd[1]: Created slice User Slice of root.
Apr 13 14:50:01 Vag2 systemd[1]: Starting User Slice of root.
Apr 13 14:50:01 Vag2 systemd[1]: Started Session 8 of user root.
Apr 13 14:50:01 Vag2 systemd[1]: Starting Session 8 of user root.
Apr 13 14:50:01 Vag2 CROND[2596]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Apr 13 14:50:01 Vag2 systemd[1]: Removed slice User Slice of root.
Apr 13 14:50:01 Vag2 systemd[1]: Stopping User Slice of root.
Apr 13 14:56:24 Vag2 sudo[2605]:  vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/journalctl -r

Non pager

-no-pager prints logs without pager.

% sudo journalctl --no-pager

-- Logs begin at Sat 2019-04-13 14:05:21 JST, end at Sat 2019-04-13 15:02:30 JST. --
Apr 13 14:05:21 localhost.localdomain systemd-journal[84]: Runtime journal is using 6.1M (max allowed 49.5M, trying to leave 74.3M free of 489.6M available → current limit 49.5M).
Apr 13 14:05:21 localhost.localdomain kernel: Initializing cgroup subsys cpuset
Apr 13 14:05:21 localhost.localdomain kernel: Initializing cgroup subsys cpu
<ellipsis>
Apr 13 15:01:01 Vag2 anacron[2654]: Will run job `cron.daily' in 30 min.
Apr 13 15:01:01 Vag2 anacron[2654]: Will run job `cron.weekly' in 50 min.
Apr 13 15:01:01 Vag2 anacron[2654]: Jobs will be executed sequentially
Apr 13 15:02:30 Vag2 sudo[2661]:  vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/journalctl --no-pager

Conclusion

I think if you are familiar with these options, journalctl should be very convenient for you.
I know some people don't like it but I personally like it.

Top comments (2)

Thomas H Jones II • Nov 8 '19

Oof:

_CentOS 7.5.1804 _

Damn, son: 7.7 is rolling up on three months old. Time to upgrade. Gives you access to some other nifty features and rebases some software to less-buggy versions.

koh-sh • Nov 9 '19

Thanks for the heads up.
I will try 7.7, or maybe CentOS8.

DEV Community