Viewing and Managing System Logs Using rsyslog in Linux

Linux systems generate a continuous stream of logs from the kernel, services, and user applications, and managing them is essential for diagnosing issues, tracking activity, and maintaining operational visibility. rsyslog is a widely adopted logging daemon that collects, processes, stores, and optionally forwards these logs, supporting multiple input sources, advanced filtering, rule-based message routing, and remote logging capabilities beyond what basic syslog tools offer. This guide uses rsyslog to view and manage system logs, define custom rules, and forward logs to remote servers for centralized logging. By the end, you'll have custom rsyslog rules routing SSH and application logs to their own files, plus a client forwarding logs to a centralized remote server.

---

How rsyslog Works: The Input, Filter, and Output Model

rsyslog processes log messages through a structured pipeline composed of three core stages:

• Input: rsyslog collects logs from various sources such as system sockets (imuxsock), kernel messages (imklog), plain-text files (imfile), or network protocols like TCP/UDP (imtcp, imudp). Each input module defines how and where logs enter the system.
• Filter: After logs are received, rsyslog evaluates them using filters based on facility, severity, or custom expression rules, which determines which logs are processed further and how they're handled.
• Output: Filtered logs are routed to output destinations such as local log files, SQL databases, remote syslog servers, or message queues, each managed by a specific module with custom formatting and delivery rules.

Common Log File Locations in Linux

Log files are typically stored under /var/log/ and record system events, application activity, kernel operations, and security information. File names and purposes can vary slightly across distributions, but the following are among the most common:

• /var/log/syslog: Captures general system activity, including messages from system services and applications. Commonly used on Debian and Ubuntu-based systems for broad troubleshooting.
• /var/log/messages: A general-purpose log file used on Red Hat-based systems (like CentOS and Fedora), logging system messages not related to authentication or the kernel.
• /var/log/auth.log: Records authentication attempts, including user logins, sudo actions, and SSH access. Essential for auditing and security monitoring.
• /var/log/kern.log: Contains kernel-generated messages not handled by other facilities. Useful for diagnosing hardware issues or kernel module problems.
• /var/log/dmesg: Stores the kernel ring buffer from the most recent boot, showing hardware detection and initialization messages.
• /var/log/cron.log: Logs the execution of scheduled cron jobs, useful when debugging missed or failed automated tasks.

View Logs with Basic Linux Commands

Before configuring rsyslog, it's useful to know how to inspect log files with common Linux utilities.

• cat: Outputs the entire contents of a file to the terminal. Best used for small log files.
  

  $ cat <LOGFILE_PATH>
  

• more: Displays the contents of a file one screen at a time. Use Space to scroll forward, q to quit.
  

  $ more <LOGFILE_PATH>
  

• less: Opens the file in a scrollable viewer, letting you navigate forward and backward through large log files.
  

  $ less <LOGFILE_PATH>
  

• tail: Displays the last 10 lines of a file by default. Useful for checking recent log entries.
  

  $ tail <LOGFILE_PATH>
  

• tail -f: Continuously streams new log entries as they're written, ideal for real-time monitoring.
  

  $ tail -f <LOGFILE_PATH>
  

• zcat, zless, zgrep: Read or search compressed .gz log files created by log rotation without decompressing them first.
  

  $ zless <GZ_LOGFILE_PATH>
  

Filter Logs Using grep, awk, and sed

• grep: Searches for specific keywords or regular expressions within log files, useful for locating errors, warnings, or other patterns.
  

  $ grep "error" /var/log/syslog
  $ grep -i "failed" /var/log/auth.log
  

• awk: Processes log entries by splitting them into fields, useful for extracting timestamps, usernames, or specific columns from structured logs.
  

  $ awk '{print $1, $2, $5}' /var/log/syslog
  

• sed: A stream editor that filters or transforms lines of text, useful for extracting a block of logs between two keywords or substituting text.
  

  $ sed -n '/Start/,/End/p' /var/log/syslog
  $ sed 's/error/ERROR/g' /var/log/syslog
  

Configure rsyslog Using /etc/rsyslog.conf

The behavior of rsyslog is primarily controlled through its main configuration file at /etc/rsyslog.conf, which defines how incoming log messages are processed, filtered, and directed based on their source (facility), importance (severity level), and destination (action).

The general format for a rule is:

FACILITY.LEVEL    ACTION

• FACILITY: The source of the log message (for example, auth, cron, daemon, kern, mail, user).
• LEVEL: The severity of the log message (for example, debug, info, notice, warning, err, crit, alert, emerg).
• ACTION: The destination for the log (for example, a file, remote server, or program).

Common Configuration Examples

• Send auth logs to a separate file:
  

  auth.*    /var/log/auth.log
  

  Logs from the auth facility (for example, SSH, sudo) go to /var/log/auth.log.

• Store kernel logs in a separate file:
  

  kern.*    /var/log/kern.log
  

  All kernel messages are redirected to /var/log/kern.log.

• Log only critical mail errors:
  

  mail.crit    /var/log/mail_critical.log
  

  Only critical-level messages from the mail facility are logged.

• Log everything except mail and auth to a common file:
  

  *.info;mail.none;authpriv.none    /var/log/general.log
  

  Sends all info-level or higher messages to /var/log/general.log, except those from mail and authpriv.

• Discard unwanted logs:
  

  *.debug    ~
  

  The tilde (~) discards all debug-level logs to reduce noise.

• Log user messages to the console:
  

  user.*    /dev/console
  

Add Custom Logging Rules

Log SSH Activity

1. Create a new configuration file in /etc/rsyslog.d/:

$ sudo nano /etc/rsyslog.d/10-ssh.conf

2. Add the following rule to capture all messages from the authpriv facility:

authpriv.*    /var/log/ssh.log

3. Restart the rsyslog service to apply the new configuration:

$ systemctl restart rsyslog.service

SSH-related log entries are now written to /var/log/ssh.log, making it easier to monitor authentication activity independently of other system logs.

Log Activity for a Custom Application

1. Create a new rsyslog configuration file in /etc/rsyslog.d/:

$ sudo nano /etc/rsyslog.d/20-myapp.conf

2. Add the following rule to match logs from your application:

if ($programname == 'myapp') then /var/log/myapp.log
& stop

3. Restart the rsyslog service to apply the changes:

$ systemctl restart rsyslog.service

All syslog messages from myapp are now written to /var/log/myapp.log, keeping application logs organized and separate from system logs.

Forward Logs to a Remote Server

Centralizing logs from multiple systems on a single server helps streamline monitoring and troubleshooting. rsyslog can forward logs from a client to a remote server over TCP or UDP.

Configure the Client and Server

1. Create a configuration file on the client to forward all logs to the remote server:

$ sudo nano /etc/rsyslog.d/60-remote.conf

2. Add the forwarding rule to the file:

*.* @@192.2.0.3:514

Replace 192.2.0.3 with your remote rsyslog server's IP address. Use a single @ for UDP, or double @@ for TCP.

3. On the remote rsyslog server, enable TCP log reception by editing the main config file:

$ sudo nano /etc/rsyslog.conf

4. Uncomment or add the following lines to enable TCP input:

module(load="imtcp")
input(type="imtcp" port="514")

5. (Optional) Organize incoming logs by hostname and program name on the server:

$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs

This stores logs in structured folders like /var/log/remote/client1/myapp.log.

6. Restart rsyslog on both systems to apply the changes:

$ sudo systemctl restart rsyslog

After setup, the client system sends logs to the remote server, which organizes and stores them based on the defined templates.

Test the Configuration

1. Send a test log message:

$ logger -p local0.info "Test message from rsyslog"

Then check the appropriate log file (for example, /var/log/syslog or your custom-defined log path) to confirm that the message was recorded.

Best Practices for Using rsyslog

• Use separate files: For better management, store logs from different services or applications in separate files.
• Secure your logs: Protect log files with proper file permissions to prevent unauthorized access.
• Use log rotation: Set up log rotation to avoid log files growing too large.
• Centralized logging: Use remote logging to collect logs from multiple systems in one place.
• Monitor your logs: Check log files often to identify and address issues promptly.

Next Steps

• Set up logrotate alongside rsyslog to automatically archive and compress older log files.
• Ship logs from the remote rsyslog server to a centralized SIEM or log aggregation platform.
• Add facility-specific rules for other services you run, following the same pattern used for SSH and custom applications.
• Automate alerting on critical log patterns using grep/awk pipelines or a log monitoring tool.

For the full guide with additional tips, visit the original article on Vultr Docs.