DEV Community

Cover image for 5 log parsing commands
cuongnp
cuongnp

Posted on

1

5 log parsing commands

#linux #beginners #development #programming

Have you ever tried to find something in the server log file? While downloading and opening the file in an editor might seem straightforward, it's often time-consuming and unproductive. Instead, using command-line tools can be more efficient and effective. Here are some common commands you should try.

The practice file today is system.log

2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped
Enter fullscreen mode Exit fullscreen mode

1. Display the Contents of the Log File

cat Command

  • Purpose: used to display the content of files.
  • Usage: cat filename
  • Example: cat server.log 
$ cat system.log
2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped
Enter fullscreen mode Exit fullscreen mode

2. Search for lines

grep

  • Purpose: powerful command for searching text using patterns, and filtering log entries based on specific criteria.
  • Usage: cat filename | grep “filter-condition” or grep condition filename
  • Example: grep "ERROR" server.log 
$ grep "ERROR" system.log

2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
Enter fullscreen mode Exit fullscreen mode

3. Display Lines with Customize Condition

awk

  • Purpose: Introduce awk as a powerful text processing tool, ideal for manipulating data and generating reports.
  • Usage: awk condition filename
  • Example 1: Display lines with timestamps between 13:50:00 and 14:00:00: 
$ awk '/13:5[0-9]:[0-9][0-9]/ || /14:00:00/' system.log

2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
Enter fullscreen mode Exit fullscreen mode
  • Example 2: Extract and print the date and time of each entry 
$ awk '{print $1, $2}' system.log

2024-06-12 13:39:30
2024-06-12 13:40:12
2024-06-12 13:41:05
2024-06-12 13:42:16
2024-06-12 13:43:27
2024-06-12 13:44:38
2024-06-12 13:45:49
2024-06-12 13:46:50
2024-06-12 13:47:51
2024-06-12 13:48:52
2024-06-12 13:49:53
2024-06-12 13:50:54
2024-06-12 13:51:55
2024-06-12 13:52:56
2024-06-12 13:53:57
2024-06-12 13:54:58
2024-06-12 13:55:59
2024-06-12 13:56:00
2024-06-12 13:57:01
2024-06-12 13:58:02
2024-06-12 13:59:03
2024-06-12 14:00:04
2024-06-12 14:01:05
2024-06-12 14:02:06
2024-06-12 14:03:07
2024-06-12 14:04:08
2024-06-12 14:05:09
2024-06-12 14:06:10
2024-06-12 14:07:11
2024-06-12 14:08:12
2024-06-12 14:09:13
2024-06-12 14:10:14
2024-06-12 14:11:15
2024-06-12 14:12:16
2024-06-12 14:13:17
2024-06-12 14:14:18
Enter fullscreen mode Exit fullscreen mode

4. Sort Log Entries

sort

  • Purpose: Sort lines in text files.
  • Usage: Sort log entries by date, time, or any other field.
  • Example: cat system.log | awk '{print $1, $2, $3}' | sort 
$ cat system.log | awk '{print $1, $2, $3}' | sort
2024-06-12 13:39:30 [INFO]
2024-06-12 13:40:12 [ERROR]
2024-06-12 13:41:05 [INFO]
2024-06-12 13:42:16 [WARNING]
2024-06-12 13:43:27 [INFO]
2024-06-12 13:44:38 [ERROR]
2024-06-12 13:45:49 [INFO]
2024-06-12 13:46:50 [INFO]
2024-06-12 13:47:51 [INFO]
2024-06-12 13:48:52 [INFO]
2024-06-12 13:49:53 [INFO]
2024-06-12 13:50:54 [ERROR]
2024-06-12 13:51:55 [WARNING]
2024-06-12 13:52:56 [INFO]
2024-06-12 13:53:57 [ERROR]
2024-06-12 13:54:58 [INFO]
2024-06-12 13:55:59 [INFO]
2024-06-12 13:56:00 [INFO]
2024-06-12 13:57:01 [INFO]
2024-06-12 13:58:02 [ERROR]
2024-06-12 13:59:03 [INFO]
2024-06-12 14:00:04 [WARNING]
2024-06-12 14:01:05 [INFO]
2024-06-12 14:02:06 [ERROR]
2024-06-12 14:03:07 [INFO]
2024-06-12 14:04:08 [INFO]
2024-06-12 14:05:09 [INFO]
2024-06-12 14:06:10 [INFO]
2024-06-12 14:07:11 [INFO]
2024-06-12 14:08:12 [ERROR]
2024-06-12 14:09:13 [WARNING]
2024-06-12 14:10:14 [INFO]
2024-06-12 14:11:15 [ERROR]
2024-06-12 14:12:16 [INFO]
2024-06-12 14:13:17 [INFO]
2024-06-12 14:14:18 [INFO]
Enter fullscreen mode Exit fullscreen mode

5. Unique the display result

uniq

  • Purpose: Describe how uniq removes or counts duplicate lines.
  • Usage: cat filename | uniq -c
  • Example: cat server.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c 
cat system.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c
   2 Cleanup
   2 Could
   4 Failed
Enter fullscreen mode Exit fullscreen mode

Final thought

Analyzing information files is crucial for system administration, troubleshooting, and monitoring. Using a combination of command-line tools like cat, grep, awk, sort, and uniq, you can effectively manage and extract valuable insights from your log files.

profile
Sentry
 Promoted

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

Top comments (0)

profile
Pieces.app
 Promoted

The Most Contextual AI Development Assistant

Pieces.app image

Our centralized storage agent works on-device, unifying various developer tools to proactively capture and enrich useful materials, streamline collaboration, and solve complex problems through a contextual understanding of your unique workflow.

👥 Ideal for solo developers, teams, and cross-company projects

Learn more

Read next

mikeyoung44 profile image

AI Language Models Show Strange "Hyperfitting" Effect When Fine-Tuned for Precision

Mike Young -

mikeyoung44 profile image

New 4-Bit Training Method Cuts AI Model Memory Usage in Half While Maintaining Accuracy

Mike Young -

mikeyoung44 profile image

New Context Pruning Method Cuts AI Memory Use by 50% While Maintaining Accuracy

Mike Young -

mikeyoung44 profile image

AI Models Still Fail Basic Physics Tests, New Benchmark Shows 18.4% Improvement Possible

Mike Young -