DEV Community

abbazs
abbazs

Posted on

5 text processing tools `grep, sed, awk, cut, and tr` to score some marks in System Commands OPPE

Introduction

This tutorial covers five essential Linux text processing tools: grep, sed, awk, cut, and tr.

1. grep (Global Regular Expression Print)

grep is used for searching text using patterns.

Example 1: Basic text search

$ echo -e "This is an error\nThis is a warning\nThis is info" | grep "error"
This is an error
Enter fullscreen mode Exit fullscreen mode

Example 2: Case-insensitive search

$ echo -e "Warning: System overload\nwarning: Low memory" | grep -i "warning"
Warning: System overload
warning: Low memory
Enter fullscreen mode Exit fullscreen mode

Example 3: Display line numbers

$ echo -e "Line 1\nLine 2\nCritical error\nLine 4" | grep -n "Critical"
3:Critical error
Enter fullscreen mode Exit fullscreen mode

Example 4: Recursive search in directories

Assume we have a directory structure:

project/
├── file1.txt (contains "TODO: Fix this")
└── subdir/
    └── file2.txt (contains "TODO: Implement feature")
Enter fullscreen mode Exit fullscreen mode
$ grep -r "TODO" project/
project/file1.txt:TODO: Fix this
project/subdir/file2.txt:TODO: Implement feature
Enter fullscreen mode Exit fullscreen mode

Example 5: Invert match

$ echo -e "Success: Task 1\nError: Task 2\nSuccess: Task 3" | grep -v "Success"
Error: Task 2
Enter fullscreen mode Exit fullscreen mode

Example 6: Use regular expressions

$ echo -e "Phone: 123-456-7890\nInvalid: 123-45-6789" | grep -E "[0-9]{3}-[0-9]{3}-[0-9]{4}"
Phone: 123-456-7890
Enter fullscreen mode Exit fullscreen mode

Example 7: Print only matched parts (-o option)

$ echo "The IP address is 192.168.1.1 and 10.0.0.1" | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"
192.168.1.1
10.0.0.1
Enter fullscreen mode Exit fullscreen mode

2. sed (Stream Editor)

sed is used for text transformation.

Example 1: Basic substitution

$ echo "The color is red" | sed 's/red/blue/'
The color is blue
Enter fullscreen mode Exit fullscreen mode

Example 2: Global substitution

$ echo "one two one two" | sed 's/one/three/g'
three two three two
Enter fullscreen mode Exit fullscreen mode

Example 3: In-place editing

Assume we have a file named "colors.txt" with content "The color is red"

$ sed -i 's/red/green/' colors.txt
$ cat colors.txt
The color is green
Enter fullscreen mode Exit fullscreen mode

Example 4: Delete lines

$ echo -e "Keep this\nDelete me\nKeep this too" | sed '/Delete/d'
Keep this
Keep this too
Enter fullscreen mode Exit fullscreen mode

Example 5: Print specific lines

$ seq 10 | sed -n '3,7p'
3
4
5
6
7
Enter fullscreen mode Exit fullscreen mode

Example 6: Multiple operations

$ echo -e "foo bar\nbaz foo" | sed -e 's/foo/qux/g' -e '/baz/d'
qux bar
Enter fullscreen mode Exit fullscreen mode

3. awk (Aho, Weinberger, and Kernighan)

awk is a powerful text-processing tool for working with structured data.

Example 1: Print specific fields

$ echo -e "John Doe 25\nJane Smith 30" | awk '{print $1, $3}'
John 25
Jane 30
Enter fullscreen mode Exit fullscreen mode

Example 2: Use custom field separator

$ echo "name:John:age:30" | awk -F':' '{print $2}'
John
Enter fullscreen mode Exit fullscreen mode

Example 3: Sum values in a column

$ echo -e "10\n20\n30" | awk '{sum += $1} END {print sum}'
60
Enter fullscreen mode Exit fullscreen mode

Example 4: Conditional printing

$ echo -e "John 25\nJane 35\nMike 40" | awk '$2 > 30 {print $0}'
Jane 35
Mike 40
Enter fullscreen mode Exit fullscreen mode

Example 5: Calculate average

$ echo -e "10\n20\n30\n40\n50" | awk '{sum += $1; count++} END {print sum/count}'
30
Enter fullscreen mode Exit fullscreen mode

Example 6: Use regular expressions

$ echo -e "apple\nbanana\ncherry" | awk '/^b/ {print $0}'
banana
Enter fullscreen mode Exit fullscreen mode

4. cut

cut is used for extracting sections from each line of files.

Example 1: Extract specific characters

$ echo "Hello, World!" | cut -c 1-5
Hello
Enter fullscreen mode Exit fullscreen mode

Example 2: Extract specific fields with delimiter

$ echo "John,Doe,30,New York" | cut -d',' -f 2,4
Doe,New York
Enter fullscreen mode Exit fullscreen mode

Example 3: Extract range of fields

$ echo "field1:field2:field3:field4:field5" | cut -d':' -f 1-3
field1:field2:field3
Enter fullscreen mode Exit fullscreen mode

Example 4: Extract to end of line

$ echo "Name: John Doe, Age: 30, City: New York" | cut -d':' -f 2-
 John Doe, Age: 30, City: New York
Enter fullscreen mode Exit fullscreen mode

Example 5: Exclude specific fields

$ echo "1,2,3,4,5" | cut -d',' --complement -f 2,4
1,3,5
Enter fullscreen mode Exit fullscreen mode

5. tr (Translate)

tr is used for translating or deleting characters.

Example 1: Convert lowercase to uppercase

$ echo "hello world" | tr 'a-z' 'A-Z'
HELLO WORLD
Enter fullscreen mode Exit fullscreen mode

Example 2: Delete specific characters

$ echo "hello 123 world" | tr -d '0-9'
hello  world
Enter fullscreen mode Exit fullscreen mode

Example 3: Squeeze repeating characters

$ echo "hello      world" | tr -s ' '
hello world
Enter fullscreen mode Exit fullscreen mode

Example 4: Translate characters

$ echo "hello world" | tr 'hw' 'HW'
Hello World
Enter fullscreen mode Exit fullscreen mode

Example 5: Delete all except specified characters

$ echo "hello 123 world" | tr -cd '0-9\n'
123
Enter fullscreen mode Exit fullscreen mode

Conclusion

These five tools - grep, sed, awk, cut, and tr - are powerful utilities for text processing in Linux. The examples provided demonstrate their basic usage and some advanced features. Practice with these examples and explore their man pages for more advanced usage using man command.

Top comments (0)