For more powerful linux cli, we can combine a command directly with a file, or combining two or more commands in one shot.
It's called redirection when we combine a command with a file.
# copy file_1 content into file2
cat FILE_1.txt > FILE_2.txt
# create a new file with list of content of / directory
ls / > FILE.txt
The other one is called pipeline
# list all files or folder which are matched with these keywords
ls /home/host/ | grep KEYWORD_1 | grep KEYWORD_2
Some commands that we will use often to work with file in linux :
GREP
Grep is used for filtering files or folder based on a keyword.
Basic usage :
# display all line matched
grep KEYWORD FILE.txt
# display all line matched from multiple files
grep KEYWORD FILE_1.txt FILE_2.txt FILE_3.txt
# ignore keyword case
grep -i KEYWORD FILE.txt
# display lines without the matched line
grep -v KEYWORD FILE.txt
# display with line number
grep -n KEYWORD FILE.txt
# display a count of matching lines
grep -c KEYWORD FILE.txt
# display matched line for either keyword 1 or keyword 2
grep -w "KEYWORD_1 | KEYWORD_2" FILE.txt
# display 4 lines after the keyword, the number can be customized
grep -A 4 KEYWORD
# display 4 lines before the keyword, the number can be customized
grep -B 4 KEYWORD
REGEX
We can also combine grep
command with regular expression. Be careful when using regex, because it is case sensitive.
Basic usage :
grep -E "REGEX_PATTERN" FILE.txt
This command will display all lines which are matched with the regex pattern.
SED
Sed is stand for Stream Editor. Sed can be used for searching a file, find and replace, insertion or deletion. Sed is super useful for file editing without opening it. Sed also support regular expression pattern.
Basic usage :
-
sed
withs
parameter for substitution and/
for the delimiter
# replace first unix word in each line with linux
sed 's/unix/linux/' FILE.txt
# replace all unix word in each line with linux
sed 's/unix/linux/g' FILE.txt
# replace second unix word in each line with linux
sed 's/unix/linux/2' FILE.txt
# replace with case insensitive
sed 's/unix/linux/i' FILE.txt
# replace all unix word except in fifth line
sed '5!s/unix/linux/g' FILE.txt
# replace unix word from second and next line
sed 's/unix/linux/2g' FILE.txt
# replace unix word in third line
sed '3 s/unix/linux/' FILE.txt
# replace unix word in first line until third line
sed '1,3 s/unix/linux/' FILE.txt
# replace unix word in start from third line
sed '3,$ s/unix/linux/' FILE.txt
# display only replaced lines
sed -n 's/unix/linux/p' FILE.txt
# display replaced lines twice
sed 's/unix/linux/p' FILE.txt
-
sed
withd
for deletion
# delete fifth line
sed '5d' FILE.txt
# delete last line
sed '$d' FILE.txt
# delete third line until sixth line
sed '3,6d' FILE.txt
# delete from third line and next
sed '3,$d' FILE.txt
# delete line with matched pattern
sed '/REGEX_PATTERN/d' FILE.txt
# delete line with matched case insensitive pattern
sed -i '/REGEX_PATTERN/d' FILE.txt
# delete line with matched pattern and two lines after
sed '/abc/,+2d' FILE.txt
# delete every second line, start from third line
sed '3~2d' FILE.txt
# delete blank line
sed '/^$/d' FILE.txt
# delete blank line w
sed -i '/^#/d;/^$/d' FILE.txt
-
sed
withG
for file spacing
# insert one blank line after each line
sed G FILE.txt
# insert two blank lines after each line
sed 'G:G' FILE.txt
# insert a black line above every line which matches pattern
sed '/REGEX_PATTERN/x;p;x;' FILE.txt
# insert a blank line below every line which matches pattern
sed '/REGEX_PATTERN/G' FILE.txt
-
sed
with-n p
for displaying a file
# display a file
sed -n p FILE.txt
# display only fourth line
sed -n '4p' FILE.txt
# display from second line to fifth line
sed -n '2,5p' FILE.txt
# display only last line
sed -n '$p' FILE.txt
# display from third line and next
sed -n '3,$p' FILE.txt
# display entire file except second line to fourth line
sed -n '2,4d' FILE.txt
# display only line with matched pattern
sed -n '/REGEX_PATTERN/p' FILE.txt
# matched pattern start from fifth line
sed -n '/REGEX_PATTERN/,5p' FILE.txt
# display from second line to matched pattern line
sed -n '2,/REGEX_PATTERN/p' FILE.txt
-
sed
with=
for numbering lines
# add a number in front of each line
sed = FILE.txt
CUT
As the name suggests, cut i used for cutting out the section from each line of file and writing the result to a new file. We can use this command to cut parts of a line by some of these commands :
-
cut
with-b
or--bytes=LIST
Select by specifying a byte, a set of bytes, or a range of bytes.
# display first, second, and third byte of each line
cut -b '1,2,3' FILE.txt
# display first until third byte and fifth until seventh byte
cut -b '1-3,5-7' FILE.txt
# display third and next byte
cut -b '3-' FILE.txt
# display until third byte
cut -b '-3' FILE.txt
-
cut
with-c
or--characters=LIST
Select by specifying a character, a set of characters, or a range of characters.
# display first, second, and third characters of each line
cut -c '1,2,3' FILE.txt
# display first until third characters and fifth until seventh characters
cut -c '1-3,5-7' FILE.txt
# display third and next characters
cut -c '3-' FILE.txt
# display until third characters
cut -c '-3' FILE.txt
-
cut
with-f
or--fields=LIST
Select by specifying a field, a set of fields, or a range of fields. This command needs specified delimiter written with-d
or--delimiter=DELIMITER
# display all fields without specified delimiter
cut -f '1' FILE.txt
# display first until third fields, with comma delimiter
cut -d\, -f '1-3' FILE.txt
# display from third fields, with comma delimiter
cut -d\, -f '3-' FILE.txt
# display until third fields, with comma delimiter
cut -d\, -f '-3' FILE.txt
-
cut
with--complement
Displays all bytes, characters, or fields except the selected.
# display all bytes except the selected
cut --complement -b '1' FILE.txt
# display all characters except the selected
cut --complement -c '1' FILE.txt
# display all fields except the selected
cut --complement -d\, -f '1' FILE.txt
-
cut
with-s
or--only-delimiter
Do not print lines not containing delimiters.
# display only fields which have delimiter
cut -d\, -f '1-4' FILE.txt -s
-
cut
with--output-delimiter=STRING
Modify the output delimiter.
# display fields with # delimiter
cut -d\, -f '1-4' FILE.txt --output-delimiter="#"
SORT
This command will do as the name implies.
# sort the lines ascending
sort FILE.txt
# sort the lines descending
sort -r FILE.txt
# sort and create a new file
sort -o OUTPUT_FILE.txt ORIGINAL_TEXT.txt
# sort the numbering lines in ascending
sort -n FILE.txt
# sort the numbering lines in descending
sort -nr FILE.txt
# sort and remove duplicates
sort -u FILE.txt
# checks if a file is already sorted and return unsorted lines (if any)
sort -c FILE.txt
# sort a table by the nth column
sort -k2 FILE.txt
# sort by month for list of months
sort -M FILE.txt
UNIQ
We can delete the duplicated lines with this command. Uniq basic command will return the same result with sort -u
.
Basic usage :
# print only uniq line
uniq FILE.txt
# print a number how many times a line was repeated
uniq -c FILE.txt
# print repeated lines, only display one duplicated line per group
uniq -d FILE.txt
# print repeated lines, display all duplicated lines
uniq -D FILE.txt
# prints only the unique lines
uniq -u FILE.txt
# skip nth field to be skipped while comparing uniqueness
uniq -f 2 FILE.txt
# skip nth character to be skipped while comparing uniqueness
uniq -s 2 FILE.txt
# limit the comparison to a set number of characters
uniq -w 2 FILE.txt
# make the comparison case-insensitive
uniq -i FILE.txt
TR
We can use this command to translate or delete characters.
Basic usage :
# translate from lowercase with uppercase
cat FILE.txt | tr "[a-z]" "[A-Z]"
# translate from lowercase with uppercase
cat FILE.txt | tr "[:lower:]" "[:upper:]"
# translate whitespace with tab
cat FILE.txt | tr "[:space:]" "\t"
# translate brackets with curly barckets
cat FILE.txt | tr "()" "{}"
# translate multiple whitespace with single space
cat FILE.txt | tr -s "[:space:]" " "
# delete specific A character
cat FILE.txt | tr -d "A"
# delete digit character
cat FILE.txt | tr -d "[:digit:]"
# delete all except digit
cat FILE.txt | tr -cd "[:digit:]"
AWK
Awk is used for manipulating data and generating reports.
Fields identifier :
-
$0
: entire line of text -
$1
: first field -
$2
: second field -
$6
: sixth field -
$NF
: stands for “number of fields,” and represents the last field -
NR
: display number of fields -
OFS
: output field separator -
BEGIN
: begin of the file -
END
: end of the file
Basic usage :
# view file
awk '{print}' FILE.txt
# view specific lines
awk '{print $2,$3,$6}' FILE.txt
# view last field each lines
awk '{print $NF}' FILE.txt
# view number of fields each line
awk '{print NR}' FILE.txt
# print specific lines with / separator
awk 'OFS="/" {print $2,$3,$6}' FILE.txt
# add text in the beginning of the file
awk 'BEGIN {print "some text"} {print $2,$3,$6}' FILE.txt
# add text in the ending of the file
awk 'END {print "some text"} {print $2,$3,$6}' FILE.txt
# print the lines which is matched with the given pattern
awk '/REGEX_PATTERN/' FILE.txt
# print specific field after matched pattern each line
awk '/REGEX_PATTERN/ {print $2,$3,$6}' FILE.txt
`
Top comments (2)
Awesome!
Thanks, this is helpful documentation!