TL;DR
It was me using Unix terminal every single day, typing hundred lines of terminal command, but sometimes, I encounter some unexpected behaviors at the moment I press Enter button after a command. It's pretty frustrated at some points, it turns out that the way the Shell sees the world is different than mine. I read some books (at least some chapters) about how the Shell acts upon a command, I find out it's really interesting so that I share with you today. Hope this helpful.
Because this is pretty straightforward. With only echo command, we can examine several interesting features, the "magic" of the Shell
The Expansion
Every time we press enter after a command, it get transformed by the Shell under the hood before get carried out. The process that makes it happen is call expansion. The expansion is that when you type in something, it get expanded to another thing before the shell acts upon it. For example: The * means a lot to the Shell. To illustrate this, I will use the echo command, as you recall that echo is a simple Shell built-in function that directs our typing to standard output
phuong@Arch ~$ echo Hello world
Hello world
So the * wildcard means: Match any characters in file-name. The fact that the * get transformed into other characters before being carried out by the shell so that the echo command never see the *
phuong@Arch ~/kitchen$ echo *
bash device devops gateway go hook iuh js linux ml python
Filename expansion
In fact, this behavior carried out by a mechanism, which is popularly called path name expansion. As we know, file names begin with period character (ex: .git) is hidden. The path name extension respect this rule. An expansion does not reveal hidden files.
phuong@Arch ~$ echo D*
Desktop Documents Downloads
But we can show all hidden files by following command:
phuong@Arch ~$ echo .*
. .. .3T .ansible .anydesk .bash_history .bash_logout .bash_profile .bashrc .bashrc.backup .boto .cache .config .dbshell .designer .docker
Arithmetic expansion
We are also able to use our terminal command as a calculator thanks to arithmetic expansion, but this it only works with simple integers (We need tools like bc,... for float numbers). In spite of its limitation, it might useful in some scenarios, it also supports many popular operators. Let's say: , -, *, %, /, +,...
Arithmetic expansion follows below pattern:
$((expression))
phuong@Arch ~$ echo This month has $((4*7)) days
This month has 28 days
As an alternative to double parentheses, we can also use single parentheses to nest sub expansion. Such as:
phuong@Arch ~$ echo This year has $((((4*7)*12) + $((3 * 9)))) days
This year has 363 days
Tilde expansion
When we using ~ as the beginning of the string in terminal, it expands to home directory of named user.
Let's say, I am using an account on the cloud where there are many users there and I don't known where their home directory are, I just know usernames of them. That is where it come into play
phuong@devcloud1:~$ echo ~wingzero
/home/wingzero
phuong@devcloud1:~$ echo ~phuong
/home/phuong
Brace expansion
The brace expression itself may contain either a comma-separated list of strings or a range of integers or single characters.
For the comma-separated example:
phuong@Arch ~/kitchen/ips$ echo A{A,B}
AA AB
Similarly, the range of single characters
phuong@Arch ~$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
From my point of view, this kind of expansion is one of the most interesting expansion mechanisms as it is pretty useful in specified situations. Suppose that I want to create each folder for each network IP in the subnet range. Pretending that 192.168.43.1 -> 192.168.43.13. So I confess the way that I would do 4 months a go is that: mkdir .... But this approach is prone to typo errors. So that is where the brace expansion comes and saves.
phuong@Arch ~/kitchen/ips$ mkdir 192.168.43.{1..13}
phuong@Arch ~/kitchen/ips$ ls
192.168.43.1 192.168.43.12 192.168.43.3 192.168.43.6 192.168.43.9
192.168.43.10 192.168.43.13 192.168.43.4 192.168.43.7
192.168.43.11 192.168.43.2 192.168.43.5 192.168.43.8
It is true as you may guest that the brace expansion can also be nested.
phuong@Arch ~/kitchen/ips$ echo A{a{1,2},b{9,10}}
Aa1 Aa2 Ab9 Ab10
Command substitution
Beside above expansion mechanisms, the command substitution maybe the one that you might feel familiar with because its widely used in scripting. The command substitution allows us to use the output of a command as an expansion. For instance:
phuong@Arch ~/kitchen$ ls -l $(which go)
lrwxrwxrwx 1 root root 18 Nov 2 17:25 /usr/bin/go -> /usr/lib/go/bin/go
Here is another example
phuong@Arch ~$ echo Hello $(w=world; echo $w)
Hello world
Parameter substitution
This kind of substitution is intensively used in scripting than directly in command line. For simple example, you want to print USER variable, you just add dollar symbol($) at the beginning.
phuong@Arch ~$ echo $USER
phuong
Note that if you mistyping the name of a variable, the expansion will still take place, but it results into empty string.
For the sake of simplicity, I will not go into detail. Because this powerful mechanism deserve another post for it. The scope of this post is to point out some popular expansion mechanisms and how to control them.
Quoting
So we have seen how powerful the expansion in the Shell is, that is why it could lead to some unexpected behaviors unless we know how to control it.
Now we come to interesting part.
Consider two samples below:
Example 1:
phuong@Arch ~$ echo Hello I am a string
Hello I am a string
Example 2:
phuong@Arch ~$ echo It costs me $200
It costs me 00
Feel familiar ah?
In the first example, the word splitting ditches extra white spaces from the echo command's list of arguments. In the second example, the $2 is expanded by parameter expansion mechanism, that is on purpose, the problem is $2 variable is undefined so the Shell suppress unexpected expansion which finally leads to how it behaves. Let's figure out how to fix this.
Double Quotes
If you put special characters into double quotes will lost their special meanings to the Shell, but there are 3 exceptions for that, namely dollar sign, backslash and back tick.
This is pretty useful in specified scenario. To show you what I mean, imagine that one of your friends send you a homework file named "homework part1.doc". Well, that might drive you crazy.
Which leads you to below situation
phuong@Arch ~/Downloads$ cat homework part1.doc
cat: homework: No such file or directory
cat: part1.doc: No such file or directory
The double quotes comes to save you.
phuong@Arch ~/Downloads$ mv "homework part1.doc" homework_part1.doc
phuong@Arch ~/Downloads$ cat homework_part1.doc
This is my stupid homework part, now is your turn
Once double quotes are added , our command contains a command followed by a single argument. That is how we preserve our extra white spaces, because world splitting mentioned above only take effect for multiple parameters. With everything putted in double quotes, it behave like only one parameter following the echo command
phuong@Arch ~/Downloads$ echo Hello I am a string
Hello I am a string
phuong@Arch ~/Downloads$ echo "Hello I am a string"
Hello I am a string
Single quote
If you simply need to suppress all special characters, just use single quote
phuong@Arch ~/Downloads$ echo 'cat ~/Download/.doc {a,b} $(echo bar) $((3+2)) $USER'
cat ~/Download/.doc {a,b} $(echo bar) $((3+2)) $USER
Escaping characters
Sometimes we want to quote only a single character, to do this, just preceding that character with a backslash \
phuong@Arch ~/Downloads$ echo "You own me $30"
You own me 0
phuong@Arch ~/Downloads$ echo "You own me \$30"
You own me $30
It's common to selectively prevent the expansion of a filename.
Conclusion
To summarize, the expansion mechanisms in Unix Shell is great and powerful. If we know how to control it, I believe those features surely enhance your productivity. If you have any question, feel free to drop a comment below, that would be nice to hear from you.
Thanks for your time.
References
- The Linux Command Line - William E. Shotts JR.
Top comments (0)