DEV Community

Cover image for How the Unix Shell sees the world (Expansion, Quoting)
Phuong Hau
Phuong Hau

Posted on

How the Unix Shell sees the world (Expansion, Quoting)

TL;DR

It was me using Unix terminal every single day, typing hundred lines of terminal command, but sometimes, I encounter some unexpected behaviors at the moment I press Enter button after a command. It's pretty frustrated at some points, it turns out that the way the Shell sees the world is different than mine. I read some books (at least some chapters) about how the Shell acts upon a command, I find out it's really interesting so that I share with you today. Hope this helpful.

Because this is pretty straightforward. With only echo command, we can examine several interesting features, the "magic" of the Shell

The Expansion

Every time we press enter after a command, it get transformed by the Shell under the hood before get carried out. The process that makes it happen is call expansion. The expansion is that when you type in something, it get expanded to another thing before the shell acts upon it. For example: The * means a lot to the Shell. To illustrate this, I will use the echo command, as you recall that echo is a simple Shell built-in function that directs our typing to standard output

phuong@Arch ~$ echo Hello world
Hello world

So the * wildcard means: Match any characters in file-name. The fact that the * get transformed into other characters before being carried out by the shell so that the echo command never see the *

phuong@Arch ~/kitchen$ echo *
bash  device  devops  gateway  go  hook  iuh  js  linux  ml  python

Filename expansion

In fact, this behavior carried out by a mechanism, which is popularly called path name expansion. As we know, file names begin with period character (ex: .git) is hidden. The path name extension respect this rule. An expansion does not reveal hidden files.

phuong@Arch ~$ echo D*
Desktop Documents Downloads

But we can show all hidden files by following command:

phuong@Arch ~$ echo .*
. .. .3T .ansible .anydesk .bash_history .bash_logout .bash_profile .bashrc .bashrc.backup .boto .cache .config .dbshell .designer .docker

Arithmetic expansion

We are also able to use our terminal command as a calculator thanks to arithmetic expansion, but this it only works with simple integers (We need tools like bc,... for float numbers). In spite of its limitation, it might useful in some scenarios, it also supports many popular operators. Let's say: , -, *, %, /, +,...
Arithmetic expansion follows below pattern:

$((expression))

phuong@Arch ~$ echo This month has $((4*7)) days
This month has 28 days

As an alternative to double parentheses, we can also use single parentheses to nest sub expansion. Such as:

phuong@Arch ~$ echo This year has $((((4*7)*12) + $((3 * 9)))) days
This year has 363 days

Tilde expansion

When we using ~ as the beginning of the string in terminal, it expands to home directory of named user.
Let's say, I am using an account on the cloud where there are many users there and I don't known where their home directory are, I just know usernames of them. That is where it come into play

phuong@devcloud1:~$ echo ~wingzero
/home/wingzero
phuong@devcloud1:~$ echo ~phuong
/home/phuong

Brace expansion

The brace expression itself may contain either a comma-separated list of strings or a range of integers or single characters.

For the comma-separated example:

phuong@Arch ~/kitchen/ips$ echo A{A,B}
AA AB

Similarly, the range of single characters

phuong@Arch ~$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

From my point of view, this kind of expansion is one of the most interesting expansion mechanisms as it is pretty useful in specified situations. Suppose that I want to create each folder for each network IP in the subnet range. Pretending that 192.168.43.1 -> 192.168.43.13. So I confess the way that I would do 4 months a go is that: mkdir .... But this approach is prone to typo errors. So that is where the brace expansion comes and saves.

phuong@Arch ~/kitchen/ips$ mkdir 192.168.43.{1..13}
phuong@Arch ~/kitchen/ips$ ls
192.168.43.1   192.168.43.12  192.168.43.3  192.168.43.6  192.168.43.9
192.168.43.10  192.168.43.13  192.168.43.4  192.168.43.7
192.168.43.11  192.168.43.2   192.168.43.5  192.168.43.8

It is true as you may guest that the brace expansion can also be nested.

phuong@Arch ~/kitchen/ips$ echo A{a{1,2},b{9,10}}
Aa1 Aa2 Ab9 Ab10

Command substitution

Beside above expansion mechanisms, the command substitution maybe the one that you might feel familiar with because its widely used in scripting. The command substitution allows us to use the output of a command as an expansion. For instance:

phuong@Arch ~/kitchen$ ls -l $(which go)
lrwxrwxrwx 1 root root 18 Nov  2 17:25 /usr/bin/go -> /usr/lib/go/bin/go

Here is another example

phuong@Arch ~$ echo Hello $(w=world; echo $w)
Hello world

Parameter substitution

This kind of substitution is intensively used in scripting than directly in command line. For simple example, you want to print USER variable, you just add dollar symbol($) at the beginning.

phuong@Arch ~$ echo $USER
phuong

Note that if you mistyping the name of a variable, the expansion will still take place, but it results into empty string.

For the sake of simplicity, I will not go into detail. Because this powerful mechanism deserve another post for it. The scope of this post is to point out some popular expansion mechanisms and how to control them.

Quoting

So we have seen how powerful the expansion in the Shell is, that is why it could lead to some unexpected behaviors unless we know how to control it.
Now we come to interesting part.

Consider two samples below:
Example 1:

phuong@Arch ~$ echo Hello I am a          string
Hello I am a string

Example 2:

phuong@Arch ~$ echo It costs me $200
It costs me 00

Feel familiar ah?
Alt Text

In the first example, the word splitting ditches extra white spaces from the echo command's list of arguments. In the second example, the $2 is expanded by parameter expansion mechanism, that is on purpose, the problem is $2 variable is undefined so the Shell suppress unexpected expansion which finally leads to how it behaves. Let's figure out how to fix this.

Double Quotes

If you put special characters into double quotes will lost their special meanings to the Shell, but there are 3 exceptions for that, namely dollar sign, backslash and back tick.

This is pretty useful in specified scenario. To show you what I mean, imagine that one of your friends send you a homework file named "homework part1.doc". Well, that might drive you crazy.

Alt Text

Which leads you to below situation

phuong@Arch ~/Downloads$ cat homework part1.doc 
cat: homework: No such file or directory
cat: part1.doc: No such file or directory

The double quotes comes to save you.

phuong@Arch ~/Downloads$ mv "homework part1.doc" homework_part1.doc 
phuong@Arch ~/Downloads$ cat homework_part1.doc 
This is my stupid homework part, now is your turn

Once double quotes are added , our command contains a command followed by a single argument. That is how we preserve our extra white spaces, because world splitting mentioned above only take effect for multiple parameters. With everything putted in double quotes, it behave like only one parameter following the echo command

phuong@Arch ~/Downloads$ echo Hello I am a          string
Hello I am a string
phuong@Arch ~/Downloads$ echo "Hello I am a          string"
Hello I am a          string

Single quote

If you simply need to suppress all special characters, just use single quote

phuong@Arch ~/Downloads$ echo 'cat ~/Download/.doc {a,b} $(echo bar) $((3+2)) $USER'
cat ~/Download/.doc {a,b} $(echo bar) $((3+2)) $USER

Escaping characters

Sometimes we want to quote only a single character, to do this, just preceding that character with a backslash \

phuong@Arch ~/Downloads$ echo "You own me $30"
You own me 0
phuong@Arch ~/Downloads$ echo "You own me \$30"
You own me $30

It's common to selectively prevent the expansion of a filename.

Conclusion

To summarize, the expansion mechanisms in Unix Shell is great and powerful. If we know how to control it, I believe those features surely enhance your productivity. If you have any question, feel free to drop a comment below, that would be nice to hear from you.

Thanks for your time.

References

  • The Linux Command Line - William E. Shotts JR.

Top comments (0)