Balthazar Rouberol

Posted on Apr 24, 2020 • Edited on Apr 26, 2020

Shell Productivity Tips and Tricks

#linux #tutorial #beginners #bash

Initially published on my blog.

This article is part of a self-published book project by Balthazar Rouberol and Etienne Brodu, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!

If you are interested in the project, we invite you to join the mailing list!

Tab completion
Keyboard shortcuts
Navigating through history
Shell expansions
Real-life examples
Summary
Going further

Shell productivity tips

I estimate that I spend around 50% of my day working in my text editor
and my terminal. Any way I can get more productive in these environments
has a direct and measurable impact on my daily productivity as a whole.

If you spend a good chunk of your day repeatedly hitting the left and
right arrow keys to navigate in long commands or correct typos, or
hitting the up or down arrow keys to navigate your command history, this
chapter should help you get more done quicker. We will cover some shell
features you can leverage to make your shell do more of the work for
you.

On a personal level, I probably use some of these up to 30 times a day,
sometimes even without thinking about it, and it gives me a real sense
of ownership of my tool.

In the immortal words of Kimberly “Sweet Brown” Wilkins:

Ain't nobody got time for that.

Tab completion

When you are typing in your shell, I suggest you treat the
Tab key as a superpower. Indeed, the same way your phone
keyboard can autocomplete words for you, so can your shell. It can
suggest completions of command names and even command arguments or
options! This works by pressing Tab (twice for bash and
once for zsh).

Note: One of the reasons zsh might be favored over bash is its more
powerful auto-completion system, giving more results out-of-the-box and
allowing you to navigate through the auto-completion options.

Here is an example of bash auto-completing a command name:

$ mkd<Tab>
mkdep  mkdir

Here is an example of bash auto-completing a command argument:

$ man mkd<Tab>
mkdir         mkdirat       mkdtemp       mkdtempat_np

And finally, an example of bash auto-completing a command option:

$ python -<Tab>
-    -3   -B   -E   -O   -OO  -Q   -R   -S   -V   -W
-b   -c   -d   -h   -i   -m   -s   -t   -u   -v   -x

I suggest you get used to using auto-completion as much as possible. It
can save you keystrokes, as well as make you discover command options
you didn't know about.

Pro-tip: if you are using bash, you can get install the
bash-completion¹ package (using your system package-manager) in
order to enable auto-completion for a wide variety of commands that do
not support it out-of-the-box.

Keyboard shortcuts

The shell uses a library called readline² to provide you with many
keyboard shortcuts to navigate, edit, cut, paste, search, etc, in the
command line. Mastering these will help to dramatically increase your
efficiency, instead of copying and pasting with your mouse, and
navigating the command with the ↑ and ↓ arrow
keys.

The default shortcuts are inspired by the emacs³ terminal-based
text editor. If you are already familiar with it, a lot of the default
readline shortcuts might feel familiar. emacs isn't the only famous
text editor in the history of computers though: another one, dating back
from 1976, is vi.⁴ vi and emacs are designed in two very
different ways, and have two very different logics. It is possible that
one might “click” more than the other for you. If you happen to be
familiar with the vi editor and are accustomed to its navigation
system, you can replicate it in your shell as well by adding set -o vi
in your shell configuration file. If you are using zsh with the Oh My
Zsh framework that we introduced in the previous chapter, you can also
use the vi-mode plugin to do this.

The advantage of using the same navigation logic and shortcuts in your
text editor and your terminal is that is blurs the line between both,
and brings consistency to your terminal environment. If you have no clue
how emacs or vi work though, I would probably suggest you don't
worry about all this for now and experiment with the default terminal
shortcuts.

Navigating the current line

The following navigation shortcuts allow you to move quickly your cursor
in the current command saving you from relying solely on the
→ and ← arrows.

Navigation	Shortcut
Go to beginning of line	`Ctrl` - `A`
Go to end of line	`Ctrl` - `E`
Go to next word	`Alt` - `F`
Go to previous word	`Alt` - `B`
Toggle your cursor between its current position and the beginning of line	`Ctrl` - `X` - `X`

If you however prefer using the vi navigation system, you will first
need to type Esc to switch from the Insertion mode to an
emulation of vi's normal mode, in which you can navigate in your
text using the following shortcuts:

Navigation	Shortcut
Go to beginning of line	`^`
Go to end of line	`$`
Go to next word	`w`
Go to previous word	`b`
Move to the end of the previous word	`e`

You can go back to editing your command line by hitting the i key.

Deleting and editing text

These shortcuts allow you to quickly edit the current command more
efficiently than by just using the Delete key.

Edition	Shortcut
Delete current character	`Ctrl` - `D`
Delete previous word	`Ctrl` - `W`
Delete next word	`Alt` - `D`
Edit the current command in your text editor	`Ctrl` - `X` `Ctrl` - `E`
Undo previous action(s)	`Ctrl` - `-`

The equivalent vi-style shortcuts are:

Edition	Shortcut
Replace current character by another (ex: e)	`r` - `e`
Delete current character	`x`
Delete previous word	`d` - `b`
Delete next word	`d` - `w`
Edit the current command in your text editor	`v`
Undo previous action(s)	`u`

Cutting and pasting

The shell provides you with shortcuts to cut and paste commands quickly
without using your mouse.

Action	Shortcut
Cut current word before the cursor	`Ctrl` - `W`
Cut from cursor to end of line	`Ctrl` - `K`
Cut from cursor to start of line	`Ctrl` - `U`
Paste the cut buffer at current position	`Ctrl` - `Y`

The equivalent vi-style shortcuts are:

Action	Shortcut
Cut current word before the cursor	`d` - `w`
Cut from cursor to end of line	`d` - `$`
Cut from cursor to start of line	`d` - `^`
Paste the cut buffer at current position	`p`

Controlling the terminal

Finally, these shortcuts will let you interact with the terminal itself.

Action	Shortcut	Equivalent command
Clear the terminal screen	`Ctrl` - `L`	`clear`
Close the terminal screen	`Ctrl` - `D`	`exit`
Send current command to the background.	`Ctrl` - `Z`

Even mastering some of these shortcuts should make you immensely more
productive at typing commands and navigating command-line interfaces. I
suggest you take time to experiment until you feel more accustomed with
them. I can guarantee that you will feel the productivity boost!

A unified command-line editing experience

These shortcuts do not just work in your shell, but in any application
using the readline library to allow the user to type and edit
commands. Learning these shortcuts will thus make you productive in all
types of command lines that you might encounter in your career, such as
python, irb, sqlite3, etc.

To make sure you get a smooth and homogeneous editing experience in all
command lines you use in your system, you can set your preferred mode in
the readline configuration file itself.

$ cat ~/.inputrc
set editing-mode vi  # or emacs

Navigating through history

If you find yourself typing a certain command times and times again, you
should probably be aware of how to navigate and search your shell
history, in order to save time and keystrokes.

While the obvious way to re-execute a previous command might seem to
just bash on the ↑ key until you find the command you want,
there are faster and smarter ways to accomplish this.

Searching the history

A very useful and time-saving trick is searching for a command into your
shell history instead of re-typing it from scratch. You can search your
command history by typing Ctrl - R which opens a
reverse-i-search (backwards search) prompt, in which you can search
for previously executed command containing a given search pattern.

Type Ctrl - R to navigate through the results,
until you find the one you were looking for and type the
Enter key to execute it.

$ <Ctrl-R>
(reverse-i-search): echo <Ctrl-R> <Enter>
$ echo "hello world"
hello world

If you want to stop the search, either hit
Ctrl - C or Ctrl - G to be
sent back into the regular shell prompt.

History search works by looking into the shell history file
(~/.bash_history for bash and ~/.zsh_history for zsh by
default). Every time you execute a command, it will be added to your
shell history file (with a maximum number of retained commands defined
by the HISTSIZE environment variable).

Note: The location of your shell history file can be configured by setting the
HISTFILE environment variable.

Rewriting history

If you want to remove a sensitive command from your history, you can
simply edit your $HISTFILE history file and remove it.

$ secret-command --password 1234qwerty  # oh no! that should not be in my history!
$ grep secret-command $HISTFILE
secret-command --password 1234qwerty
$ sed -i '/secret-command/d' $HISTFILE  # deletion of history line containing 'secret-command'
$ grep secret-command $HISTFILE
$ # it's not in history anymore

You can also use the history built-in command to display your whole
history

$ history | tail -n 5
  496  mkdir test
  497  secret-command --password 1234qwerty
  498  cd
  499  man history
  500  history | tail -n 5

Each history line is prefixed by its index in the history. You can then
use history -d <index> to remove the associated line from history.

$ history -d 497
$ history | tail -n 7
  496  mkdir test
  497  cd
  498  man history
  499  history | tail -n 5
  500  history -d 497
  501  history | tail -n 7

Note: This only works with bash, not zsh.

Avoiding history

There is a trick you can use if you want to fly under the radar and
never have a command recorded in history in the first place. Simply
prefix your command by a space.

Note: If you are using zsh, you need to add setopt HIST_IGNORE_SPACE in
your ~/.zshrc to make sure that behavior is enabled.

$  secret-command --password 1234qwerty  # notice the space at the start of the command!
$ history | tail -n 2
  502  history | tail -n 7
  503  history | tail -n 2

Shell expansions

The shell can perform expansions, meaning it can replace portions of the
command before executing it. Relying on expansions allows you to type
less and rely on the shell itself to do the heavy lifting. While there
are multiple types of expansions, we will only cover 5:

history expansion: quickly access previous commands and arguments from history
tilde expansion: replace the ~ path prefix
pathname expansion: expand a path pattern into a list of files
braces expansion: expand a pattern between braces into a longer sequence
command expansion: replace a sub-command by its output

Expansions are extremely powerful. When used right, an expansion can
literally save you from writing a script.

Note: As we only over what we think are the most useful expansions and
shortcuts, feel free to refer to the bash manual, section EXPANSION
if you want to see the full list.

History expansion

Your shell has multiple tricks in its sleeve to allow you to quickly
reference previous commands or arguments in history with a minimum of
keystrokes. While this section only provides you with what we feel are
the most useful of them, feel free to go to the HISTORY EXPANSION
section of the bash manual.

Event designators

An Event designator is a reference to a command line entry in the
history list. It allows you to quickly refer to a previous command
without having to re-type it.

`!-n`

!-n refers to the nth latest command: !-1 refers to the latest
command, !-2 to the command before that, etc.

$ echo "hello world!"
hello world!
$ cd
$ !-2  # !-1 is "cd" and !-2 is 'echo "hello world!"'
$ echo "hello world"
hello world

!! is a shortcut for !-1, aka the latest command.

$ echo "hello world!"
hello world!
$ !!
$ echo "hello world"
hello world

Note: !! is oftentimes used in conjunction with sudo, to re-execute the
previous command with superuser privileges when it failed, due to a lack
of permission.

$ vim /etc/myfile
vim: /etc/myfile: Permission denied
$ sudo !!
$ sudo vim /etc/myfile

`^string1^string2`

^string1^string2 is used to repeat the previous command in which
string1 is replaced by string2.

$ cat ./myfile
Just a file full of junk
$ ^cat^rm
$ rm ./myfile

I personally use and abuse of this technique when I'm about to
irremediably delete some resources (files, folders, containers, etc),
and I want to make sure I'm about to delete the right things by
listing these resources first. If you are familiar with SQL queries, it
is the equivalent of executing a SELECT query before changing the
SELECT to DELETE to make sure you're not going to delete more than
you wanted to.

Word designators

Word designators are used to select desired words from a previous
command (by default, the latest). They can be very useful when you want
to type a new command that uses arguments previously typed in a previous
command.

`!^`

!^ maps to the first argument of your latest command.

$ touch first.txt second.txt last.txt
$ vim !^
$ vim first.txt

`!$`

!$ maps to the last argument of your latest command.

$ touch first.txt second.txt last.txt
$ vim !$
$ vim last.txt

Combining event and word designators

You can even combine event and word designators in more complex shapes
by using the following syntax

[EVENT DESIGNATOR]:[WORD DESIGNATOR]

For example, you could use the !! event designator to select the last
command, and the 2 word designator to select the second argument.

$ touch first.txt second.txt last.txt
$ vim !!:2
$ vim second.txt

Tilde expansion

For each unquoted word starting with ~ in the command, all characters
preceding a forward slash (/) will be considered a tilde prefix.
Depending on its actual value, the tilde prefix can be expanded several
ways, although the simple ~ is probably its most common use.

Tilde prefix	Expansion
`~`	Your home directory
`~+`	Your current working directory
`~-`	Your previous working directory

Example

$ ls ~
Android                code       Downloads              Music
AndroidStudioProjects  Desktop    Dropbox                Pictures
bin                    Documents  Firefox_wallpaper.png  Videos

This lists the content of your home directory, and is the equivalent to
ls $HOME. You can combine the tilde with a suffix to compose an
absolute path to some file or folder in your home directory.

$ cd ~/code
$ pwd
/home/br/code

Pathname expansion

Pathname expansions allow you to write an short path pattern and have it
expanded in a list of files and directories, saving you from tedious
copy-pastes or a possibly long (and error-prone) command writing.

`*`

The glob, or wildcard * character matches any string. It allows
you to give a pattern to the shell, that it will then expand to all
files and directories matching the pattern. The wildcard can be prefixed
or suffixed, which will further specify our pattern. For example,
*.jpg matches all files ending with the .jpg extension, and
README.* matches all files named README whatever their extension.

Let us consider the following file and directory structure.

$ tree
.
|-- pic1.jpg
|-- pic2.jpg
|-- pic3.jpg
|-- pic4.jpg
\__ pics
|   |-- pic5.jpg
|   |-- pic6.jpg
|   \__ pic7.jpg
\__ sounds
    \__sound1.mp3

2 directory, 8 files

We want to move all jpg files into our pics directory. Instead of
running 4 different mv commands or manually typing a long mv
command, we can run just one using a pathname expansion.

$ mv *.jpg pics
$ tree
.
\__ pics
|   |-- pic1.jpg
|   |-- pic2.jpg
|   |-- pic3.jpg
|   |-- pic4.jpg
|   |-- pic5.jpg
|   |-- pic6.jpg
|   \__ pic7.jpg
\__ sounds
    \__sound1.mp3

2 directory, 8 files

*.jpg was expanded to all files ending with .jpg, causing the shell
to actually run mv pic1.jpg pic2.jpg pic3.jpg pic4.jpg pics, causing
all 4 jpg files to be moved to the pics directory in a single
command.

Note: We could have executed the following commands for the same result:

mv pic* pics would have moved all files with name starting by pic to the pics directory
mv pic*.jpg pics would have moved all files with name starting by pic and ending with .jpg to the pics directory

You can use * several times within the same pattern. For example
ls */* will list all files and directories located in a subdirectory.

$ ls */*
sounds/sound1.mp3   pics/pic2.jpg       pics/pic4.jpg       pics/pic6.jpg
pics/pic1.jpg       pics/pic3.jpg       pics/pic5.jpg       pics/pic7.jpg

Like in our second example, we can also use */*.jpg to list all jpg
files located in a subdirectory.

$ ls */*.jpg
pics/pic1.jpg   pics/pic3.jpg   pics/pic5.jpg  pics/pic7.jpg
pics/pic2.jpg   pics/pic4.jpg   pics/pic6.jpg

`**`

** is expanded to all files and directories in the children
directories, with a depth limit of 1.

$ touch README.txt
$ mkdir sounds/lyrics
$ touch sounds/lyrics/sound1.txt
$ tree
.
|-- README.txt
\__ pics
|   |-- pic1.jpg
|   |-- pic2.jpg
|   |-- pic3.jpg
|   |-- pic4.jpg
|   |-- pic5.jpg
|   |-- pic6.jpg
|   \__ pic7.jpg
\__ sounds
    \__ lyrics
    |   \__sound1.txt
    \__sound1.mp3

3 directories, 10 files
$ ls **
README.txt

pics:
pic1.jpg pic2.jpg pic3.jpg pic4.jpg pic5.jpg pic6.jpg pic7.jpg

sounds:
lyrics     sounds.mp3

ls ** was expanded into ls README.txt pics/ sounds/, which does not
include the content of sounds/lyrics because of the depth limit of 1.

`**/`

**/ is expanded into all directories and subdirectories with a depth
limit of 1 starting from our first directory.

$ tree
.
|-- README.txt
\__ pics
|   |-- pic1.jpg
|   |-- pic2.jpg
|   |-- pic3.jpg
|   |-- pic4.jpg
|   |-- pic5.jpg
|   |-- pic6.jpg
|   \__ pic7.jpg
\__ sounds
    \__ lyrics
    |   \__sound1.txt
    \__sound1.mp3


3 directories, 10 files
$ ls **/
pics/:
pic1.jpg pic2.jpg pic3.jpg pic4.jpg pic5.jpg pic6.jpg pic7.jpg

sounds/:
lyrics     sounds.mp3

sounds/lyrics/:
sound1.txt

ls **/ was expanded into ls sounds/ sounds/lyrics pics/. It thus
listed all files located in our subdirectories.

Brace expansion

A brace expansion is a mechanism by which the shell can generate
multiple strings based on a sequence of tokens defined within curly
braces. The brace expansion pattern can be preceded by an optional
preamble and followed by an optional postscript.

$ mkdir ~/test/{pics,sounds,sprites}
$ ls ~/test
pics  sounds  sprites

~/test/{pics,sounds,sprites} was expanded into
~/test/pics ~/test/sounds ~/test/sprites causing the shell to execute
mkdir ~/test/pics ~/test/sounds ~/test/sprites (which will be expanded
further into
mkdir /home/br/test/pics /home/br/test/sounds /home/br/test/sprites by
a tilde expansion).

We could have done the same thing by factoring the final s of each
token into a postscript.

$ mkdir ~/test/{pic,sound,sprite}s

A brace expansion can also have a sequence pattern {x..y[..incr]}
where x and y are either an integer or a single character, and
incr is an optional increment value.

$ touch ~/test/sounds/noise-{1..5}.mp3
$ ls ~/test/sounds
noise-1.mp3 noise-2.mp3 noise-3.mp3 noise-4.mp3 noise-5.mp3

The default increment is 1 if the sequence end is greater than its
start, and -1 otherwise. However, we could specify a custom increment
value if we want.

$ touch ~/test/pics/pic{1..10..2}.jpg
$ ls ~/test/pics
pic1.jpg pic3.jpg pic5.jpg pic7.jpg pic9.jpg

Command expansion

Your shell can replace a command surrounded by $() with its output.

I personally like use to commands expansions can to iterate over a
command's result, or by combining it with a heredoc redirection:

$ cat <<EOF > aboutme
My name is $(whoami)
and I live in $HOME
EOF
$ cat aboutme
My name is br
and I live in /home/br

Real-life examples

Moving a pattern of files contained in directories and subdirectories

What is really powerful with these expansions is that, like almost
everything in the shell, they can be combined. The following example
combines a pathname expansion, a brace expansion and a tilde expansion.

$ tree
.
|-- README.txt
\__ pics
|   |-- pic1.jpg
|   |-- pic2.jpg
|   |-- pic3.jpg
|   |-- pic4.jpg
|   |-- pic5.jpg
|   |-- pic6.jpg
|   \__ pic7.jpg
\__ sounds
    \__ lyrics
    |   \__sound1.txt
    \__sound1.mp3
$ mv **/*.{jpg,mp3} ~/assets/
$ tree
|-- README.txt
\__ pics
\__ sounds
    \__ lyrics
        \__sound1.txt
$ ls ~/assets
README.txt pic1.jpg   pic2.jpg   pic3.jpg   pic4.jpg   pic5.jpg   pic6.jpg   pic7.jpg   sound1.txt

Using these expansions, we were able to move all jpg and mp3 files
located in directories and subdirectories to the assets directory
located in your home directory, in exactly 27 characters!

Renaming multiple directories

We could use a for loop, pathname expansion and a command expansion to
rename all directories contained in the bcurrent directory to their
uppercase equivalent.

$ for dir in */; do
    mv "$dir" "$(echo $dir | tr '[:lower:]' '[:upper:]')"
  done

Let's decompose that command into its different steps:

the */ glob pattern is expanded over the list of directories, on which we iterate via a for loop
we execute echo $dir | tr '[:lower:]' '[:upper:]', which will convert the current directory name to uppercase
the $(echo $dir | tr '[:lower:]' '[:upper:]') command is expanded into the uppercase directory name
the directory is renamed into an uppercase name
the for loop iterates over the next directory name
we move on to the next directory and repeat the previous steps for each of them

Note: Iterating over paths with a for loop is brittle as it breaks if a path
contains a space. We will later see how to properly do it using the
find command.

Summary

Your shell has so many productivity tricks and shortcuts up its sleeve
it can be a little bit daunting. I suggest you don't try to learn them
all at once, but really just experiment with them and see what feels
natural. Even mastering some of them will make you more productive!

What if there is an action you find useful but you just don't like the
keyboard shortcut? Luckily for you, the next chapter will dive into how
to personalize and customize your shell.

Going further

5.1: Create a directory. Use a bash expansion to move into that
directory without typing its name a second time.

5.2: Print your 4th last command typed into your terminal without
re-typing it.

5.3: Create the following empty files README.txt,
requirements.txt and TODO.txt in a single command, without typing
.txt more than once.

5.4: Delete all the files created in the last question without
typing .txt more than once.

5.5: Create the following directory tree in a single command.

files
|-- 1
|   |-- 1a
|   |-- 1b
|   |-- 1c
|   |-- 2a
|   |-- 2b
|   |-- 2c
|   |-- 3a
|   |-- 3b
|   \-- 3c
|-- 2
|   |-- 1a
|   |-- 1b
|   |-- 1c
|   |-- 2a
|   |-- 2b
|   |-- 2c
|   |-- 3a
|   |-- 3b
|   \-- 3c
\-- 3
    |-- 1a
    |-- 1b
    |-- 1c
    |-- 2a
    |-- 2b
    |-- 2c
    |-- 3a
    |-- 3b
    \-- 3c

5.6: Remove all subdirectories starting with 3 created in the
previous command, while keeping the top 3 directory.

5.7: Re-execute the command from exercise 5.3 by looking backwards
into your shell history.

Essential Tools and Practices for the Aspiring Software Developer is a self-published book project by Balthazar Rouberol and Etienne Brodu, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!

The book will help you set up a productive development environment and get acquainted with tools and practices that, along with your programming languages of choice, will go a long way in helping you grow as a software developer. It will cover subjects such as mastering the terminal, configuring and getting productive in a shell, the basics of code versioning with git, SQL basics, tools such as Make, jq and regular expressions, networking basics as well as software engineering and collaboration best practices.