Bash scripting for everyday actions

This article is the first in a series of posts about automating everyday actions. We’ll start with Bash shell scripting, which allows you to write scripts to automate dull, repetitive tasks. You can also find it on my blog.

The great advantage of Bash shell scripting compared to writing a full CLI tool to do what we need is that it is very easy to do, especially for those familiar with the Bash shell already, but it is only feasible to build Bash scripts that can be used for a very limited range of applications.

The problem to solve

Let’s identify a problem anyone can have at some point in life: editing some files according to the folder they’re stored in.

A relatable, if not too specific, story of someone who needs to learn Bash scripting

Let’s say you spend your summer vacation in Italy because you love when the weather is extremely hot, have heard there are beautiful cities, ancient churches, Roman Empire ruins and you want to taste typical Italian food in the place where it all started. However much time you decide to spend, Italy isn’t too big, there are fast trains, and you decide to visit several cities. Just like any tourist, you take pictures everywhere and, by the time you’re back home, you organize your pictures in folders.

The current situation

At this point you can at least figure out where you were when each picture was taken from the date on which it was taken from the hotel/rail/bus/domestic plane reservations you must have because rental cars with automatic transmissions are very expensive and less than 20% of Americans can drive a manual. You arrange your pictures into subfolders: you have folders for each place you visited and you put those folders in three folders called North, Center and South because it’s nice to be able to know that when you’ll look at the pictures in the future.

The problem

That in particular is a problem: looking at the pictures will now require you to browse and select pictures in each subfolder manually, and that’s especially painful on some less advanced I/O devices like TV remotes.

It would be ideal to be able to take all of those neatly organized pictures you have, put them all in one folder with a watermark telling you where you’ve taken them so it won’t look like you would have been better off just downloading some pictures from the Web when you show them to your friends (or kids) 10 years after you’ve been there.

Doing that by hand will require another 10 years and your friends or kids will already have visited all of the places you’ve visited fifty times by the time you are done. Fortunately, there is a way to make that quick and easy, and that is by embracing Bash shell scripting.

Solving the problem

Solving this problem requires two different levels of difficulty depending on what operating system you’re running. If you’re running most GNU/Linux distributions or macOS, Bash is your default shell, so it’s already installed and you can go on with the next section without having to install anything else. If you’re running any other Unix or Unix-like operating system that doesn’t install Bash by default, installing Bash is generally very easy and you can find specific instructions online in the very unlikely case you don’t already know how to install a package or port called bash on your OS of choice.

Running Bash on Windows

If you’re running Windows, you’ll have to install the WSL (Windows Subsystem for Linux) by installing one of the packages available in the Windows Store that include Bash. You just need to open the Windows Store, search for Ubuntu (for example) and install it. When starting for the first time, it will prompt you to enter a username and password you’ll need to remember. The password won’t even be shown to you in the form of asterisks, in case you’re confused by the fact it seems like you’re not actually typing anything in.

After that, you’ll be able to access Bash in any directory on your PC by running the

bash
~~~

command. You can exit the Bash shell by running the

exit

command.

More details about how to use it will be provided in the rest of the article.
## Bash scripting: the basics

At its simplest, a Bash script is just a list of shell commands separated by newlines or concatenated together using pipes or some of the many script-oriented constructs Bash includes.

### A quick introduction to Bash and the Unix command line

This section will be a very quick introduction to the usage of the Bash shell and the Unix command line in general, given that most shells are very similar when it comes to the most basic tasks. There are plenty of books available online that will teach you how to use it, many of which are aimed directly at Linux users, but they also apply to other Unix-like operating systems and to the Windows Subsystem for Linux.

The first thing to understand about any command line interface is that it’s like using a file manager: at any point you’re operating in a specific directory, called the *working directory*. Running the command (by typing it in and then pressing enter){% raw %}

pwd

will return the current working directory.

The directory structure of Unix-like operating systems is a tree that branches out from the single *root* directory, the path of which is simply the character */*. Other directories are chained after that separated by forward slashes. For example, the *home* directory at the root of the tree is found at path{% raw %}

/home

and a hypothetical *user* directory inside that would be at path{% raw %}

/home/user

Paths can be also expressed as *relative* paths, based on the current CLI working directory. The current working directory is expressed as *./* and the parent directory (the directory that contains the working directory) is expressed as *../*.

You can change the working directory using the {% raw %}`cd` command followed by the path of the directory, expressed either as a relative path or an absolute path. For example, if you want to change the working directory to the parent directory, you’d write

cd ../

The trailing slash can be omitted and, if you’re moving into a subdirectory you can omit the ./ at the start, making the commands{% raw %}

cd ./Pictures/italy_pics/

and{% raw %}

cd Pictures/italy_pics

equivalent.

Files can be copied using the {% raw %}`cp` command, which takes two arguments: the path to the file to be copied and the path where you want the copy to be created, including the file name if you want the copy to have a different name: if you have a file called pic001.png in the italy_pics subdirectory and you want to copy it to the current working directory retaining the original file name, you’d run one of the following three commands (in decreasing order of command length)

cp italy_pics/pic001.png ./pic001.png
cp italy_pics/pic001.png ./
cp italy_pics/pic001.png .

The command to move files is {% raw %}`mv` and you use it just like `cp`, except for the fact that it can be used to rename files by trying to move a file into the same directory it came from but with a different file name:

mv italy_pics/pic001.png italy_pics/pic1.png

While using the interactive shell, you can use the *Tab* keyboard button to get automatic completion of commands and arguments when there is only one choice or get a list of possible option. This is not relevant for Bash scripting, but will be more relevant in the coming articles.

## Running a Bash script

Bash is an interpreted language and Bash scripts are ran mostly just like .py files.

To run a bash script saved in a file called {% raw %}`script.sh`, open a terminal window in the same directory as the script and run

bash script.sh

But there is actually a better way: just like with Python scripts, you can add a line at the top of the file, called the shebang line.

The shebang line consists of the two characters {% raw %}`#!` followed by the path to the interpreter to be used to run the script. In the case of bash, it is found at (or symlinked to) /bin/bash in pretty much every environment in which Bash installed, so you can add

!/bin/bash

at the very top of your script so that the shell knows what interpreter to run.

This is useful because you can make the script executable with{% raw %}

chmod +x script.sh

and then run it just like any executable with{% raw %}

./script.sh

## ~/bin

If it’s a script you think you’ll need to use often, you can either add it to the systemwide binary file paths (where the packages you download are installed) or create a {% raw %}`bin` folder in your home directory and copy the script there. For example, rename the script file to the command to the name you want to give to the command, for example `myfirstscript` using `mv` and then create the `~/bin` directory and copy the file there with the following three commands (in an interactive Unix shell or Bash shell on Windows):

mv script.sh myfirstscript
mkdir ~/bin
cp myfirstscript ~/bin/

and you can run the script simply by running{% raw %}

myfirstscript

from any working directory as long as you’re using the same user account you used to copy the file.


## Writing a basic Bash script

Let’s start writing a Bash script by making a script that copies all of our organized pictures into a single directory and renameames them according to the place where they were taken. This is not quite as good as the watermark we wanted, but let’s do one thing at a time

Open any text editor and create a file called {% raw %}`picorganizer` in the `~/bin` directory. The first thing you’ll need to add is the shebang line

!/bin/bash

Make it executable right away by opening a terminal and running{% raw %}

chmod +x ~/bin/picorganizer

## What our script will actually need to do

To solve the problem we have, we need to:

1. List the files in the directories we need  to copy the files from

2. For each file we need to take the following three actions
    * Copy the files in the target folder
    * rename the file to a progressive number
    * add a watermark of the place where it was taken

## Finding the files we need to copy

The example directory tree we'll be working with (that you can get by running the {% raw %}`tree` command) will be the following

italy_pics/
├── Center
│   ├── Assisi
│   ├── Florence
│   ├── Marche
│   ├── Pisa
│   ├── Rome_Lazio
│   └── Siena
├── North
│   ├── EmiliaRomagna
│   ├── Genoa_CinqueTerre
│   ├── Milan_Lombardy
│   ├── Trentino
│   ├── Turin
│   └── Venice
└── South
├── Bari_Apulia
├── Basilicata
├── Calabria
├── Campobasso
├── Naples_Campania
└── Sicily

where each city/region name is a directory containing the pictures taken in that place. These are example places in Italy and do not necessarily represent places I would recommend going to, don't judge me for a semi-pseudo-random selection of places.

To make the script aware of what we're working with, we need to get a list of files and directories and store them in a Bash variable. Let's start by learning the command to list files and directories.

In the Bash interactive shell, we can use the{% raw %}

ls

command to simply list the files and directories contained in the working directory, or you can run it with a path argument, like this:{% raw %}

ls /path/to/dir

to list the files and directories contained in {% raw %}`/path/to/dir`.

We can save the output of the `ls` command to a variable called `list` by writing, in our Bash script, the following:

list=$(ls)

where {% raw %}`$(command)` means *whatever `command` prints to standard output*. You can then use that variable just by prefixing the variable name with `$` or by prefixing it with `$` and enclosing the variable name in square brackets: just

ls

is equivalent to{% raw %}

files=$(ls)
echo "$files"

and to{% raw %}

files=$(ls)
echo "${files}"

This is not actually what we need right now, though: Bash's {% raw %}`for in` loop is able to iterate over files in the working directory very easily, and we can just nest them and get to the pictures very quickly:

for area in ; do
for city in ${area}/; do
for picture in ${area}/${city}/*; do
# do something with ${area}/${city}/${picture}
done
done
done

We can simply copy them all to another directory renamed to reflect where they were taken by adding the {% raw %}`cp` command to the innermost `for` loop:

mkdir ../italy_pics_organized
i=1
for area in ; do
for city in ${area}/; do
for picture in ${city}/; do
extension=${picture##.}
cityname="${city##*/}"
cp ${picture} "../italy_pics_organized/${i}-${cityname}(${area}).${extension}"
((i++))
done
done
done

There are a few things I haven't yet explained and used here. Now I'll explain them.

First of all, Bash doesn't have variable types: every variable is a string and it doesn't implement any mathematical operators or commands directly, so we need to use arithmetic expansion, which supports some specific [Shell Arithmetic operators](https://www.gnu.org/software/bash/manual/html_node/Shell-Arithmetic.html).

The {% raw %}`${parameter##word}` expression used to get the city name and extension in the following way: it looks inside the `parameter` for the pattern (`word` in this notation) we specify after `##` (in our case it's `*.` for the extension and `*/` for the city name) and only returns the rest of the `parameter`, deleting the pattern (but keeping it in the original variable). You can find more information about this and the rest of what can be done with parameter expansion using the `$` sign [here](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html).

# Using ImageMagick to Add a Watermark

We are doing something, and the script isn't going to get much more complicated than that, but we aren't adding a watermark yet. That's because there isn't a built-in tool to do that. No worries, though: the shell is expandable in the easiest way possible: by installing some software that provides a CLI interface.

The tool for the job when it comes to image manipulation is *ImageMagick*, which you can install by following the instructions on its own [official download page](https://imagemagick.org/script/download.php).

On Linux, what I actually recommend you to do is to install the *ImageMagick* package on Fedora/RHEL/CentOS by running

sudo dnf install ImageMagick

on Fedora or RHEL/CentOS 8 or by running{% raw %}

sudo yum install ImageMagick

on RHEL/CentOS 7 or earlier.

On Ubuntu, you can install the *imagemagick* package using APT by running{% raw %}

sudo apt install imagemagick

When using WSL with Ubuntu installed on top of Windows, you need to follow instructions for installation on Ubuntu while inside the Bash shell interface.

ImageMagick provides, among other things, a command called {% raw %}`convert`, which can be used, in conjunction with the `annotate` functionality, to add watermarks to images by running a command that looks like the following:

convert input.png -fill "textcolor" -pointsize textsize -gravity WhereTheTextWillBe -annotate +offsetHorizontal+offsetVertical "watermark text" output.png

where you need to replace {% raw %}`textcolor` with either a color name or an RGB hexadecimal color code (e.g. `green` or `#76ff03`), `textsize` with a number specifying the size of the font (e.g. `10` for a small font, `100` for a big font), `WhereTheText` will have to be replaced with something along the lines of `NorthEast` or `SouthWest` according to where you want the text to be, and `paddingHorizontal` and `paddingVertical` are offsets that can be used to move the text around or, more often, away from the edges. `input.png` and `output.png` have to be replaced with paths to the input and output pictures.

For our example, the command I chose, with `${picture}`, `${watermarktext}` and `${saveto}` being variables, is:

convert ./${picture} -fill "white" -pointsize 90 -gravity SouthEast -annotate +30+30 "${watermarktext}" "${saveto}"

So the final script is:{% raw %}

!/bin/bash

mkdir ../italy_pics_organized
i=1
for area in ; do
for city in ${area}/; do
for picture in ${city}/.jpg; do
cityname="${city##/}"
extension="${picture##*.}"
saveto="../italy_pics_organized/${i}-${cityname}(${area}).${extension}"
watermark="${cityname} (${area})"
convert ./${picture} -fill "white" -pointsize 90 -gravity SouthEast -annotate +30+30 "${watermark}" "${saveto}"
((i++))
done
done
done

After Ben Sinclair in the comments noticed that this wouldn't handle spaces in the path properly, I need to point out that you need to change the character used by Bash to separate items to loop through in the {% raw %}`for` loop by adding two lines at the top like the following:

IFS='
'

which sets the separator to the newline character ({% raw %}`\n`, aka the `LF` character in character encoding specifications), so that the script ends up being this:

!/bin/bash

mkdir ../italy_pics_organized
i=1
IFS='
'
for area in ; do
for city in ${area}/; do
for picture in ${city}/.jpg; do
cityname="${city##/}"
extension="${picture##*.}"
saveto="../italy_pics_organized/${i}-${cityname}(${area}).${extension}"
watermark="${cityname} (${area})"
convert ./${picture} -fill "white" -pointsize 90 -gravity SouthEast -annotate +30+30 "${watermark}" "${saveto}"
((i++))
done
done
done

# Tips For the Future

Here are a few things I didn'tell you you might need to know in the future when working with Bash or browsing Bash-related documentation.

## The Difference Between # and $

Usually, when reading documentation about Unix command line usage (including the sections of my [book about cross-platform mobile app development](http://carmine.dev/programmingflutter/) that concern Linux installation or CLI usage, for example) you might find that the commands are prefixed with the character {% raw %}`$`, like in the following example:

$ ls -alh

or are prefixed with {% raw %}`#`, like in the following example:

vim /etc/fstab

Those prefixed with {% raw %}`$` are meant to be executed as an unprivileged, regular user. Those prefixed with `#` are meant to be executed by the `root` account or by using `sudo`.

## Don't Delete Your Stuff: a (Not So) Funny Anecdote

You might want to clean up and use the command

rm -rf *

if you know the working directory is going to be each of the directories in which you have pictures organized *the old way*, for example. Only use such a destructive command if you’re 100% sure there is no way for it to get executed in the wrong folder. Make sure to at least turn it into something that only deletes files with the extension you want to delete like this{% raw %}

rm -rf *.jpg
rm -rf *.jpeg

If you’re thinking nobody would be so dumb not to think of it, there is at least one exception in the world. Some years ago I was a bit too confident and wrote a shell script that did some cleaning up afterwards. At some point during the execution of the script, it executed {% raw %}`rm -rf *` *in my home folder*. That’s not great.

I only figured it out when it was halfway through deleting the Documents folder, and it had already deleted the `~/bin` folder containing, ironically, most of my commonly used (and harmless) bash scripts (some of which I already was using on some remote servers and that I was able to recover) and itself in the process. I had recent backups of most of the important stuff, so it wasn’t the end of the world for me, but I can’t say it wasn’t annoying.

## The Bash if

Bash has an `if` clause, I just didn't feel like adding more complication to the script (even though it would have been better for it) by adding functionality that requires its use, its basic syntax is

if [[ condition ]]; then
# do something
fi

You can find more information about it online, and online you'll also find a lot more information on Bash than what it's made sense to include in this post.


Stay in touch with me [on Twitter @carminezacc](https://twitter.com/carminezacc) or follow [my blog](https://carmine.dev/) to know when the next post (about making a full-featured CLI tool with Python) comes out. Also, if you're interested in mobile development, check out [my book on Flutter](https://carmine.dev/programmingflutter).


Thanks to Ben Sinclair for finding out I had accidentally left spaces around the assignment operator in the two {% raw %}`files=$(ls)` code snippets, that I should have added `./` before ${picture} so that there's no chance the directory name will be interpreted as an option if it starts with an hyphen and for noticing that you might have spaces in one of the directory names or in the name of one of the picture, which would have broken the script.

Comments

[Ben Sinclair]

Nice post.

My pedantry begins:

files = $(ls) isn't going to work because you can't have spaces around the assignment operator.

The script as you have it will do unexpected things if there are spaces in any of the path or filenames. I'd suggest using a find command and splitting the result by / instead, then using the resulting variables inside double-quotes.

It will also have issues if the filename starts with - because convert will interpret it as a flag.

[Carmine Zaccagnino]

I tested the script and had two versions of it at some point (one in the body of the text and one I was executing), I must have left snippets of the wrong one in, you're correct, I'll fix it right now, thank you very much for noticing that.
P.S.: The issue with spaces around the assignment is in the first part of the post, I didn't pay as much attention to that as I should have and I usually leave spaces intentionally around operators when programming, I fixed that already. It wasn't there in the final script because I paid more attention with that one because I was in my usual vim Bash-scripting environment and not in a GUI text editor writing Markdown like I was for the earlier section.
P.S. II: Regarding the potential issue with spaces in the path, I didn't think that was likely in the specific example, but I'll address that ASAP. I'll also credit you for your spot-on observations.
P.S. III: Regarding the issue with filenames starting with -, it actually would only be an issue if the first directory's name (the area) starts with -, and that is easily fixed by changing the command from convert ${picture} (...) to convert ./${picture} (...)

[Pra3t0r5]

Excellent post! Since I got my hands in Linux, I automate every repetitive task. The time you can save compared to the time that it takes to implement the automation is incredible.
For more complex automations I tend to use python but just because of PyPI

[Carmine Zaccagnino]

Thank you! Python is much more powerful, but a bit less intuitive for some tasks. I'll be posting about automating tasks with it in the near future too.

[Paweł Kowalski]

In the old days when using SVN, i created a file called . (dot)
So i wanted to remove it, so i removed it... rm -rf . and commited.
Since SVN is not distributed, you can guess what happened :D

[Carmine Zaccagnino]

I bet it wasn't great fun when it happened! These kind of mistakes are fun to talk about sometimes (long after the fact) as much as they are annoying to deal with when they happen. Making mistakes is a great way to learn, no amount of warnings and cautionary tales will stop you from making mistakes at some point, while repeating such a mistake more than once is much more rare, in Italian we would say "Non tutti i mali vengono per nuocere", which means something along the lines of "Not all that's bad ends up hurting you in the long term".

[Paweł Kowalski]

Yeah, first couple seconds were not fun... :D

But it quickly turned out, we had a guru sysop that literally brought back backup within 10 minutes after calling him, so i could laugh pretty soon :D

[davidsonkillian]

Bash scripting can be incredibly useful for automating everyday actions and tasks on your computer. Here are some common examples of tasks that you can automate using Bash scripting:

File Management:

Renaming multiple files at once based on a specific pattern.
Moving, copying, or deleting files and directories based on certain conditions.
Sorting files dddcodigo.com.br/ddd-34/ into different directories based on their file type or metadata.
Text Processing:

Searching for specific text patterns or keywords within files.
Extracting information from text files or structured data formats (e.g., CSV, JSON).
Formatting or modifying text data to meet specific requirements.

[Amrita Technologies]

Linux automation tasks are a fundamental part of managing and maintaining Linux systems, offering numerous benefits for efficiency, reliability, and consistency. Here are some comments and insights on Linux automation tasks:

[Amrita Technologies]

Linux automation tasks are a cornerstone of modern system administration, offering a wide range of benefits. However, it's important to approach automation thoughtfully, considering the specific needs of your environment and maintaining a balance between automation and human oversight. When done right, Linux automation can lead to more efficient, secure, and reliable systems.

[Bobby Iliev]

Great article! 🙌

You should check out this Open-Source Introduction to Bash Scripting Ebook on GitHub as well!