writing quick and dirty scripts using bash

You have a few lines you've been typing in your shell, and you'd like to just run it the way you run any other CLI including options. If you're writing a shell script, here's a few tips to help you get a head start.

1. #! the shebang line
2. usage docs
3. flags and options
4. error handling
5. dealing with output

This doc assumes you're using bash in an OSX/Linux environment.

1. the shebang line

You'll usually see a line at the top of shell scripts #!/bin/bash for example. This is called the shebang line - its a directive for the shell to run this file using the program at that path. For example, if this were a node.js script, I would have something like:

#!/usr/bin/node

console.log('hello world')

Since we want to run this script using bash, we'll set the shebang line to bash. You could do #!/bin/bash - but this will default to your user's system install of bash. You might try #!/usr/local/bin/bash if you have another version of bash at that directory. It's hard to account for where your current shell may be - in this case, you can simply ask the environment where it should be with env.

#!/usr/bin/env bash 
# will run the `bash` that is currently in your environment (i.e. in your $PATH). 

2. usage docs

One way a user might expect to see how to use the tool would be to just type the naked command, e.g. mytool. The following will print the usage docs for the tool if there are no arguments passed to the tool

function usage {
    echo "usage: `basename "$0"` /path/to/file"
    echo ''
    echo 'ex. mytool /tmp/foobar'
    echo ''
    echo 'Does something useful to the target file'
    echo ''
    echo 'Options:'
    echo '-h : show this doc
}

if [ "$#" -lt 1 ]; then
  usage
fi

2. options, flags, arguments, and env vars

Once you want to set a CLI flag or an option, move to getopts. The syntax can be a little confusing, but hopefully this breakdown makes sense.

• one limitation is that you can only set short flags like -x -V
• the definition of all the CLI flags happens in the parameter to getops: :ht:
• the first : which means that you disable the default error handling of getopts
• the second h which means that we have an option (-h) with no arguments
• the third part is the t: (a colon follows the flag name) which means that the option (-t) requires an argument

while getopts ":ht:" opt; do
  case $opt in
    h)
      usage
      ;;
    t)
      type=$OPTARG
      ;;
    \?)
      echo 'Invalid option: -$OPTARG'
      ;;
  esac
done

shift $(($OPTIND - 1))
remaining_args=$@

In this example, we have a CLI with -h -t as options with -t requiring an argument. The last two lines allow you to get the rest of the arguments provided to your script. They all wind up in the $@ variable.

If you ran mytool -t foo /path/to/file (with these options), you'd have the result:

type="foo"
remaining_args="/path/to/file"

If you want to check if a variable is already defined, and abort if not, you can use bash conditionals.

if [ -z "$SOMETHING_IMPORTANT" ]; then
  echo '$SOMETHING_IMPORTANT wasn't defined'
  usage
fi

There's a whole host of things you can check, -z is one of my more frequently used ones - see http://tldp.org/LDP/abs/html/comparison-ops.html

If you want to set default values for environment variables (and allow a user to override them if they want) - this will set USE_HEADLESS_CHROME to 0 if it was not defined already. USE_HEADLESS_CHROME=${USE_HEADLESS_CHROME:-0}

3. Dealing with output

Are you trying to run a long-running command, but you want to save the output for further debugging later and you also want to see it running? Enter tee! It lets you pipe your output to BOTH stdout & a file at the same time.

apt-get update -y | tee apt-get.log   # view and save the output
apt-get update -y > /dev/null 2>&1    # ignore all output
apt-get update -y 2>&1 > /dev/null    # Keep only stderr

See an interesting Stack Overflow discussion here about redirects in bash.

4. Error handling

The first time you run your script, you'll likely see that even though some commands are failing, the script continues. No, I did not want to create a file named 404 Not Found. /facepalm.

set -e will cause your shell script to abort when any command fails. Conversely, if you want to ignore any errors and have the script continue, use set +e (which is the default)

set -e
false
echo "You won't see this line"

Unfortunately, set -e won't work if you're using pipes and still want to abort the script whenever an error happens anywhere along the pipe. You'll have to use set -o pipefail to enable that behavior.

ps aux | grep java | awk '{print $2}' | tee pids.log # grepping for java fails, but the pipe continues

Sometimes, you might expect an error to happen and you want to do something about it. $? will contain the exit code of the last command executed. You can even assign it to a variable to use to exit later.

set +e                                  # so that errors do not cause the script to abort
apt-get install $somethingReallyNew     # I know this is going to fail
apt_exit_code=$?
if [ "$apt_exit_code" != 0 ];then  
    echo 'whoops, need to run apt-get update first'
    exit $apt_exit_code
fi

You can also include these options in your shebang line at the top of your script: #!/usr/bin/env bash -eo pipefail

Bash is pretty powerful for small scripts available on any linux machine like seeing running processes named 'foo' - ps | grep foo. I'd much prefer to use a higher-level language like node.js or ruby, but for small scripts that do one small thing - its hard to beat bash. Hopefully with these guidelines, you can get started immediately slapping together a quick and dirty tool in bash.

---

For more bash scripting resources, I've found these links particularly helpful:

• An interactive explainer of shell commands - https://explainshell.com
• Writing Safe Shell Scripts - https://sipb.mit.edu/doc/safe-shell/

Comments

[Ondrej]

I genuinely admire how you DevOps people have nerves from steel with scripting in Bash/Zsh/Fish/other shells. I always avoid it at all cost and use for (client & server-side) scripting Ruby or Python because it seems to me much easier.

[Danny Perez]

Ruby/JS are my go-tos for anything more than 20 lines of bash. I don't know of that many people that actually enjoy writing bash (though I have met a few). I generally only do it out of necessity for small things.

The company I work at has a mix of languages in use among engineers which means some of ruby, python2, python3, and js. Each one tries to avoid installing the other. The easiest way to convince these teammates to use some tools is to just say "it only uses bash."

...then one day we needed to introduce some python scripts into our bash scripts, and all hell broke loose with people's python installation. obligatory XKCD - python environments

[Ondrej]

"The company I work at has a mix of languages in use among engineers which means some of ruby, python2, python3, and js. Each one tries to avoid installing the other. The easiest way to convince these teammates to use some tools is to just say "it only uses bash." "
That is pretty clever :D

[Alexey Melezhik]

Hi Danny! You might take a look at Sparrow, it mitigates some Bash scripting difficulties ( documentation / input parameters handling / distribution / versioning ) yet letting write your nifty scripts in Bash in any style.