Don't Use Bash for Scripting (All the Time)

Writing scripts is a subset of coding we sometimes can't avoid nor should be afraid of. The standard tool for writing scripts is Bash for UNIX environments and PowerShell for Windows environments. In this post, I explain when it's appropriate to use Bash for scripting and when it's not.

The Good Side of Bash

Bash can be considered a ubiquitous language, meaning it's installed nearly on any system you land on. It's hard to find a Linux-based operating system where either Bourne Again Shell (bash) or its older sibling Bourne Shell (sh) doesn't come pre-installed.

Working with code requires using a shell for executing a series of simple repeating tasks. It would be a burden to type these by hand every time you want something to happen. Using containers is a way to work around this problem but for environments where those are not supported, you would likely use Bash.

Bash shines in small and efficient scripts which are designed mainly for one thing: repeating instructions you would otherwise type manually from top to bottom. Given a standard project this typically involves installing external packages, running tests, and performing general maintenance tasks for the project.

Deploying a codebase for a local development environment typically involves installing dependencies with your chosen package manager, setting necessary environment variables and launching a development server. These could be baked into a shell script for easier access. Likewise preparing your codebase for tests, running them, and finally cleaning up is a good use case for a Bash script.

However, if the tasks presented above require more complex steps Bash can become cumbersome very quickly.

The Ugly Side of Bash

I'm declaring this out loud. Syntax of Bash is ugly and has a steep learning curve. Thanks to modern tools not many a developer begin their programming journey setting up UNIX environments with Bash. Therefore, we should not assume every new developer contributing to a software project knows their way around complex Bash scripts.

Consider the following example of assigning a default value to a variable where one is not given:

name="${1:-World}"
echo "Hello, $name!"

The last line looks easy. We are just printing a message with a variable but what on earth is going on in the first line? Syntax quirks like wrapping the statement in braces and the mysterious notation of 1:- make this very hard to understand. Without a good whitespacing it's painful to read as well. Anyhow, this line tells Bash to assign a string value of World to a local variable $name when the user has not given anything as the first parameter for the script.

Let's do the same in Python.

name = args.name or "World"
print(f"Hello, {name}")

Assume we have parsed the arguments using Python's argparse module. Python provides a logical or keyword for checking if the value on the left is interpreted falsy and a default assignment should be done instead. This is a lot more beginner-friendly.

A similar approach can be followed with Javascript:

const name = args.name || "World"
console.log(`Hello, ${name}`)

The only difference to Python is replacing or with the standard logical notation of ||. Granted, you'll have to do a bit more work by parsing the script arguments but the result will be more readable. When it comes to maintaining software one of my favorite phrases which everyone should learn is:

Without readability, there can be no maintainability. Without maintainability, there can be no value. Readability is everything.

Some might argue that scripts need not be readable as they only include so-called throwaway code. This is a problematic statement since I've witnessed mission-critical systems being glued together with scripts for the lack of better tools. Have you ever taken a maintenance job in a project built on top of several 100+ lines long scripts maintained by a task force of one developer? Having an indifferent attitude to script maintenance places businesses in grave risks.

Making It Rain With Bash

Writing good Bash scripts takes years of practice. It's not a fruitless path should you choose to follow it but requires a special set of tools and skills to generate value. To not fail with Bash there are two important pieces of advice to adopt.

The first advice is to use the unofficial Bash strict mode to avoid unnecessary debugging. It consists of two lines making your script behave in the following way:

• The script exits immediately upon encountering an error (-e flag)
• The script exits immediately upon encountering an undefined variable (-u flag)
• The script exits immediately when one or more calls in a piped statement fail (pipefail option)
• Internal field separator (IFS) is set to accept newlines and tabs making the notorious Bash array handling easier and more logical

There are some caveats for using the strict mode so I suggest reading the entire linked article and enabling or disabling the settings as you go.

The second advice is to install ShellCheck for linting and analyzing your script while writing it. This is a magnificent tool that will find incorrect array iterations or variable substitutions for you to fix before even running the script. ShellCheck is definitely among the finer static analyzers I've used and it comes pre-installed in Travis CI runners meaning you can have your Bash scripts checked against defects on each pushed commit. Believe me, you should.

Speaking of debugging your code, it's possible with Bash although I've never found the native debugger (-x flag) or throwing print statements around stick with me. If maintaining your script requires periodic debugging sessions you should either enable the strict mode or refactor the logic.

Scripting Without Bash

Let's imagine you have written a fairly complex Bash script spanning over tens or – in the worst case – hundreds of lines. It now needs to be refactored for one reason or another. This is an excellent position to drop the Bash!

First, make a note of what is the main language of your codebase. If it's PHP, Python, or Javascript you are in luck. I'm not too familiar with Go or Rust but I've seen and used some great command-line tools written with them so I shall say you're in luck with those as well.

Next, you likely have a shebang line on top of your script which reads something like #!/bin/bash. Change this to eg. #!/usr/bin/env node replacing node with your desired code interpreter.

You might already know that this line makes your script executable given the right permissions by commanding ./script; or just script if you place it in a folder included in the $PATH environment variable. From here, start converting your script line by line to a new language importing necessary modules where needed. In the end, your script may become longer but it will definitely be more robust and maintainable.

One obstacle you might encounter while refactoring is the ability to run shell commands. In Bash, you don't need to do anything other than writing the command (unless you try to parse and validate its output in which case I wish you luck). With other languages, you have to invoke them either by using a built-in or an external module. Notable picks are execa for Node and delegator.py for Python. There should be similar modules for handling child processes within scripts for all the popular languages. Many of these modules allow to run commands asynchronously and handle the output in a flexible way which is something you might have a hard time to implement with Bash.

If built-in language features are not enough there are handy frameworks created for writing command-line tools. Take for example oclif for Node and click for Python. If your business logic relies heavily on command-line do not hesitate adopting these instead of Bash.

How I Did It

As an example, I present a script I wrote for creating quick drafts for this Gatsby site. The script is written in Javascript and has the following features which would be difficult or even insane to implement with Bash:

• Interactive prompts and ability to pass given data through validator functions
• Transforming sentences to SEO-friendly slugs (eg. Blog Post Title to blog-post-title)
• Coloring terminal output without using weird ANSI codes
• Asynchronous programming
• Converting structured data to YAML front matter

I'm not saying it would be impossible to do these things with Bash but it would be a hot mess of weird awk patterns I would end up debugging for hours.

One concrete upside is that I'm able to use logic from several 3rd party modules to achieve most of these. With Bash, I would have to find the proper logic from somewhere, copypaste them to dedicated files and source those in my code. While this approach would have worked, it would have been frustrating and prone to errors.

For this topic, I like to cite Sindre Sorhus who argues strongly for using small and focused modules instead of reinventing the wheel:

Some years ago. Before Node.js and npm. I had a large database of code snippets I used to copy-paste into projects when I needed it. They were small utilities that sometimes came in handy. npm is now my snippet database. Why copy-paste when you can require it and with the benefit of having a clear intent. Fixing a bug in a snippet means updating one module instead of manually fixing all the instances where the snippet is used.

I want programming to be easier. Making it easier to build durable systems. And the way forward, in my point of view, is definitely not reinventing everything and everyone making the same stupid mistakes over and over.

Actually, if you want tips on creating efficient command-line tools with Javascript go and check some of Sorhus' repositories on GitHub.

There might come a day Bash has a good (nested) dependency system and friendlier syntax. As for the syntax part, it has been improved in Friendly Interactive Shell (fish) which I'm using daily. The Fish developers advertise its syntax is "simple, clean, and consistent" which I do agree. However, writing your scripts with exotic shell languages might have more risks than gains unless your whole application is written in the same language.

Until then I'm most comfortable scripting with languages I use to write my business logic with.

Conclusion

The main take of this post is not to bash Bash but not to use it for every task at hand. Instead, you should design your scripting needs properly before implementing, and notice when the complexity grows too large to handle reasonably within the limits of Bash.

Using Bash for small scripts is a valuable skill in order to quickly carry out repeating tasks – for everything else use whatever the language you prefer.

Comments

[Ada-Claire]

The strong points of bash (ksh, and sh) are the ubiquity. I am a strong proponent of using the right tool for the job, and many of the other scripting languages out there (python, node, etc) don't come close to the ubiquity of shell scripts.
I can agree that to the uninitiated there is a lot of syntax that looks really odd, but with well structured scripts (making proper use of functions, built-ins, etc), bash scripts can be quite understandable and powerful.
To take full advantage of shell scripting (which can be quite performant and portable), script authors should learn more about the shell and take advantage of shell-isms. Bash, for example, has native support for REGEX (in version 4.x up, I'm not sure how far back it goes) that negates the need for many grep operations. In addition one should seldom use grep piped to sed or awk; those programs (nay, programming languages) contain all that's necessary to do the same work without spawning another executable.
Using IFS and array variables is a much cleaner solution than using cut. Default values, variable splitting, and no-op operations are indispensable for moderate to advanced scripts.
Against your point, however, bash/shell scripts are nice because they are concise. The "hello world" example you contrived is a good point: the shell script is 2 lines. The python code is 2 lines plus the setup for argparse. Once one is familiar with the syntax, scripts like those are easier to read in bash because they're shorter.
I also like the points about using shellcheck and strict-mode. My experience is also that posix-mode is a good tool to use in scripts where either the bash version isn't guaranteed, or where bash may not be used at all. There are also some really nice testing frameworks (like BATS) that will help with code correctness. Any sufficiently advanced scripts should also be modular (and make use of the source built-in), with perhaps a build that will make a single executable script for deployment. Bash scripts, like any other code, should also be source managed unless they are sufficiently small, straightforward, and localized, that source management just doesn't make sense.
I, unfortunately, see many places where heavier scripting languages like python or node are shoehorned in to do a job where one or more bash scripts would do the job with less effort. We are entering a time when fewer and fewer people are as familiar with the basics of shell scripting, yet there are far more tools and libraries available to make shell scripts easier to write.
In conclusion: I agree with the premise that bash/shell scripting should not be used everywhere for every task, but I disagree with you on where to draw that line. Each and every task ought to be evaluated to determine which tool is best for that job, and many times a simple (or moderately complex) shell script will get the job done just as fast (or faster) with less effort than a heavyweight scripting language will (because shell scripts are designed to do one thing very well, and other scripting languages may be designed with other goals).

[Niko Heikkilä]

Good points. An analogy could be made with text editors. On the other side we have Vim which is super powerful editor in the hands of a wizard but many still prefer to use eg. VS Code for getting job done nice and easy. I use both, by the way.

[Ada-Claire]

I do too ;-)

[Joshua Mervine]

I started reading this with the intention of sharing "all the counter points". Admittedly, not a good way to start reading something. After reading through it though, your points are well thought-out and communicated. So kudos on the great post!

This points that I would make (on both sides) have mostly already been made.

Point: Bash is everywhere.
Counter point: Same with Python 2.7 (for the most part). Precompiled Go binaries can run anywhere.

Point: Bash do almost everything you need in a script.
Counter point: Ruby, Python, Go, etc. can do it better and simpler.

Point: Bash doesn't require external dependancies.
Counter point: openssl, libcurl, etc. are external dependancies, they're just typically already installed. Python, Ruby, etc. can also be written without external dependancies.

I could go on.

As Aaron Cripps mentions in his comment, "I am a strong proponent of using the right tool for the job."

All that said, personally, I'm a huge fan of Bash. With solid programming practices and the use of a linter, it can be clean, concise, portable and readable.

[Niko Heikkilä]

Thanks for reading the article thoroughly. 👍🏽

[Niko Heikkilä]

Hi, I've written far too many Bash scripts to understand its downsides. The link you provided doesn't really change my views, would you care to elaborate where I have made mistakes?

Furthermore, your so-called "lies" are just opinions. It's not too hard to have some respect while commenting. Remember this and I don't have to report you.

[Ben Overmyer]

This is very hostile.

I'm guessing that you identify with Bash, and since Bash is part of your identity, you took the original post as a personal attack.

There is no need for this.

There are better ways to disagree with someone than to attack someone's competence and integrity.

[yvs2014]

Let's do the same in Python.

name = args.name or "World"
print(f"Hello, {name}")

😕

% python -c 'name="World"; print(f"Hello, {name}")'
  File "<string>", line 1
    name="World"; print(f"Hello, {name}")
                                       ^
SyntaxError: invalid syntax

% python3 -c 'name="World"; print(f"Hello, {name}")'
  File "<string>", line 1
    name="World"; print(f"Hello, {name}")
                                       ^
SyntaxError: invalid syntax

[Niko Heikkilä]

You're using Python 3.5 or earlier? f-strings are quite delicious as of 3.6. :)

[yvs2014]

Yep! It's just the example of many assumptions underpinning python code for scripting, and so on

[Niko Heikkilä]

Yes, for maximum compatibility the old % way of concatenating would be safest, of course.

[Niko Heikkilä]

Thanks for your input. Since you're just obviously misreading intentionally I'm signing this discussion off with a notice that opinionated blog posts and research articles are two different things. Learn that.

[Brian Masinick]

What are your thoughts on using binary compiled software versus scripting languages?

One thing to consider is that when the tools you write are regularly used there is a point at which a software application becomes a more suitable tool than a script, no matter how convenient the tool or script was when first created.

Thoughts?

[Niko Heikkilä]

Some of the larger scripts I create with different languages I actually put in Docker images so they are quite like binaries. Helpful when moving those between platforms.

I'm trying to learn Golang better to do this without containers as well.

[Brian Masinick]

Do developers use C or C++ anymore for anything other than operating system code?

I used to be an application software maintainer for a commercial UNIX operating system group. The classic old utilities were written in C. The desktop utilities (Motif, CDE, etc.) we're written in C++.

That was in the mid to late 1990s.

Most newer Linux distributions have C code for core applications and a somewhat greater variety of coding languages for applications.

Golang is an example of a comparatively newer application language.

[Brian Masinick]

Do you see application development being done frequently with interpretive languages?

[Niko Heikkilä]

Your scenario highlights a work culture where junior dev skills are fundamentally underestimated and undervalued. If you get kicks from lecturing to people, then show us a better example and write a post here why (in your opinion) Bash should be used for scripting. After all, DEV is a free platform.

[Tobias Sette]

tl;dr "Shell should only be used for small utilities or simple wrapper scripts." google.github.io/styleguide/shell.xml

It is all I need to know.

[Ada-Claire]

That's a nice resource that I haven't seen before, thanks for that!

... definitely going to stick to those guidelines in future scripts.

[Bruce Davidson]

I have to agree. The more complex the task, the more tedious bash seems. Especially when I need to pair it with .bat for cross platform projects, python or node make a lot more sense. My preference is coffeescript. I find Cakefile to be very helpfull.

[Niko Heikkilä]

Task runners are great tools for running maintenance scripts on projects. My usual pick is pyinvoke but, naturally, these should be selected depending on the project language.

[Corey McCarty]

I personally use python scripts for anything that stands alone or anything that benefits from internal classes. Anything running in my build server and needing system utilities like scp and ssh to push data around goes in a bash script for ease. This came about after spending a great deal of time chaining bash commands together into one liners that did a great deal of heavy lifting for me regularly. I stepped from one liners into bash scripts, and eventually decided that a good bit of the work that I was doing would be more easily written in python, and for the most part I've been able to keep with python for all things that are not deploying my java applications.

[Niko Heikkilä]

You are just trolling here, friend.

No one in this discussion is hating Bash, no matter how much you like to throw that around. No one is telling it's a mistake to use Bash; there are just alternative tools that might serve you better depending on the use case. No one is coming to delete your Bash scripts.

Like I said, you are intentionally drawing a straw man argument to this discussion which is not welcome here nor does it contribute to anything. Let it go, take a walk, and chill.

[Ben Overmyer]

Being a good developer or not isn't the issue. It's your choice of language.

Calling out statements as being opinion is fine. Disagreeing with them and providing a better option is fine. But calling them lies and insulting the intelligence of the author? That's not fine.

I'm guessing from the quality of your English that it's not your native language. Perhaps there's a misunderstanding here in terminology. For example, "lie" is not the same as "falsehood." The former implies malicious intent, whereas the latter does not.

[joseph putz]

Im inheriting a project that is moving files between remote servers. The bash programs written are long very difficult to understand. I can't understand how this would be an acceptable engineering tool with it being without basic debugging and error handling.

[Niko Heikkilä]

Closest guess is that the solution was made in a hurry, and therefore, quick Bash scripts seemed like the fastest solution. This is acceptable up to a point, but refactoring / rewriting needs to be ultimately done at some point.

[Brian Masinick]

codeproject.com/?cat=2 has quite a few projects that use C++ in conjunction with the libraries and API's for a wide variety of system, application and programming interfaces.

Admittedly many of them are close to the system level as opposed to rapidly developed tools and conveniences.