Perfect Elixir: Testing Shell Scripts with Bats

Automated testing is crucial for reliable software development, yet shell scripts are frequently left untested, despite managing critical tasks like deployments, database migrations, and CI pipelines. Neglecting tests here can lead to subtle bugs and fragile workflows. Today, let's make sure all our code is testable, treating shell scripts with the same discipline we apply to application code.

ℹ️ BTW test automation doesn't mean every line absolutely must be tested; rather, it's about ensuring we can test. There are cases where skipping tests can make sense — but we can never do so out of laziness.

Table of Contents

• Shell Script Testing with Bats
• Bats Helper Libraries
  • Creating a Bats Download Manager
• Mocking
  • jasonkarns/bats-mock 🛑
  • grayhemp/bats-mock ✅
• Speeding Up Bats Tests
• Integrating Bats Into the Workflow
• Conclusion

 

Shell Script Testing with Bats

Bats, Bash Automated Testing System, is a testing framework for shell scripts, with a nice test-runner and various commands to test what a script does when run. Let's see how it works!

First we need Bats installed, and pkgx makes that easy (to refresh on pkgx please refer to the "Setting Up an Elixir Dev Environment" article):

$ git-nice-diff -U1 .
/.pkgx.yml
@@ -7 +7,2 @@ dependencies:
   postgresql.org: =17.0.0
+  github.com/bats-core/bats-core: =1.11.1

We can now write a test, e.g. here we assert bin/doctor --help prints out usage information:

$ cat test/bin/doctor.bats
@test "--help outputs usage" {  
  run bin/doctor --help  

  [ "$status" -eq 0 ]  
  echo "$output" | grep -Fq "Usage: bin/doctor [options]"  
}

And to run our test-suite:

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used

1 test, 0 failures

It's a great start, but we quickly hit a blocker: How do we test how our script behaves if a tool such as pkgx is missing? We can't mess with the real pkgx because it's core to our entire project, so the solution is dependency injection — inject mocks instead of real commands. This is similar to what we covered in the previous "Automating Tests in Elixir Projects" article, and the Bats community has created mock libraries that promises to make it simple to create mocks. So let's go explore them.

But wait, before we try those helper libraries, we first need to talk about how we download libraries.

 

Bats Helper Libraries

The Bats ecosystem includes excellent helper libraries, but incorporating these libraries into our project isn't straightforward, with common approaches relying on Git submodules or committing their code into our repository directly.

But Git submodules rely on cumbersome commands, and committing external source files is a hack that goes against how we deal with dependencies. Basically, both add complexity. Especially compared to how simple Elixir handles it with mix deps.get, where well-defined dependencies are simply pulled down.

So, we'll create a lightweight Bats Download Manager to automate the fetching of Bats libraries, so we can use helper libraries without adding new hacky ways of dealing with source code to our project.

ℹ️ BTW simple doesn’t mean easy, e.g. Git submodules are easy to start but add cognitive complexity with their commands and mental model. Rich Hickey’s “Simple Made Easy”) speaks into this. Optimizing for simplicity — even when it's initially harder — pays off long-term.

 

Creating a Bats Download Manager

A good software principle is to download external files as dependencies, so we're creating our own "Bats download manager" to handle this for us.

Our first goal will be to download the library bats-assert. We can easily download it via the GitHub CLI tool, and since the download will be a tarball archive we'll also need the tar command to unpack it. These are both available via pkgx:

$ git-nice-diff -U1 .
/.pkgx.yml
@@ -8 +8,3 @@ dependencies:
   github.com/bats-core/bats-core: =1.11.1
+  cli.github.com: =2.68.1
+  gnu.org/tar: =1.35.0

To make use of the tools we wrap them in a shell script that takes arguments for a repository and its release-tag:

#!/usr/bin/env bash  
set -euo pipefail  

BATS_DEPS="$(realpath --relative-to="$(pwd)" "$(dirname "$0")/../.bats_deps")"  

usage() {  
  echo "Bats manager to download helper libraries"  
  echo ""  
  echo "Usage:"  
  echo "  batsman <owner/repo> <tag>"  
}  

 "$1" == "--help"  && { usage; exit 0; }  
 "$#" -ne 2  && { usage; exit 1; }  
repo="$1"  
tag="$2"  
destination="$BATS_DEPS/$repo"  
sha=$(gh api "/repos/$repo/commits/$tag" --jq .sha)  

rm -rf "$destination" && mkdir -p "$destination" && cd "$destination"  
gh release download "$tag" -R "$repo" --archive "tar.gz" -O "release.tar.gz"  
tar -xzf "release.tar.gz" --strip-components=1  
rm "release.tar.gz"  

version_info="$repo $tag (GitHub release) ${sha:0:8}"  
echo "$version_info" > .bats_version.txt  
echo "$version_info -> $destination"

ℹ️ BTW this is just the basics of a download manager, it can be extended as far as is needed. E.g. in this later commit security is improved by checking for specific SHAs.

It's now straightforward to download bats-core/bats-assert (which itself depends on the library bats-core/bats-support, so we'll actually download both):

$ bin/batsman bats-core/bats-assert v2.1.0
bats-core/bats-assert v2.1.0 (GitHub release) 78fa631d -> .bats_deps/bats-core/bats-assert

$ bin/batsman bats-core/bats-support v0.3.0
bats-core/bats-support v0.3.0 (GitHub release) 24a72e14 -> .bats_deps/bats-core/bats-support

$ ls .bats_deps/bats-core/bats-assert/
LICENSE  load.bash  package.json  README.md  src  test

Now we can use the nice assert-functions that come with the bats-assert library:

$ git-nice-diff -U1 .
/test/bin/doctor.bats
L#1:
+setup() {
+  load "$(pwd)/.bats_deps/bats-core/bats-support/load"
+  load "$(pwd)/.bats_deps/bats-core/bats-assert/load"
+}
+
 @test "displays help message when --help is used" {
   run bin/doctor --help
-  [ "$status" -eq 0 ]
-  echo "$output" | grep -Fq "Usage: bin/doctor [options]"
+  assert_success
+  assert_line "Usage: bin/doctor [options]"
 }

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used
1 test, 0 failures

Great stuff. And to ensure our entire team downloads these dependencies, we add a simple check to bin/doctor that prompts everyone to run the proper bin/batsman calls:

$ git-nice-diff -U1 .
/bin/doctor
L#56:
   "mix ecto.create"
+check "Check Bats helpers" \
+  "ls .bats_deps/bats-core/bats-assert > /dev/null" \
+  "bin/batsman bats-core/bats-assert v2.1.0 && bin/batsman bats-core/bats-support v0.3.0"

We now have a simple way for the whole team to consistently fetch the needed Bats libraries, mirroring the familiar dependency management we know from mix deps.get.

And now we can return to the original problem: How can we test bin/doctor without actually having it run all the commands it depends on?

 

Mocking

We have Bats working, we can download helper libraries, now it's time to find an effective way to test shell scripts in isolation from the external commands they call. We need a reliable way to mock out those command calls, and for that we'll explore mocking libraries and see how they can help us achieve this.

 

jasonkarns/bats-mock 🛑

The bats-mock library by Jason Karns promises straightforward mocking of external commands. Let’s download and integrate it:

$ bin/batsman jasonkarns/bats-mock v1.2.5
jasonkarns/bats-mock v1.2.5 (GitHub release) 24f995f9 -> .bats_deps/jasonkarns/bats-mock

Then we load it, and use its stub command:

$ git-nice-diff -U1 .
/test/bin/doctor.bats
L#3:
   load "$(pwd)/.bats_deps/bats-core/bats-assert/load"
+  load "$(pwd)/.bats_deps/jasonkarns/bats-mock/stub"
 }
@@ -11 +11,12 @@ setup() {
 }
+
+@test "advise to reinitialize devenv if pkgx is absent" {
+  stub which "pkgx : exit 1"
+
+  run bin/doctor
+
+  assert_failure
+  assert_line "Suggested remedy: dev off; dev || source bin/bootstrap"
+
+  unstub which
+}

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used
 ✓ advise to reinitialize devenv if pkgx is absent

2 tests, 0 failures

This initial success is encouraging: we stub which to simulate pkgx is missing, and verify our script responds correctly. Simple and effective!

Unfortunately, the approach quickly breaks down in more complex cases. For example, this stub:

stub psql '-U postgres -c "\q" : echo ""'

should simulate psql producing no output — but the library fails with obscure errors:

$ bats -r test/bin/doctor.bats
bin/doctor.bats
 ✗ advise to create a user when psql fails to connect
   (from function `unstub' in file .bats_deps/jasonkarns/bats-mock/stub.bash, line 46,
    in test file test/bin/doctor.bats, line 40)
     `unstub psql' failed
1 tests, 1 failure

This appears to be a known bug. Rather than get caught up troubleshooting brittle quoting issues, let's acknowledge this limits and move on quickly to try other libraries.

 

grayhemp/bats-mock ✅

Sergey Konoplev has authored the grayhemp/bats-mock library, which supports creating mocked commands. It doesn't mock existing commands, rather it generates generic mocks and we're responsible for injecting them into our scripts as needed. Let's see how that works.

First, fetch the library:

$ bin/batsman grayhemp/bats-mock v1.0-beta.1
grayhemp/bats-mock v1.0-beta.1 (GitHub release) ac1a4475 -> .bats_deps/grayhemp/bats-mock

And then we change bin/doctor to allow the which command to be overwritten by specifying the environment variable _WHICH:

$ git-nice-diff -U1 .
/bin/doctor
L#34:
+WHICH=${_WHICH:-which}
 section "Running checks…"
 check "Check system dependencies" \
-  "command which pkgx && which erl && which elixir" \
+  "command $WHICH pkgx && $WHICH erl && $WHICH elixir" \
   "dev off; dev || source bin/bootstrap"
 check "Check developer environment" \
-  "which stat | grep -q '.pkgx'" \
+  "$WHICH stat | grep -q '.pkgx'" \
   "dev off; dev"

Finally we create a generic mock and plug it into the _WHICH variable (by exporting it, it automatically becomes part of the environment that run uses):

$ git-nice-diff -U1 .
/test/bin/doctor.bats
L#3:
   load "$(pwd)/.bats_deps/bats-core/bats-assert/load"
+  load "$(pwd)/.bats_deps/grayhemp/bats-mock/src/bats-mock"
 }
@@ -10 +11,14 @@ setup() {
 }
+
+@test "advise to reinitialize devenv if pkgx is absent" {
+  export _WHICH="$(mock_create)"
+  mock_set_side_effect "$_WHICH" "case $1 in pkgx) exit 1;; esac"
+
+  run bin/doctor
+
+  assert_failure
+  assert_line "Suggested remedy: dev off; dev || source bin/bootstrap"
+   "$(mock_get_call_args $_WHICH)" = pkgx 
+}

And all that works:

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used
 ✓ advise to reinitialize devenv if pkgx is absent

2 tests, 0 failures

That's pretty cool. Using the environment variable mechanism to inject commands is a bit more verbose, but it also makes the code nicely explicit. A worthwhile tradeoff.

But we should stress-test this library further, and if we create tests for all the code-branches of bin/doctor it can end up looking like this:

$ bats -r test/bin/doctor.bats
doctor.bats
 ✓ displays help message when --help is used
 ✓ reports healthy system (all stubs are by default configured for success)
 ✓ executes all stubs in their expected order
 ✓ verifies Elixir is available before running mix
 ✓ advise to reinitialize devenv if pkgx is absent
 ✓ advise to reinitialize devenv if erl is absent
 ✓ advise to reinitialize devenv if elixir is absent
 ✓ advise to toggle the devenv if stat is not provided by pkgx
 ✓ advise to start the database server if pgrep cant connect
 ✓ advise to create a user when psql fails to connect
 ✓ advise to install hex if hex is not installed locally
 ✓ advise to run mix setup if mix dependencies are unavailable
 ✓ advise to run mix setup if mix dependencies are outdated
 ✓ advise to run ecto if the database is missing
 ✓ advise to advise download Bats helpers if theyre missing

15 tests, 0 failures

ℹ️ BTW I subsequently refactored the tests to bring down complexity, so each test is kept focused and simple, and delegate into helper-functions. Without this refactor it was grievously complex trying to keep an overview of e.g. the repeated calls to mock_create and mock_set_side_effect. This article drags on if we also cover those details but if you're interested the full /test/bin/doctor.bats is available here.

 

Speeding Up Bats Tests

We've made great progress on testing shell scripts, with an easy and simple approach to downloading the Bats framework and helper libraries.

But tests have gotten slow to run.

Here I've added a bin/bats-test script to simplify running tests (see commit "Add initial bin/bats-test script" for details), and it shows our current performance:

$ bin/bats-test
Running all tests...
…
36 tests, 0 failures in 30 seconds

30 seconds!? That's slow. Let's try some straightforward optimizations to see what we can do about this.

Bats can run tests in parallel (to follow the code changes for this see commit "Run bin/bats-test tests in parallel"):

$ bin/bats-test
Running all tests (via 14 parallel jobs)...
…
36 tests, 0 failures in 19 seconds

19 seconds is nearly a 40% improvement, so that did help. But it's still slow…

Shell scripts tend to not change as often as application-code, could we skip tests until they actually change? We could persist a checksum when tests pass, and if a new run has the same checksum then skip the tests (to follow this change see commit "Only run bin/bats-test tests if checksums indicate a change"):

$ bin/bats-test
Running all tests (via 14 parallel jobs)...
…
38 tests, 0 failures in 22 seconds

$ bin/bats-test
No changes since last test run, skipping 38 tests.

0 seconds is certainly fast 😂, but of course doesn't alleviate the pain when tests do have to run. We could keep going to optimize even further, but despite our fingers itching to do more our time will be better spent moving on (for this article, at least), because all told we've successfully improved how we work with shell scripts:

• Developers can rely on bin/bats-test to run shell tests as fast as possible (even if that speed leaves something to desired, and can be improved incrementally over time by those who wish to optimize).
• For the cases where developers are not working on shell scripts the tests take no time, meaning friction is minimized for everyone that works elsewhere in the system.

 

Integrating Bats Into the Workflow

In the "Streamlining Development Workflows" article we introduced the core workflow of running bin/shipit to quickly push changes, which internally runs mix test to handle Elixir's test automation. We need to extend this mechanism to also run Bats tests.

Adding more test-commands directly to bin/shipit makes it crowded though; let’s create a dedicated bin/test script instead:

$ cat bin/test
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.shhelpers"
step --with-output "Running Elixir tests" "mix test"
step --with-output "Running Shell tests" "$(dirname "$0")/bats-test"

$ bin/test
• Running Elixir tests:
Running ExUnit with seed: 248176, max_cases: 28
.......
Finished in 0.05 seconds (0.02s async, 0.03s sync)
7 tests, 0 failures
• Running Shell tests:
No changes since last test run, skipping 45 tests.

And we rewire bin/shipit to call our new script:

$ git-nice-diff -U0 .
/bin/shipit
@@ -6 +6 @@
-step --with-output "Running tests" "mix test"
+"$(dirname "$0")/test"

With this we have easily integrated shell testing into our workflows, and team members won’t need extra setup; they’ll just automatically benefit from the added tests.

 

Conclusion

Shell scripts deserve testing just as much as our application code, yet they're often overlooked. In this article, we've shown it's straightforward to integrate robust shell testing into our structured workflows using Bats and dependency injection.

Instead of accepting untested scripts as the status quo, we've chosen to uphold strong engineering principles and committed to simple, decoupled tools and techniques to accomplish that.

Now we can write our shell scripts with the same sense of safety ✅ as our Elixir applications, with isolated and reliable tests ✅ fully integrated into our ultra-fast bin/shipit workflow ✅.

Happy testing! 🚀