Running Your BATS Tests Against Multiple Bash Versions in GitHub

In this post I'll go over the github workflow I created that allows me to test bash scripts using BATS against multiple versions of Bash.

TL;DR: See my github workflow : bats-multi-bash.yml

My Bash Script

The script in question is my Bash-TPL project, which is a light-weight shell-scripting template engine.

My Testing Framework

I chose BATS as the testing framework, as its comprehensive, tests are easy to write, and its Bash all the way down !

I had already written an extensive suite of tests, so I was confident they would reveal any compatibility issues once I figured out how to run them against older Bash versions.

Finding Older Bash Versions

In researching how I was going to run my tests against older bash versions, I discovered the Official Bash Docker Repository

Local Testing

This made running the tests on a specific bash version very easy to do on my local machine:

$ docker --rm -it -v"${PWD}:/bash-tpl" bash:3.2

bash-3.2# apk add bats diffutils
# ...
OK: 8 MiB in 21 packages

bash-3.2# cd /bash-tpl
bash-3.2# bats test

1..92
ok 1 BASH_TPL_TAG_DELIMS: Should do nothing when unset or empty
ok 2 BASH_TPL_TAG_DELIMS: Should error on invalid input
ok 3 BASH_TPL_TAG_DELIMS: Should process valid input
# OK looking good !
# ...
# DOH!
not ok 42 misc-function: escape_regex
not ok 43 misc-function: normalize_directive
# ...

Hate Compatibility Much?

So it turned out that my script only worked on Bash 5.1 :(

Checking Multiple Bash Versions

Once I identified all of the compatibility issues, I was able to work on fixing them, using the above technique to test against multiple Bash versions. Specifically, I ran tests against:

• Bash Version 3.2 (bash:3.2)
• Bash Version 4.4 (bash:4.4)
• Bash Version 5.0 (bash:5.0)
• Bash Version 5.1 (bash:5.1)

Auto-Detecting Future Issues

Having successfully fixed the compatibility issues, and making a new release, I set out to create a github workflow to run these tests for me in the future.

Workflow Research

Bash Docker Images

I had to determine if I could utilize the previously discovered Bash docker images within a github workflow.

Run a Single Bash Image

My first goal was to determine if I could run the workflow within just a single docker image.

Thanks to the well-documented Workflow Syntax For Github Guide, I discovered the jobs.<job_id>.container configuration.

jobs:
  my_job:
    container:
      image: "bash:3.2"

This will run your job steps inside the specified container !

Configuring BATS

Next I needed to install BATS into my fresh new Bash container.

I knew I could issue an apk add bats command, but I wanted to have more control over the actual version of BATS being used.

GitHub Actions Marketplace

I figured I probably wasn't the first person who wanted to run BATS in a workflow, so I searched the github actions marketplace.

Sure enough, there was already an action for configuring BATS:

• https://github.com/marketplace/actions/setup-bats-testing-framework

Most importantly, it allows you to declare which version of BATS you want to install.

For me, that was the shiny new 1.5.0:

jobs:
  my_job:
    steps:
      - name: Setup BATS
        uses: mig4/setup-bats@v1
        with:
          bats-version: 1.5.0

Multiple Bash Versions

Now that I could successfully run my tests using a single container, it was time to figure out how to use multiple containers.

Enter The Matrix

From the github workflow documentation of the jobs.<job_id>.strategy.matrix strategy:

You can define a matrix of different job configurations. A matrix allows you to create multiple jobs by performing variable substitution in a single job definition. For example, you can use a matrix to create jobs for more than one supported version of a programming language, operating system, or tool. A matrix reuses the job's configuration and creates a job for each matrix you configure.

Not Exactly New

I first learned about the github workflow matrix strategy when implementing github-repo-stats to save stat history for all of my repositories.

But What About Containers?

But I couldn't find any examples of a matrix being used for container setup.

... Favors The Bold

I decided to give a try anyway - And It Worked !

jobs:
  my_job:
    strategy:
      matrix:
        bash: ["bash:3.2", "bash:4.4", "bash:5.0", "bash:5.1"]
    container: 
      image: ${{ matrix.bash }}

Putting It All Together

Now that I had all the pieces worked out, I could finally implement my workflow.

Below is the version of the workflow at the time of posting:

bats-multi-bash.yml

name: BATS Tests

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  workflow_dispatch:

jobs:
  bats:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        bash: ["bash:3.2", "bash:4.4", "bash:5.0", "bash:5.1"]
    container: 
      image: ${{ matrix.bash }}
    steps:
      - name: Install packages
        run: apk add diffutils

      - name: Setup BATS
        uses: mig4/setup-bats@v1.2.0
        with:
          bats-version: 1.5.0

      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run bash-tpl tests
        run: bats test/

      - name: Run template tests
        run: bats test/tpl

Results

The BATS tests run successfully on all specified Bash versions, and they're fast!

Conclusion

Thank you for reading - I hope you found this post helpful. Please let me know if you have any comments or questions.

-TekWizely