DEV Community

loading...
Cover image for Github Action: How to parallelize tests dynamically by folder?

Github Action: How to parallelize tests dynamically by folder?

slashgear_ profile image Antoine Caron Originally published at slashgear.github.io on ・1 min read

Once again, I gave myself a challenge to learn how to use Github Action. I tried to see if it was possible to dynamically launch a job for each folder in a directory.

We often want to run tests in parallel to reduce the time of continuous integration. Often tools like Jest or CucumberJS have parallelization features, but these often face the limits of the power available on the machines on which you want to run them.

tests/
  user/
    foo.feature
    bar.feature
  account/
    example.feature
  live/
    a.feature
    video.feature
Enter fullscreen mode Exit fullscreen mode

I wondered if it wouldn’t be more efficient to run the tests in parallel directly with the CI tool so that only small runners would be needed. In the previous example, I would like to be able to run in parallel on a separate runner for each folder in test/.

A Solution with Github action

name: CI

on:
  push:
    branches: [main]

jobs:
  directories: # Job that list subdirectories
    runs-on: ubuntu-latest
    outputs:
      dir: ${{ steps.set-dirs.outputs.dir }} # generate output name dir by using inner step output
    steps:
      - uses: actions/checkout@v2
      - id: set-dirs # Give it an id to handle to get step outputs in the outputs key above
        run: echo "::set-output name=dir::$(ls -d */ | jq -R -s -c 'split("\n")[:-1]')"
        # Define step output named dir base on ls command transformed to JSON thanks to jq
  loop:
    runs-on: ubuntu-latest
    needs: [directories] # Depends on previous job
    strategy:
      matrix:
        dir: ${{fromJson(needs.directories.outputs.dir)}} # List matrix strategy from directories dynamically
    steps:
      - run: echo ${{matrix.dir}}
Enter fullscreen mode Exit fullscreen mode

It’s up to you to adapt it to your needs!

Discussion (0)

pic
Editor guide