On-demand running of tests via the CI pipeline

In all the CI/CD tools I've used (Jenkins, TeamCity, Bitbucket Pipelines, Gitlab), I would say Buildkite is my favourite so far 👍. Being a fan of the sayings "right tool for the right problem" and "it doesn't matter what tool you use", it still helps that your DevOps tool offer flexibility with a cleaner and minimalist interface unlike the very cluttered Gitlab 🫣.

In one of its flexible feature, it allows the user with a conditional modal before the pipeline proceeds to the next steps. They call it block step 🧱 in Buildkite.

Not all our automated tests are embedded into our build pipelines. We usually enable happy path smoke tests to make our micro-service pipelines run faster. Hence, we allow users to run ad-hoc/on-demand tests based on their preferences.

We created a standalone test pipeline called "tests"

We set the CI variables / steps with default values then call the pipeline.yml in our code repository:

  STAGING_BASE_URL: https://staging.mydomain.net   
  STAGING_PASSWORD: def456
  TEST_BASE_URL: https://test.mydomain.net 
  TEST_PASSWORD_TEST: abc123
  SPECS: tests/**/*.spec.ts # will run the whole test suite
  BUILDKITE_PARALLEL_RUN: 1 # default to 1 but can be overriden
  ENVIRONMENT: test

steps:
  - label: ':pipeline: Initiate pipeline'
    command: buildkite-agent pipeline upload .buildkite/pipeline.yml

In our code repository, we have two files, the .buildkite/pipeline.yml which triggers the block modal:

steps:
  - block: ':playwright: Run tests on demand'
    # branches: '!master'
    fields:
      - select: ':earth_americas: Select environment'
        key: 'environment'
        default: 'test'
        options:
          - label: 'test'
            value: 'test'
          - label: 'staging'
            value: 'staging'
      - text: ':test_tube: Test spec path / file (e.g. tests/api/**/*.ts)'
        default: 'tests/e2e/**/*.ts'
        key: 'specs'
      - select: ':man-running: Parallel runs'
        key: 'parallelism'
        default: '3'
        options:
          - label: '1'
            value: '1'
          - label: '2'
            value: '2'
          - label: '3'
            value: '3'
  - label: ':buildkite: Trigger tests'
    commands:
      - chmod 777 .buildkite/pipeline-adhoc.sh
      - .buildkite/pipeline-adhoc.sh | buildkite-agent pipeline upload

... and the .buildkite/pipeline-adhoc.sh, which captures the entered values in the block modal and also allows parameterising the parallelism option in Buildkite as the integer value (currently defaulted to 1) can only be updated via a script:

#!/bin/bash

set -e

ENVIRONMENT=$(buildkite-agent meta-data get "environment")
SPECS=$(buildkite-agent meta-data get "specs")
PARALLEL_RUN=$(buildkite-agent meta-data get "parallelism")

if [ "${ENVIRONMENT}" = "test" ]; then
  PASSWORD=${TEST_PASSWORD}
  BASE_URL=${TEST_BASE_URL}
else
  PASSWORD=${STAGING_PASSWORD}
  BASE_URL=${STAGING_BASE_URL}
fi

cat <<YAML
env:
  ENVIRONMENT: "${ENVIRONMENT}"
  SPECS: "${SPECS}"
  PASSWORD: "${PASSWORD}"
  BASE_URL: "${BASE_URL}"

steps:
  - label: ':playwright: Run tests'
    if: build.env('SPECS') != null # some components might not have spec files specified so need to skip running tests
    commands:
    - 'npx playwright test'
    parallelism: ${PARALLEL_RUN} # this is defaulted to > 1 so need to set PARALLEL_RUN as a CI variable for repo's with only 1 test spec file so CI doesn't fail with 'no spec files' error
    retry:
      manual: 
        allowed: false
        reason: 'Integration with other platforms means retries should not be performed for tests.'
    plugins: # pass details to Test Analytics
      - test-collector#v1.0.0:
          files: 'test-results/junit-*.xml'
          format: 'junit'
    artifact_paths:
      - 'results/**/*'
YAML

Once all these are set up, the pipeline can be triggered anytime and will stop on the step with block

Clicking the "Run tests on demand" button would take you to the modal where you can choose your own adventure 🤠 then clicking "Continue" button on the modal would trigger the tests based on your selections ✅