DEV Community: poponuts

Running Salesforce CLI (sfdx) on Buildkite pipeline

poponuts — Fri, 08 Aug 2025 11:47:32 +0000

Unlike typical web apps, Salesforce usually has its own ecosystem in terms of packaging and deploying its products via its Salesforce CLI (sdfx) package.

In my humble opinion, they are still in the infancy stage of the DevOps movement since there are still steps that require manual intervention. However, I think it's getting there in terms of trying to make everything smooth with one to two-click automation 🤖.

Here's a guide on how we did it with our current pipelines using Buildkite, which is flexible enough to allow prompts or as they call it blocks 🧱 in between steps. It required heaps of experimentation and iteration before we got to a stage where every developer is comfortable using it.

We initially used its docker image as most references on the web with SF CI/CD use this but we decided to use the npm package as its closer to how we do CI/CD for our other web apps.

Please refer to this boilerplate github repo as a guide. Ensure that the CI variables are defined in your Build Steps:

For our Continuous integration (CI):
Every change (update or create) to any git branch (master or feature) would trigger running the Buildkite pipeline for Salesforce.
The pipeline would run the following steps in sequence:

Download the @salesforce/cli npm package
Create a scratch org (temporary org necessary to run unit tests)
Run unit tests (BONUS: security tests using snyk and code coverage using Codecov)
(if master) Deploy source package to scratch org
Delete scratch org

Makefile:

test: .env
    @echo "Running sfdx test"
    openssl enc -nosalt -aes-256-cbc -d -in assets/server.key.enc -out assets/server.key -base64 -K $(DECRYPTION_KEY) -iv $(DECRYPTION_IV)
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm build "npm run test"
    @echo "Running snyk test"
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm snyk "snyk auth $(SNYK_TOKEN) && npm run snyk"
    @echo "Running codecov"
    docker-compose -f .buildkite/docker-compose.yml run --rm snyk "curl -Os https://uploader.codecov.io/latest/linux/codecov && chmod +x codecov && ./codecov -t $(CODECOV_TOKEN) --flag Apex"
.PHONY: test

For our Continuous deployment (CD):
The aim is to automate the current manual process of creating and installing packages in testing environments.

Every feature merge to master (a commit without tag) would trigger the following steps on the pipeline:

Prompt to select which component/s to create a package for along with the corresponding package version numbers and dependency configuration for omni-channel and/or test drive.

Package version numbers should follow the format MAJOR.MINOR.PATCH
When creating omni-channel and/or test drive package, by default package dependencies are not updated. To update package dependency, select the component to update and specify the core package version ID (if core package has already been created previously) or, leave the field blank to set dependency on the new core package when creating a new core package along with omni-channel and/or test drive package.
When creating core package only, selecting component/s to update package dependency and/or setting a core package dependency version ID will NOT update omni-channel and/or test drive dependency in sfdx-project.json.

Depending on the selection, update version number (and package dependency) in sfdx-project.json and create beta package.
Create scratch org
Install beta package(s)
Delete scratch org
Upload the updated sfdx-project.json as a build artifact on Buildkite. Devs have the option to download the updated sfdx-project.json from Buildkite in case errors arise in succeeding build steps.
Push the updated sfdx-project.json to a release branch
Open a Pull Request to master from the release branch

Following the above, Devs must assign a reviewer, review and merge the Pull Request. In addition, Devs must create a tag for the commit to trigger the following build steps on the pipeline:

Prompt to select which package(s) to promote to release version and install in testing environments.
Promote package(s) to release version
Install package(s) to automation org
Run e2e smoke tests
Have the option to install package(s) to UAT org

As you can see above, it's not very straightforward (more like a partial DevOps) 🤯. Hopefully, Salesforce is working towards improving this process and making lives easier for the developers.

Automating Android TV app with Nightwatch

poponuts — Thu, 25 Apr 2024 08:07:46 +0000

I joined a company that recently introduced a TV app 📺 for their platform. A curious cat 😺 like I am, I raised my hand and immediately jumped into finding ways to automate this piece of art. It is highly similar to Mobile apps but simple clicks and double taps just won't work, we need to use the Directional Pads (DPAD) 🎛️ and their relevant values (aka remote control for TV's).

I did some googling (aka 'research' in the digital world) and long story short, I ended up using Nightwatch 🦉 since we are also planning to launch it with Apple TV. Hence, there are advantages of having a single tool to automate both.

First, we need to install 🚀 the following:

Appium, the automation framework for mobile & TV apps and Nightwatch runs at the back of this.
Android Studio & SDK tools, IDE used for building and testing Android apps and used to create a TV app emulator
Create a TV app emulator from Android Studio via Device Manager
Java JDK & runtime, as required for any Android activities
Nightwatch which will prompt to create a boilerplate framework specifically for Mobile / TV apps.
Create a new folder called apps and place the .apk file there (this is the working package and if you don't have it, ask your friendly dev to give it to you).

Then, we need to build the test scripts in the project directory.
The configuration file: nightwatch.conf.js:

  'app.android.emulator': {
      extends: 'app',
      'desiredCapabilities': {
          // More capabilities can be found at https://github.com/appium/appium-uiautomator2-driver#capabilities
          browserName: null,
          platformName: 'android',
          // `appium:options` is not natively supported in Appium v1,but works with Nightwatch.
          // If copying these capabilities elsewhere while using Appium v1,make sure to remove `appium:options`
          // and add `appium:` prefix to each one of its capabilities,e.g. change 'app' to 'appium:app'.
          'appium:options': {
          automationName: 'UiAutomator2',
          // Android Virtual Device to run tests on
          avd: 'TV_1080p',
          orientation: 'PORTRAIT',
          // While Appium v1 supports relative paths,it's more safe to use absolute paths instead.
          // Appium v2 does not support relative paths.
          app: `${__dirname}/apps/tv-app.apk`,
          appActivity: 'com.tv.app.MainActivity', // include this if app does not start on its own - 
          // chromedriver executable to use for testing web-views in hybrid apps.
          // add '.exe' at the end below (making it 'chromedriver.exe') if testing on windows.
          chromedriverExecutable: `${__dirname}/chromedriver-mobile/chromedriver`,
          newCommandTimeout: 0
          }
      }
  },

NOTE: if you don't know the value for appActivity, open the .apk file in Android Studio then look for AndroidManifest.xml then search for "activity":

The test / spec file nav.spec.js:

describe('navigate the menus', function() {
    test('should be able to navigate left menu', async function(app) {
        await app.appium.pressKeyCode(20) // presses down once
            .waitForElementVisible('xpath', '//android.widget.TextView[@text="Search"]')

        await app.appium.pressKeyCode(20)  // presses down once
            .waitForElementVisible('xpath', '//android.widget.TextView[@text="Settings"]');

        await app.appium.pressKeyCode(23) // presses the centre pad to close the left menu again
            .waitForElementNotPresent('xpath', '//android.widget.TextView[@text="Home"]');
    });

    // guide on DPADS keys which you can place in a page object file
    // dpadCenter: 23,
    // dpadLeft: 21,
    // dpadRight: 22,
    // dpadUp: 19,
    // dpadDown: 20
});

NOTE: As mentioned earlier, to simulate real user experience, you need to use the remote or directional pads. Refer to this guide for the complete numeric values.

Lastly, we need to test and run if the automation actually works:

Run appium from a separate terminal with the same command
Open another terminal then open the newly-created emulator with the command emulator -avd TV_1080p
Run the test with npx nightwatch tests --env app.android.emulator and watch it work! 🎮

To inspect elements, it is recommended to download the Appium Inspector or use the browser version.

I also joined the Nightwatch Discord channel to ask questions as I had hiccups along the way.

Enjoy! 🎉

On-demand running of tests via the CI pipeline

poponuts — Mon, 24 Jul 2023 13:51:24 +0000

In all the CI/CD tools I've used (Jenkins, TeamCity, Bitbucket Pipelines, Gitlab), I would say Buildkite is my favourite so far 👍. Being a fan of the sayings "right tool for the right problem" and "it doesn't matter what tool you use", it still helps that your DevOps tool offer flexibility with a cleaner and minimalist interface unlike the very cluttered Gitlab 🫣.

In one of its flexible feature, it allows the user with a conditional modal before the pipeline proceeds to the next steps. They call it block step 🧱 in Buildkite.

Not all our automated tests are embedded into our build pipelines. We usually enable happy path smoke tests to make our micro-service pipelines run faster. Hence, we allow users to run ad-hoc/on-demand tests based on their preferences.

We created a standalone test pipeline called "tests"

We set the CI variables / steps with default values then call the pipeline.yml in our code repository:

  STAGING_BASE_URL: https://staging.mydomain.net   
  STAGING_PASSWORD: def456
  TEST_BASE_URL: https://test.mydomain.net 
  TEST_PASSWORD_TEST: abc123
  SPECS: tests/**/*.spec.ts # will run the whole test suite
  BUILDKITE_PARALLEL_RUN: 1 # default to 1 but can be overriden
  ENVIRONMENT: test

steps:
  - label: ':pipeline: Initiate pipeline'
    command: buildkite-agent pipeline upload .buildkite/pipeline.yml

In our code repository, we have two files, the .buildkite/pipeline.yml which triggers the block modal:

steps:
  - block: ':playwright: Run tests on demand'
    # branches: '!master'
    fields:
      - select: ':earth_americas: Select environment'
        key: 'environment'
        default: 'test'
        options:
          - label: 'test'
            value: 'test'
          - label: 'staging'
            value: 'staging'
      - text: ':test_tube: Test spec path / file (e.g. tests/api/**/*.ts)'
        default: 'tests/e2e/**/*.ts'
        key: 'specs'
      - select: ':man-running: Parallel runs'
        key: 'parallelism'
        default: '3'
        options:
          - label: '1'
            value: '1'
          - label: '2'
            value: '2'
          - label: '3'
            value: '3'
  - label: ':buildkite: Trigger tests'
    commands:
      - chmod 777 .buildkite/pipeline-adhoc.sh
      - .buildkite/pipeline-adhoc.sh | buildkite-agent pipeline upload

... and the .buildkite/pipeline-adhoc.sh, which captures the entered values in the block modal and also allows parameterising the parallelism option in Buildkite as the integer value (currently defaulted to 1) can only be updated via a script:

#!/bin/bash

set -e

ENVIRONMENT=$(buildkite-agent meta-data get "environment")
SPECS=$(buildkite-agent meta-data get "specs")
PARALLEL_RUN=$(buildkite-agent meta-data get "parallelism")

if [ "${ENVIRONMENT}" = "test" ]; then
  PASSWORD=${TEST_PASSWORD}
  BASE_URL=${TEST_BASE_URL}
else
  PASSWORD=${STAGING_PASSWORD}
  BASE_URL=${STAGING_BASE_URL}
fi

cat <<YAML
env:
  ENVIRONMENT: "${ENVIRONMENT}"
  SPECS: "${SPECS}"
  PASSWORD: "${PASSWORD}"
  BASE_URL: "${BASE_URL}"

steps:
  - label: ':playwright: Run tests'
    if: build.env('SPECS') != null # some components might not have spec files specified so need to skip running tests
    commands:
    - 'npx playwright test'
    parallelism: ${PARALLEL_RUN} # this is defaulted to > 1 so need to set PARALLEL_RUN as a CI variable for repo's with only 1 test spec file so CI doesn't fail with 'no spec files' error
    retry:
      manual: 
        allowed: false
        reason: 'Integration with other platforms means retries should not be performed for tests.'
    plugins: # pass details to Test Analytics
      - test-collector#v1.0.0:
          files: 'test-results/junit-*.xml'
          format: 'junit'
    artifact_paths:
      - 'results/**/*'
YAML

Once all these are set up, the pipeline can be triggered anytime and will stop on the step with block

Clicking the "Run tests on demand" button would take you to the modal where you can choose your own adventure 🤠 then clicking "Continue" button on the modal would trigger the tests based on your selections ✅

Running Gitlab on Backstage locally

poponuts — Tue, 24 Jan 2023 03:22:19 +0000

Since Backstage is now the default developer portal for global teams nowadays to uplift engineering productivity, understandably, we FOMO'd 👻 and decided to do a PoC on this open-source platform built by Spotify from scratch.

The easiest way to verify if this baby 👶 works is by integrating it with our existing tech stack.

Without my typical witty backstory, I realised that I, myself, usually ignore the noise and just go straight to the steps when I'm SPIKE'ing something (see what I did there - used an agile term 🫶). Here's the steps on how I did it locally on a bare basic boilerplate level:

Cloned and installed a local copy of the core Backstage repository with the following command:

npx @backstage/create-app@latest

Note: Follow the prompts such as running npm run dev and you should be able to access http://localhost:3000.

Installed the Backstage Gitlab plugin with the following command:

# From your Backstage root directory
cd packages/app
npm install @loblaw/backstage-plugin-gitlab

Added the following on the app-config.yaml (or app-config.local.yaml if you like to play safe):

integrations:
  gitlab:
    - host: gitlab.com
      token: ${GITLAB_TOKEN}

Note: GITLAB_TOKEN can be grabbed from your Gitlab account > User Settings > Access Tokens

Selected an existing local repository and included a catalog-info.yml then pushed it to the remote Gitlab repository (this step automates the integration) with the following details:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: k6
  description: performance tests
  annotations:
      gitlab.com/project-id: '41414141' # This must be in quotes and can be found under Settings --> General
spec:
  type: service
  owner: poponuts@test.com # This can be your team's distribution list too
  lifecycle: experimental

Note: type should have the value service to render Gitlab entity.

Refresh or re-run (npm run dev) the app then go to the service and you should be able to see all the Gitlab details

Note: If you like it more fancy and have a separate Gitlab tab as per image above, the .tsx component files should be customised.

Easy use of multiple environment variables - Cypress 10+ edition

poponuts — Mon, 05 Dec 2022 23:14:16 +0000

A couple of years back (which seemed like an eternity), I wrote this piece where we can use environment json files to easily switch between environment variables in Cypress.

However, since the use of plugin/index.ts / plugin/index.js is deprecated (or highly discouraged) now with Cypress 10+, a little tweaking is required to utilise cypress.config.js / cypress.config.ts without impacting much of the codebase (considering my rusty Cypress skills now). News flash: the legacy index file can still be used so read til the end of this 😉.

Though I'm still not a fan but we are using Typescript now 🫢 in my new company so if you are using Javascript, just do some refactoring.

Here's my new code all in cypress.config.ts file:

const localEnv = {
  "domain": "localhost:8080",
  "description": "This is my local machine!",
  "protocol": "http://",
  "subDirectory": "helloworld"
};

const testEnv = {
  "domain": "test1.domain.com",
  "description": "This is the awesome stage environment",
  "protocol": "https://",
  "subDirectory": "/helloworld"
};

const envConfig = (environment) => {
    switch(environment) {
        case 'local': return localEnv;
        case 'test':
        default: return testEnv;
    }
};

    e2e: {
        env: {
            ...envConfig(process.env.ENVIRONMENT?.toString()), // override with the environment configuration based on the value passed in CI (e.g. ENVIRONMENT=test)
            grepFilterSpecs: true,
        },
    },

Note that we can still add other environment variables that is generic to all environments like grepFilterSpecs so we don't need to include it in each of the configs.

Basically, the premise is we can only use Cypress.env on the test specs but since config.config.ts runs in node as part of the setup process, then we can't use it yet. It can be reached using process.env. So, when you use something like ENVIRONMENT=test npx cypress run on your CLI, the following will appear on the Project settings of your Cypress runner:

BONUS: The legacy index file can still be used with the following lines on cypress.config file:

const legacyConfig = require('./cypress/plugins/index');
...
    e2e: {
        setupNodeEvents(on, config) {
            return legacyConfig(on, config); // since we still use the old plugin file to override configs
        }

This is how my AWS Deep Racer model finished best in 11 seconds

poponuts — Sat, 09 Jul 2022 10:47:17 +0000

My new company flew me to Sydney 🛩️ to attend a 2-day team building event which included half of the day putting our AWS Deep Racer model into action by competing in an actual physical race.

Despite the workload, I spent time outside working hours to develop our team's model. Since I am a very competitive person (yes, I trash talk my kids when I win a Mario Kart game 🏎️ over them - teaches them resilience 👀), the motivation to win was multi-fold: bragging rights, winning cool prizes (we each got Echo dot and AWS notebooks!) and most importantly, my inclination to learn "real" machine learning (I used to be part of a team "selling" AI/ML but it's all just big data 🤷).

The awesome AWS team had a couple of workshops with everyone on how to prepare your models (creating accounts, training, simulation, racing with the company-specific community race). Non-technical or newbies don't need to be scared as the AWS console for Deep Racer is quite a user-friendly UI. We only had one track to train on which is the re:Invent 2018 so it was easier to tweak your models.

I trained multiple models (and iterations) after deep-diving into github repositories and browsing through multiple online articles

Each models were trained between an hour and three hours based on different reward functions. One of the things I learned is that longer training would not mean a better time (as it might pick up some bad habits too as you can see in the graph below where it shows a bit of downward trend at the end)

We were allowed to submit any models we have (as many times as we want) and enter into the simulation virtual race. Overall, my entry came 3rd with a time of 34.724 seconds having no lap going "off-track"

You can see above that I tried to iterate my superpumped model (sorry, I just watched the Amazon Prime series so that's the first thing that came to my mind, although I'm not a fan of Uber's toxic culture) by tweaking the values in the hyper parameters and reward functions but the version 2 did the best time (not bad considering there were 20 entries). Each team then had to download the physical car model then submit to the AWS team the day before the actual physical race.

During "race" day 🏁 (thanks AWS team for preparing the track and all the set-ups), we had to drive our models on the track one team at a time for 3 minutes each (so the car can go as many laps as possible and the best lap will be recorded). We were the last team to drive the vehicle (a.k.a. RC model with a camera) and as they say, "saved the best for last". Our car clocked 11.04 seconds, while the runner-up was 8 seconds slower (yep, big gap which validated the hard work!). AWS team also said the fastest ever was 7 seconds so our result was not bad for a first timer 😎. The other thing to note is that there is a "speed function" introduced while driving the car. The AWS team initially set it to 41% and you would need to adjust it accordingly while the car is moving (e.g. turn up the speed when it's a straight path while try to decrease it when it's curving). There were a couple of times where the car went off-track so I realised, I had to slow it down.

I know you might have skipped the babble above so for those who are just here to check how I did it, click the repository link below containing my reward function:
my awesome aws deeprace model ⬅️ ⬅️ ⬅️
... with training configuration below:

I ran it for 100 minutes (1 hour and 20 minutes) because when I iterated and changed the duration to 3 hours, it just went ballistic and did no good.

The model focused on accuracy instead of speed where it tries to stay at the centre as much as possible. It rewards the model the closer it is while it penalises if it veers away.

By no means I am taking full credit for this as I checked out a number of articles and tried out a dozen of models (hence, used up all my training credits). I used the Example 4 from the reference below with only changing the Loss type from Mean Square to Huber (by accident... but it worked out well in the end):
https://www.linkedin.com/pulse/samples-reward-functions-aws-deepracer-bahman-javadi?trk=public_profile_article_view

Integrate tests with Buildkite Test Analytics

poponuts — Sun, 26 Jun 2022 11:17:57 +0000

Buildkite introduced a new feature called Test Analytics 📈 this month and in my attempt to rank high on SEO on this subject, you're in the right article on how I did it (considering you already have access to Buildkite).

Just a bit of context (which no one cares), I just joined a scale-up where we started tracking stuff and understandably, it includes the quality of our delivery lifecycle (or else, what's the point of my existence 😂). One of the metrics I personally wanted to check is how predictable and trust-worthy our tests are. The tendency is to ignore tests (especially on the Slack alerts where it becomes normal) when there is pressure to deliver and the amount of failing tests exceed the succeeding ones. That is just unacceptable for me as a Quality engineer and something that should be closely monitored.

"You can't improve what you don't measure." — Peter Drucker

For our jest frontned unit tests, there is a readily-available integrator (which makes life so easy!) where you just simply need to install (and obviously, include in your source code repo) the npm package called buildkite-test-collector and include the reporter in your config file (e.g. jest.config.js):

reporters: [
  'default',
  'buildkite-test-collector/jest/reporter'
],
testLocationInResults: true

For our cypress end-to-end tests, it is a bit trickier as it involves a couple more steps. You need to use the default junit reporter and include the reporter in your config file (e.g. cypress.json or cypress.config.ts for Cypress 10 and above):

  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "results/junit-[hash].xml",
    "toConsole": true
  }

Once done, you need to go to your test or build pipeline (or wherever you run your tests) > Build steps (or pipeline.yml), and import the JUnit XML file. As lazy as I am (some call it smart 😎), I used the Test collector plugin in this instance.

steps:
  - label: '🧪 Run cypress tests'
    commands: 'npx cypress run --record --parallel --ci-build-id $BUILDKITE_BUILD_NUMBER'
    plugins: # pass details to Test Analytics
      - test-collector#v1.0.0:
          files: 'results/junit-*.xml'
          format: 'junit'
    artifact_paths:
      - 'results/**/*'

Once the configuration is done, include the BUILDKITE_ANALYTICS_TOKEN value on your pipeline's Build Steps or pipeline.yml

Once the tokens are set, run your pipelines and then the magic happens! 🪄

Note: Unsure why the test summary is not included in the main test suite page for the unit tests (maybe the Buildkite Gods can help us and I will update this write-up).

Click a specific test suite and the "devil (if heaps of failing tests) is in the details" 👿

This is good alternative to Cypress dashboards too but free of charge if you're already subscribed to Buildkite.

Ways to reduce execution time on automated tests

poponuts — Sun, 20 Mar 2022 00:34:52 +0000

I was asked a few months ago on potential ways in reducing test automation run time so I opened my Notion editor ✍🏻 and started compiling a list based on my experience. I've been in companies where tests were running for HOURS! (imagine the long feedback loop and re-work required if something goes wrong! 😱).

This might be handy to Quality Engineers looking for a test automation strategy. Let's start with the quick wins 💨.

✅ Use explicit waits instead of implicit waits - This is a well-known anti-pattern in the test automation community, putting a default waiting time attracts longer run-time and flakiness. Explicit waits such as waiting for an expected element to be visible should be used instead.

✅ Avoid usage of xpath (if possible) - (I understand it is a bit controversial as automation engineers rely on this regularly but hear me out) but xpath identification is generally slower than other direct attributes, so using id or name (if unique) should be the top choices. It is also harder for maintenance as it is not easily visible on the DevTools when troubleshooting issues.

✅ Include unique tag / custom attributes on selectors - Instead of traversing the whole DOM, it is usually easier to have a unique identifier for a specific element such as adding data attribute such as data-test-id on the actual source code.

✅ Run via headless mode - Headless are much faster than real / headed where the main difference is that the screen / browser is hidden. This can be achieved by CLI options (e.g. npx cypress run --headless - most frameworks would have this option).

✅ Use API / libraries to quickly generate test data - Instead of creating test data via the UI, it is significantly faster via API or libraries. Plug-ins such as faker or running API's can be included in the test scripts before any UI functionality is performed.

✅ Bypass the UI if not testing those functionalities - Similar to the above, for example, if we are not testing the login functionality, then there should be multiple ways to bypass this. Although, we can still have a separate UI test for login, the other tests (after login) can be checked without going through the long UI login process.

✅ Set a scheduled run of tests on a higher frequency - This won't reduce the run-time but would be beneficial on continuous health check. A sub-set / smoke tests can be run multiple times in a day or a longer test (e.g. regression) can be run daily overnight via the CI/CD tool.

✅ Parallelisation + Deterministic tests - Parallelisation allows higher test coverage in a shorter run-time. By making the tests "deterministic" (independent of any other tests, data, environment, etc.), it is easier to de-couple tests and run them in parallel. This can be easily configured as most CI/CD tools are capable of running parallel tests.

At the same time, let's take a step back and wear our holistic hat. The following may require longer effort and big mindset shift 🌟.

✅ Choose the tooling & language - Modern automation tools such as Webdriverio, Playwright & Cypress have smarter ways to make the tests run faster and are scripted via NodeJS / Javascript which allows asynchronous calls so it does not need to wait sequentially for each commands.

✅ Clean up the database / environment if possible - If there is a way to easily spin up an environment with minimal dataset or a script can clean up the database, then this will speed up the run-time. Assistance from environment owners and building up automated jobs can help fast-track these activities. Note: If you are worried about performance difference with a Production environment, then it's best practice to use a dedicated Production-like environment for Performance testing.

✅ Built-in documentation and having developers contribute to the code - In the modern testing, even the developers are involved in the test automation. As they would usually know good coding patterns, they can help improve the speed of the tests. Pair programming / testing and continuous collaboration would help build a scalable and robust test automation framework. Include in-line comments help resources to understand the scripts better.

✅ Reduce by increasing unit tests as compared to e2e - Following the test pyramid principles, there should be a significantly higher coverage of unit and integration tests as compared to e2e tests. Both unit and integration tests run significantly faster too as compared to tests that involve the UI. It is ideal to focus on high-value E2E tests that are critical to the customers and stakeholders. Duplicate tests that have been covered in the earlier level of tests should be considered for streamlining the test suite.

Again, this is just on my experience and there might be some better ways so just pick the bits and pieces that fit your need 👋.

Trigger your E2E tests from another repo's Bitbucket Pipeline build

poponuts — Fri, 06 Aug 2021 03:50:33 +0000

There are two school of thoughts on whether End-to-end (E2E) tests should be on same repository or on a separate repository. If its taking you weeks or months to debate on that with your team, then there could be a middle-ground 🤠.

If there is a separate team / resource working on the E2E tests and another team / resources working on the source code, then how about a compromise to make it work? This is by simply triggering the E2E tests on any changes made in the source code.

This is highly possible by using Atlassian's Pipe - trigger pipeline.

On the application / source code's bitbucket-pipeline.yml file, simply add the following as a step:

    - step: &run-automated-tests
        name: Run E2E and API automated tests
        script:
          - pipe: atlassian/trigger-pipeline:4.2.1 # https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/trigger-pipeline
            variables:
              BITBUCKET_USERNAME: $BITBUCKET_USERNAME
              BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
              REPOSITORY: 'my-cypress-tests'
              # ACCOUNT: '<string>' # Optional
              BRANCH_NAME: 'master' # Optional
              # CUSTOM_PIPELINE_NAME: '<string>' # Optional
              # PIPELINE_VARIABLES: '<json>' # Optional
              # WAIT: 'true' # Optional - if set to true, it waits for the triggered pipeline to complete. If the triggered pipeline fails, this pipe will also fail.
              # WAIT_MAX_TIMEOUT: '<string>' # Optional
              # DEBUG: '<boolean>' # Optional

BITBUCKET_USERNAME can be hard-coded to your Atlassian username or this can be set under the Repository settings > Repository variables section of Bitbucket to encrypt secrets.
BITBUCKET_APP_PASSWORD would need to be generated via Personal settings > App passwords then creating a new app password.

Alternatively, a shared account can be created by your Atlassian administrator that can be used across CI/CD pipeline and would not need dependency on resources going in and out of the team / company.

To add alerting, simply integrate the tests with Slack (include as many stakeholders as possible 😂

This is a good opportunity to say "Sorry Mr. Software Engineer, you can't proceed until you fix your breaking changes (and no, I'm not fixing my tests either!) Just kidding! Collaborate with your developers as much as possible to resolve the issues 😇

Uses Bitbucket Pipeline but too poor to buy Cypress Dashboard for parallelisation? Look no further...

poponuts — Sun, 01 Aug 2021 06:40:51 +0000

If you've been a "Google" developer like me then I'm sure by now, you've heard of sorry-cypress, an alternative open-source tool for the Cypress dashboard that usually comes with a price tag 🤑.

I tried using that on my previous company and was doing my head in for a few months before I finally gave in that's impossible to use that using an old CI pipeline like TeamCity.
Fast-forward to today where my current company is using Bitbucket Pipeline and I thought to myself "finally, just a a unicorn yaml file that does it all!"... well, not too fast, big boy! 👀

Unfortunately, the way docker is used in the Atlassian suite is a bit different (In the interest of "that's not what I came here for", I won't explain it here the way the support team explained it to me - as if anyone cares?). As such, I swallowed my ego (that I can do it on my own), and finally gave in sending a desperate support ticket to the Atlassian team begging to assist me! ✌🏻

To their credit, the Atlassian team went beyond my expectations in terms of support (don't worry, I gave them the highest rating possible in terms of service!). After much iteration and never-ending back and forth time-consuming correspondence ⛑, the one below finally worked in achieving Parallelisation of Cypress tests in Bitbucket Pipeline
using the bitbucket-pipelines.yml:

image: cypress/base:14.16.0
options:
  max-time: 60

definitions:
  caches:
    npm: $HOME/.npm
    cypress: $HOME/.cache/Cypress

  # job definition for running tests
  steps:
    - step: &tests
        name: Run all tests
        size: 2x # (optional) recommended by Atlassian team but adds up to your minutes
        services:
          - docker
        caches:
          - npm
          - node
          - cypress
          - docker
        script:
          - docker run -d -p 1234:1234 agoldis/sorry-cypress-director # port bindings to enable free parallelisation
          - apt update && apt install parallel -y # workaround recommended by Atlassian team that runs background job services in Linux where Bitbucket Pipeline is hosted
          # include below the no. of parallel instances you want your (obviously, the more the better but it intermittently messes up reporting
          - echo "npx cypress run --record --key $CYPRESS_DASHBOARD_KEY --parallel --ci-build-id $BITBUCKET_BUILD_NUMBER --browser chrome --spec cypress/integration/$TESTS;" >> jobs.txt
          - echo "npx cypress run --record --key $CYPRESS_DASHBOARD_KEY --parallel --ci-build-id $BITBUCKET_BUILD_NUMBER --browser chrome --spec cypress/integration/$TESTS;" >> jobs.txt
          - echo "npx cypress run --record --key $CYPRESS_DASHBOARD_KEY --parallel --ci-build-id $BITBUCKET_BUILD_NUMBER --browser chrome --spec cypress/integration/$TESTS;" >> jobs.txt
          - 'parallel -j 3 :::: jobs.txt' # >> logs.log'
        artifacts: # store any generates images and videos as artifacts
          # - logs.log # (optional) include this for better troubleshooting
          - jobs.txt
          - cypress/screenshots/**
          - cypress/videos/**
          - cypress/test-reports/*.xml

    - step: &tests-non-parallel # non-parallel tests
        name: Run all tests
        caches: # not needed as taking it from the docker image
          - node
          - cypress
        script:
          - cypress_version=$HOST npx cypress run --browser chrome --spec cypress/integration/**/*.js # all tests
        artifacts:
          # store any generates images and videos as artifacts
          - cypress/screenshots/**
          - cypress/videos/**
          - cypress/test-reports/mocha/*.json
          - cypress/test-reports/*.xml

pipelines:
  default:
  - step:
      name: Prepare dependencies
      caches:
        - npm
        - cypress
        - node
      script:
        - npm ci
        - npx @bahmutov/print-env BITBUCKET # let's see the environment variables right away
        # - npm audit # check security vulnerabilities of npm packages
        - sed -i -E 's@(api_url:) (.*)@\1 "http://localhost:1234"@g' /*/.cache/Cypress/*/Cypress/resources/app/packages/server/config/app.yml # reconfigure cypress agent to localhost
  - step: *tests

  custom:
    non-parallel: # non-parallel tests
      - step:
          name: Prepare dependencies
          caches:
            - npm
            - cypress
            - node
          script:
            - npm ci
            - npx @bahmutov/print-env BITBUCKET # let's see the environment variables right away
            # - npm audit # check security vulnerabilities of npm packages
            - sed -i -E 's@(api_url:) (.*)@\1 "http://localhost:1234"@g' /*/.cache/Cypress/*/Cypress/resources/app/packages/server/config/app.yml # reconfigure cypress agent to localhost
      - step: *tests-non-parallel

A few things to note:

You can hard-code the CYPRESS_DASHBOARD_KEY and TESTS or set them under the Repository settings > Repository variables section of Bitbucket (as long as you have admin access). CYPRESS_DASHBOARD_KEY can also be a fixed / random value and you don't necessarily need to create an account in Cypress dashboard to generate one. TESTS on the other hand can be set to **/*.js to run all tests or can be more specific e.g. cypress/integration/mytest.js
On your cypress.json file, ensure that the projectId key it set to the same value in CYPRESS_DASHBOARD_KEY
e.g. "projectId": "8ghcrb"
BITBUCKET_BUILD_NUMBER is already a reserved variable in Bitbucket and you would need this parameterised so the parallel tests know its running on the same build.

If you are having xvfb issues, you want to include the command on your scripts as recommended by Cypress team:

- apt-get install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb

Move over Zephyr, XRAY, TestRail... How to use JIRA itself as a test management tool

poponuts — Fri, 30 Jul 2021 12:08:18 +0000

Remember the days when ~~HP Quality Centre~~ ALM used to be the gold standard of test management... well, look at how the tides have changed. Reminds me of how Blockbuster 🎟️ was too late to compete with Netflix and now, they've gone bust. Makes you wonder why Microfocus bought the tool just a couple of years back when everyone is already moving into a more agile, less admin-like (robotic) logging of test cases. It doesn't help that its slow-performing and costs a leg and arm to license it every year 🤯.
In any case, that's not the point of this article (I know, I always gets side-tracked 🤦‍♂️).
My point is that no one needs test management anymore (sheepishly looking at my non-existent sponsors 🤪)! The tendency is teams treat the quantity of test cases as the success metrics for quality, which I beg to disagree! The number of test cases you have does not mean you will deliver a quality product. Imagine, having a single test case for login then another for making a payment - understandably, they don't have the same amount of weight. As the saying goes, the devil is in the detail. Sad truth is having a thousand of tests in a fancy test report always sell 😔!
While there is movement moving towards testing embedded as part of development process and reporting is done on the CI / automation stage, unfortunately, not all companies are there yet. Hence, some might still need to have a test management tool. However, instead of paying for a JIRA plug-in like Zephyr or XRAY, or an integration like qTest or TestRail, you can use JIRA itself. Less context-switching, less steps 🥂!
Spoiler alert: You need to have JIRA admin rights to make this happen.
Here's how with my "Marie Kondo"-inspired workflow:

Go to Project settings > Issue types then create an issue type called Test (and/or sub-task called Sub-test if you would like it visible under the User Story)
Go to Project settings > Fields > Custom fields then create a new custom field with field type Text Field (multi-line)called Test Scenarios
Go to Project settings > Field configurations then find the newly-created custom field. Click Renderers hyperlink then select Wiki Style Renderer to update
Click the horizontal ellipsis then select Contexts and default value
Under the Default Value section, click the Edit Default Value hyperlink
Enter the sample table below then hit Set Default to save

||Test scenario||Test data||Screenshot||Status||
|Scenario 1|Test data 1|<upload here>|Pass / Fail|

Go to Project settings > Workflows then create a workflow called Test Workflow and create a flow like below:
Link the workflow, issue type, and custom field together by googling how to do this (sorry, this part is something that would prolong my article unnecessarily so I'll leave it up to you to sort it out 🤪)

You would have something like this where the tests are visible (and no multiple layers / steps needed outside the user story to see the statuses of your test scripts / cases)

Note: You can do this (if you're still transitioning from waterfall to agile) or if you want to do hardcore "Agile" then just include them inside the User story's "Acceptance criteria" 😁

Hour glass? Pyramid? Trophy? It's simply test strategy!

poponuts — Sat, 19 Jun 2021 09:03:36 +0000

If you are involved in startups or scale-ups long enough, thousand pages of test strategy and million lines of code of ~~regression~~ e2e test suite won't cut it 😴.

Modern quality engineering thinking suggests that the percentage of testing done at the earliest and lowest level (unit tests) should be significantly higher than integration and e2e tests. Industry-standard says it should be around 70% of unit tests while the rest of the tests should share around 30% of coverage 🤔. IMHO, it depends! Without understanding your company's current landscape, then there is no silver bullet or "one-fit-for-all" approach. What Quality gurus are preaching nowadays: there are no best practices but only good practices; Good practices in the context of the company's current and future structure, skillset, tech stacks.

Usually, if you are someone new to the company and you were given the golden hand to resolve the pain points, you get excited to be the white knight to save the day (maybe I was just talking about myself here 😝). There is nothing wrong with that as that is human nature but, most of the time, it's not as straight-forward as it seems.

Focusing too much on numbers / quota sways a resource to think that reaching the agreed threshold is good enough even if those tests don't add value. Furthermore, focusing on tooling / framework leans toward just following the requirements / happy path flows, without doing further analysis on what are the other possible scenarios (e.g. edge or negative cases).

In any case, after reviewing my current company's needs, I came up my own version of the test pyramid or on what I see as more of an umbrella with sprinkles of rain water hovering above it (my editing skills jus't cant find time to draw an umbrella-looking diagram). Regardless of what kind of figure / diagram you make out of it, this is simply our long-term continuous test strategy - which is appropriate enough for a lean + agile company like ours. This will eventually evolve as the company grows and structures change ♻️.

My attempt to use the objects and colours on Google Slides:

In my view, percentage of test coverage in each stage matters. However, what matters more is the time spent on two zones I have identified: Spending maximum amount of time on PROACTIVE ZONE while spending the minimum amount of time on the REACTIVE ZONE.

The ability to plan on delivering quality feature / product has been lost with Agile's thinking of delivering fast.
At the same time, as the role of testers are slowly evolving into being the caretaker of the quality of the whole delivery lifecycle, they (or even your developers who loves testing 😏#sarcasm) should be involved in all levels of this diagram. As you can see, testing is continuous and the time spent on quality as early as possible, the better (I can't emphasise how shift-left mindset sets the culture of delivery quality products! 🕶️).

Just to reiterate, the test strategy should evolve based on the changing needs of the company. It shouldn't be fixed on a simple test pyramid. Furthermore, what works for your old company doesn't mean it would work for your new one (my diagram had multiple iterations too). It's worth mentioning that it depends on the products / service offerings too. For example, if your company is offering API and web hook integrations then understandably, there should be more integration and contract tests involved, if your company is offering retail catalogues then there should be more visual or UI tests involved, and so on and so forth.

In hopefully my next article, I will try to deep dive into those test frameworks.