DEV Community: Mayank Shukla

Optimizing CircleCI Configuration: Minimize config.yml size with Dynamic Configuration

Mayank Shukla — Wed, 03 Jan 2024 10:15:14 +0000

We have a wdio-based automation project that does end-to-end functional testing. We use CircleCI as the CI/CD tool to run automation regression. When setting up a project with CircleCI, we need to enter all configuration details in the config.yml file under the .circleci folder. Earlier, we only had staging regression and prod regression jobs and their respective workflows in the config.yml file.

As mentioned in this article, we also started doing screenshot testing for our application. With time, our configuration file expanded and became cluttered, accommodating various workflows like staging sanity, staging regression, prod regression, performance score testing, and screenshot testing jobs.

We explored various approaches to reduce the size of our configuration file. Later, we came across the Dynamic Configuration implementation of CircleCI. In the beginning, it appeared perplexing, but we eventually understood the purpose of using continuation and path-filtering orbs. In this article, we will share how we utilised the continuation orb to fulfil our requirements with CircleCI.

What were the requirements?

Below were our requirements:

Weekly trigger staging-regression job on staging branch.
Weekly trigger prod-regression job on master branch.
Weekly trigger staging-regression-other-features job on staging branch.
Weekly trigger prod-regression-other-features job on master branch.
Weekly trigger sign-up, registration, and setup jobs on staging branch.
As per the need, we should be able to trigger screenshot testing and sanity jobs.

Dynamic Configuration implementation

We created multiple YAML files for our different jobs and workflows. We used the generate_config.js script to choose the workflow file that we wanted to execute. We implemented the below-outlined structure for organising yml files for our different jobs.

Script to generate a new configuration YAML file based on the parameter

We created a script to generate a new YAML configuration based on the job we want to execute. The new configuration file will be a child configuration file.

/** Below script is implemented in generate_config_script.js file*/

const fs = require('fs');
const path = require('path');

function readFile(fileName) {
  const filePath = path.join(__dirname, `${fileName}.yml`);

  return fs.readFileSync(filePath, 'utf8');
}

function generateCircleCIConfig() {
  const workflow = process.env.CONFIG_TXT;
  const workflowConfig = readFile(workflow);

  console.log(workflowConfig);
}

generateCircleCIConfig();

The above script is executed from the parent configuration file config.yml present in the .circleci folder.

version: 2.1

setup: true

orbs:
  node: circleci/node@5.0.2
  continuation: circleci/continuation@0.1.2

parameters:
  config_txt:
    type: string
    default: staging_sanity

jobs:
  setup:
    docker:
      - image: cimg/node:18.16.1-browsers
    executor: continuation/default
    steps:
      - checkout
      - run:
          name: Generate config
          command: |
            node ./circleci-jobs/generate_config_script.js > generated_config.yml
          environment:
            CONFIG_TXT: << pipeline.parameters.config_txt >>
      - continuation/continue:
          configuration_path: generated_config.yml

workflows:
  setup:
    jobs:
      - setup

In the above parent config.yml file,

setup: true: This line indicates that the current config.yml file is a parent configuration file.
As mentioned in CI Documentation, the continuation orb is used to execute the child configuration file created based on the output of generate_config_script.js.
config_txt is a parameter that is passed to generate_config_script.js, based on which it creates a child configuration file. The default value is staging_sanity, so if no explicit parameter value is passed on CircleCI, then it by default triggers the staging-sanity job.

Also, make sure to define the parameters declared in the parent config.yml file in the child generated_config.yml file as well. Otherwise, it will throw an error Unexpected argument. The official documentation says Pipeline parameters declared in the setup configuration must also be declared in the continuation configuration. These parameters can be used at continuation time.

In the parent config.yml file, only one workflow can be defined, which is responsible for generating the child configuration file. For our project, we required defining multiple workflows for our jobs, and to achieve the same with dynamic configuration, we used triggers.

Schedule Triggers on CircleCI

The next step was to create a scheduled pipeline to periodically trigger a specific job.
Pipelines are scheduled to periodically run a specific job by adding a trigger for each of the jobs that need to be executed.
Triggers are added from the Triggers Section present under the Project Settings option for the CircleCI Project.

For each of the jobs mentioned above, we created a new Trigger to periodically run a specific job.

Challenges faced in configuring triggers

We filled out the mandatory information needed to create triggers, but we faced challenges in deciding on the value for the field Repeats Per Hour for the trigger. We were unsure whether this field would trigger the job a certain number of times every hour based on the field's name. We intended our trigger to run just once on the designated day. When we got in touch with CircleCI support, they clarified that the field decides how many times a trigger would be executed in each hour chosen in the Start Time (UTC) selection.

For e.g., if the selected "Start Time (UTC)" is 3:00 (UTC) and 7:00 UTC and "Repeats Per Hour" is set to 2, then the trigger will be executed twice between each selected time, i.e., it will be executed twice between 3:00 (UTC) and 4:00 (UTC) and twice between 7:00 UTC and 8:00 (UTC).

As mentioned in the CI Documentation, setting the start time for a trigger does not guarantee the job will be triggered precisely at the "Start Time" selected. This means that for a trigger, when we select "Start Time" as 6:00 AM UTC and "Repeat Per Hour" as 1, the job will be triggered once between 6:00 AM UTC and 7:00 AM UTC and will not always be triggered exactly at 6:00 AM UTC.

Next, we declared the pipeline parameter named config_txt for our trigger and set its value to the <child_job>.yml file name containing the workflow that we wanted to trigger. For instance, in order to trigger the production regression workflow, we set the config_txt parameter value to prod_regression. Similarly, we created triggers for each workflow that we wanted to execute.

We thought the trigger was scheduled for now. However, the trigger did not execute as scheduled. There was an error block-unregistered-user in our workflow. On investigation, we noticed that for all the triggers, we have "Attribution Actor" set as Scheduled Actor (Scheduling System), which is the default. As mentioned in the article, if the option "Prevent unregistered user spend" is turned ON under the "Usage Control" tab present in the "Plan" section of the project, then the workflow won't be triggered for unregistered users and error block-unregistered-user will be displayed.

This was exactly the case with our project, and to resolve the issue, we selected "Attribution Actor" as one of the users registered in the project. With this, we were able to trigger and execute our workflow successfully.

We have created a demo project circleci-dynamic-configuration-demo that you can clone and explore for better understanding.

References:

Setup Image Comparison Service of WDIO to make UI Testing more impactful

Mayank Shukla — Thu, 04 May 2023 11:37:58 +0000

In the previous article, we talked about WebdriverIO. A tool that has become popular very fast in the field of test automation due to its powerful advantages. In this article, we will talk about one of its advantages that has advanced our test automation suite. We have an application that helps our clients develop stunning websites. Whenever a new feature is developed that may impact the UI of the site, WebdriverIO can help us test the change in the UI of the sites. WebdriverIO provides a service called as wdio-image-comparison-service, which performs pixel-by-pixel image comparison.

We already have a wdio-based automation project that does end-to-end functional testing. We expanded it to screenshot test all the pages for thousands of sites. We explored various tools for image comparison. We then discovered that WebdriverIO furthermore offers an image comparison tool. Then we began looking into how we could incorporate this "wdio-image-comparison-service" into our already-running automation project.

Set-up

Referring to the official doc for wdio-image-comparison-service, we installed the module:

npm install --save wdio-image-comparison-service

We added the below options to the Test runner services in wdio.conf.js file:

services: [
  'chromedriver',
  [
    'image-comparison',
    // The options
    {
      // Some options, see the docs for more
      baselineFolder: join(process.cwd(), './tests/'),
      formatImageName: '{tag}-{logName}-{width}x{height}',
      screenshotPath: join(process.cwd(), '.tmp/'),
      savePerInstance: true,
      autoSaveBaseline: true,
      blockOutStatusBar: true,
      blockOutToolBar: true,
      disableCSSAnimation: false,
      ignoreColors: true,
      ignoreTransparentPixel: true,
    },
  ],
],

Above is the default setup. With this default setup, we thought that we had completed configuring the setup and the next step was to create a spec file to compare screenshots.

browser.saveScreen("homepage") function saves the screenshot as a baseline image with the name homepage.png.
browser.checkScreen("homepage") function saves the screenshot of the current page and compares it with the baseline image having the same name i.e. homepage.png.

However, we encountered an issue with the default setup. When we attempted to create a spec to compare screenshots of the two different doodles, it did not detect any mismatches, and the spec passed. We will explain this issue in more detail below:

We created a spec file called image_compare.js as follows:

describe("Image Comparison", () => {
  it("saves the baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
    await browser.saveScreen("homepage");
  });

  it("compares with baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");

    const result = await browser.checkScreen("homepage");

    expect(result).toEqual(0);
  });
});

In the above spec, we were comparing screenshots of the Teacher's Day - 2021 homepage and Teacher's Day - 2022 homepage. And we were expecting some mismatch percentage. But the spec got passed. When we investigated, we found that wdio-image-comparison-service saved the baseline image under .tmp folder. The .tmp folder has actual as a subfolder. Under the actual folder, it saved the baseline image.

In our case, when we ran the spec, it overrode Teacher's Day - 2021 homepage.png with Teacher's Day - 2022 homepage.png inside .tmp/ folder. Inside ./tests/ folder also, it saved Teacher's Day - 2022 homepage.png. When we investigated more on this, then we got to know that internally this package uses fs-extra plugin and it has limited Windows support. But, running the same script on Mac or Linux did not result in any issues or failures. We also found out that this is an existing issue. We have also added a comment there about what could be a workaround.

Workaround

As we previously said, the baseline picture is saved by 'wdio-image-comparison-service' by default under the '.tmp' folder. In 'wdio.conf.js', we removed the following code from the'services' section:

baselineFolder: join(process.cwd(), './tests/'),
screenshotPath: join(process.cwd(), '.tmp/'),

Then, we specified the testOptions in the it block where screenshots are compared.

const testOptions = {
  actualFolder: join(process.cwd(), './.tmp/checkActual'),
  baselineFolder: join(process.cwd(), './.tmp/actual'),
  diffFolder: join(process.cwd(), './.tmp/testDiff'),
  returnAllCompareData: true,
};

checkActual: The current image will be saved in this folder. It is called as actualFolder.
actual: The baseline image will be saved in this folder. It is called as baselineFolder.
diffFolder: This image gets created with all the mismatches highlighted after the two screenshots are compared.
returnAllCompareData: By default, check methods of this service returns only mismatch percentage. When this option is passed as true, then it returns the current image filename, folders that contain the path of actual, baseline, and diff, and misMatchPercentage.

After these modifications, the updated code is as follows:

wdio.conf.js

services: [
  'chromedriver',
  [
    'image-comparison',
    // The options
    {
      // Some options, see the docs for more
      savePerInstance: true,
      autoSaveBaseline: true,
      blockOutStatusBar: true,
      blockOutToolBar: true,
      disableCSSAnimation: false,
      ignoreColors: true,
      ignoreTransparentPixel: true,
    },
  ],
],

image_compare.js

import { join } from 'path';

describe("Image Comparison", () => {
  it("saves the baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
    await browser.saveScreen("homepage");
  });

  it("compares with baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");

    const testOptions = {
      actualFolder: join(process.cwd(), './.tmp/checkActual'),
      baselineFolder: join(process.cwd(), './.tmp/actual'),
      diffFolder: join(process.cwd(), './.tmp/testDiff'),
      returnAllCompareData: true,
    };

    const result = await browser.checkScreen("homepage", testOptions);

    expect(result).toEqual(0);
  });
});

When we passed testOptions as a parameter to the checkScreen function, then wdio easily identified where our baseline images are saved and with which image to compare.

Now, if you run the above spec, then it will properly do the comparison, highlight the mismatch, and return the mismatch percentage.

Teacher's Day - 2021	Teacher's Day - 2022

Image Difference

The wdio-image-comparison-service is not just limited to capturing and comparing screens, it can also compare the full page screen, and elements, and also checks if the website supports the keyboard's TAB key. Please refer to wdio-image-comparison-service options.

Mobile Screenshot Testing

If screenshot testing for mobile devices is needed, then just resizing your browser is never a good idea because this module compares visuals of what the end user is going to see. Wdio Devtools can be helpful in such cases. Wdio Devtools is in itself a distinct, extensive subject that offers additional benefits.

We just need to pass service as devtools in wdio.conf.js file. We can emulate a specific device by await browser.emulateDevice('iPhone X'). Please refer to Device emulation.

We have created a small demo project wdio-image-comparison-demo that you can clone for better understanding. You can explore other options available for wdio-image-comparison-service.

References:

WebdriverIO supports Chaining without multiple await statements

Mayank Shukla — Wed, 16 Feb 2022 15:14:58 +0000

Ever since WebdriverIO got launched, major companies adopted this tool for automation. It became popular very fast due to its powerful advantages. Since the launch, there have been lots of changes and improvements being made to the tool. In this article, we'll be discussing one of the improvements that have really helped us in writing automation scripts in async mode.

WebdriverIO is asynchronous by nature. Earlier, WebdriverIO used to provide the ability to run commands in sync mode using node-fibers. However, due to some breaking changes in Chromium, WebdriverIO discontinued the support for sync mode. Please refer Sync vs. Async Mode and this issue for more information.

The test used to look like this:

With Sync mode:

describe('Sync mode', () => {
  it('does not need await', () => {
    $('#myBtn').click(); // Chaining works

    // Chaining works here when using Chain Selector
    $("//div[@class='field']").$("//input[@type='email']").getTagName(); 
  })
})

With Async mode:

describe('Async mode', () => {
  it('needs await', async () => {
    await (await $('#myBtn')).click(); // Needs await keyword twice for chaining

    // Similarly in the case below, await keyword is used thrice while using Chain Selector
    await (await (await $("//div[@class='field']").$("//input[@type='email']"))).getTagName();
  })
})

As you can see in the above example, for chaining await keyword is been used more than once. This can be confusing for someone who is not familiar with the async/await concept.

WebdriverIO comes with element chaining support now

Since v7.9, WebdriverIO started supporting element chaining. The same async code now can be written as follows:

describe('Async mode', () => {
  it('needs await', async () => {
    await $('#myBtn').click(); 

    await $("//div[@class='field']").$("//input[@type='email']").getTagName();
  })
})

Now the question comes,

Here we are awaiting $("//div[@class='field']") which means $("//div[@class='field']") returns a promise. So how come we can call .$("//input[@type='email']") on the promise returned by $("//div[@class='field']")?

Similar question I faced before while writing test cases. For this, I raised an issue on GitHub, and it was answered by WebdriverIO developer team. Let's look into it in more detail below.

WebdriverIO returns a Promise compatible object

WebdriverIO returns a promise compatible object which allows you to do either:

const emailDivField = await $("//div[@class='field']");
const emailFieldTag = await emailDivField.$("//input[@type='email']").getTagName();

const emailFieldTag = await $("//div[@class='field']").$("//input[@type='email']").getTagName();

Promise compatible objects are custom objects which implement the promise interface.

Caveats

I was upgrading my project with latest version of WebdriverIO i.e. v^7.16.13. Lessons that I learnt are:

Chaining won't work for parameters:

If you are passing element as a parameter along with await keyword, then in this case chaining won't work.

Example:

Here, we have Utility class where we have defined a generic function isDisplayed(). This function validates if the list of elements, passed as argument args, are visible in the UI.

class Utility {
  async isDisplayed(args) {
    for (const element of args) {
      let isDisplayed = element.isDisplayed();

      if (!isDisplayed) return false;
    }

    return true;
  }
}

export default new Utility();

We have LoginPage PageObject class. LoginPage has 2 elements pageHeading and contactHeading.

class LoginPage {
  get pageHeading() {
    return $("//h2[text()='Login Page']");
  }
  get contactHeading() {
    return $("//h4[text()='Contact Us']");
  }
}

export default new LoginPage();

In the spec file, we are validating if those elements are visible in the UI.

describe('Login screen', () => {
  it('displays all expected headings', async () => {
    const elements = [
      await loginPage.pageHeading,
      await loginPage.contactHeading,
    ];
    let boolVal = await utility.isDisplayed(elements);
    expect(boolVal).to.be.true;
  });
});

In the Utility class, below line

let isDisplayed = element.isDisplayed(); // Returns Promise

won't work as we are calling isDisplayed() method in a synchronous manner. But it actually needs await keyword.

let isDisplayed = await element.isDisplayed(); // Works

Also passing await keyword along with parameters won't work. You can skip using await keyword while passing parameters as shown below:

const elements = [
  loginPage.pageHeading,
  loginPage.contactHeading,
];
let boolVal = await utility.isDisplayed(elements);

Use of async/await to handle array of promises

When you want to fetch an array list, then use Promise.all

async getDropdownOptions() {
  const dropdownOptions = await this.dropdownOptions;
  return await Promise.all(
    dropdownOptions.map(function (option) {
      return option.getText();
    }),
  );
}

await Promise.all won't resolve promise inside function

async getDropdownOptions() {
  const dropdownOptions = await this.dropdownOptions;
  return await Promise.all(
    dropdownOptions.map(function (option) {
      return option.getText().split('\n')[1]; // Error 
    }),
  );
 }

In above example, you will get an error that says getText().split() is not a function. The reason is getText() function returns a promise. You cannot perform a string operation on a promise.

async getDropdownOptions() {
  const dropdownOptions = await this.dropdownOptions;
  return await Promise.all(
    dropdownOptions.map(async function (option) {
      return (await option.getText()).split('\n')[1];
    }),
  );
}