DEV Community: Yaroslav

AI-Powered Unit Tests: A Step-by-Step Approach with ChatGPT

Yaroslav — Mon, 24 Jul 2023 15:39:28 +0000

Hello, Community.

I am working as QA Automation / SDET, and I want to share my experience using ChatGPT for writing unit tests with specific examples.

Unit tests are an essential part of the software development process, allowing us to verify the correctness of individual components of the program. Well-written unit tests help ensure code stability and reliability, facilitate refactoring, and improve the overall product quality.

One way to use ChatGPT is to create templates for unit tests. This model can help you formulate general testing rules that can be used to create multiple test cases.

For example, you can ask the model to generate test templates to check the correctness of form validation, exception handling, or interactions with the database. This allows you to quickly create new test cases based on predefined templates.

However, writing unit tests can be a resource-intensive and not very exciting process that may require a lot of time. In my interactions with developers and QA professionals, I noticed that most of them prefer to write unit tests manually and do not enjoy tasks related to writing them.

In this topic, I will provide examples of how I significantly reduced the time spent on writing unit tests.

Task:

There is a method that creates a selector based on a given attribute and matcher

export function getSelectorWithAttributeType(
 key: string,
 attributeType: AttributeType,
 attributeMatcher?: AttributeMatcher,
) {
 const attributeMatcherString = attributeMatcher
   ? attributeMatcher
   : AttributeMatcher.EQUALS;
 return `[${attributeType}${attributeMatcherString}"${key}" i]`;
}

where AttributeType is a string, and attributeMatcher is a comparison literal.

And there is a set of methods that use this method as a wrapper for specific attributes, for example:

export function getRoleSelector(
 key: string,
 attributeMatcher?: AttributeMatcher,
) {
 return getSelectorWithAttributeType(
   key,
   AttributeType.ROLE,
   attributeMatcher,
 );
}

There are more than 10 such methods, and it's good to have at least one unit test for each. While they might not be modified frequently, if one of them breaks, many tests across multiple projects may start failing.

Let's try to train ChatGPT on this task so that we don't need to do anything other than copy/pasting.

First, we describe the task to ChatGPT:

Next, we get the code block as a response:

import { getSelectorWithAttributeType, AttributeMatcher } from './your-file';

describe('getSelectorWithAttributeType', () => {
  it('should return selector with EQUALS attribute matcher when no attribute matcher is provided', () => {
    const key = 'value';
    const attributeType = 'data-key';
    const selector = getSelectorWithAttributeType(key, attributeType);
    expect(selector).toBe('[data-key="value" i]');
  });

  it('should return selector with the provided attribute matcher', () => {
    const key = 'value';
    const attributeType = 'data-key';
    const attributeMatcher = AttributeMatcher.CONTAINS;
    const selector = getSelectorWithAttributeType(key, attributeType, attributeMatcher);
    expect(selector).toBe('[data-key*="value" i]');
  });

  it('should return selector with case-insensitive matching', () => {
    const key = 'value';
    const attributeType = 'data-key';
    const attributeMatcher = AttributeMatcher.EQUALS;
    const selector = getSelectorWithAttributeType(key, attributeType, attributeMatcher);
    expect(selector).toBe('[data-key="value" i]');
  });
});

It doesn't look bad, but not all matchers were used.

Next, we get the code block:

import { getSelectorWithAttributeType, AttributeMatcher } from './your-file';

describe('getSelectorWithAttributeType', () => {
  it('should return selector with CONTAINS attribute matcher', () => {
    const key = 'value';
    const attributeType = 'data-key';
    const attributeMatcher = AttributeMatcher.CONTAINS;
    const selector = getSelectorWithAttributeType(key, attributeType, attributeMatcher);
    expect(selector).toBe('[data-key*="value" i]');
  });

  it('should return selector with EQUALS attribute matcher', () => {
    const key = 'value';
    const attributeType = 'data-key';
    const attributeMatcher = AttributeMatcher.EQUALS;
    const selector = getSelectorWithAttributeType(key, attributeType, attributeMatcher);
    expect(selector).toBe('[data-key="value" i]');
  });

  it('should return selector with STARTS_WITH attribute matcher', () => {
    const key = 'value';
    const attributeType = 'data-key';
    const attributeMatcher = AttributeMatcher.STARTS_WITH;
    const selector = getSelectorWithAttributeType(key, attributeType, attributeMatcher);
    expect(selector).toBe('[data-key^="value" i]');
  });

  it('should return selector with ENDS_WITH attribute matcher', () => {
    const key = 'value';
    const attributeType = 'data-key';
    const attributeMatcher = AttributeMatcher.ENDS_WITH;
    const selector = getSelectorWithAttributeType(key, attributeType, attributeMatcher);
    expect(selector).toBe('[data-key$="value" i]');
  });
});

Now it's more or less what we need. We adjust imports, reuse key and attributeType in expect, and the first part of the work is done.

Next task:

As we are using the base function in wrappers, we only need to check that the base function was called.

Next, we get the code block:

import { getDataAidSelector, getSelectorWithAttributeType, AttributeType, AttributeMatcher } from './your-file';

describe('getDataAidSelector', () => {
  it('should call getSelectorWithAttributeType with key and attributeMatcher for DATA_AID', () => {
    const key = 'value';
    const attributeMatcher = AttributeMatcher.CONTAINS;
    const getSelectorSpy = jest.spyOn(getSelectorWithAttributeType, 'getSelectorWithAttributeType');
    getDataAidSelector(key, attributeMatcher);
    expect(getSelectorSpy).toHaveBeenCalledWith(key, AttributeType.DATA_AID, attributeMatcher);
    getSelectorSpy.mockRestore();
  });

  it('should call getSelectorWithAttributeType with key and default attributeMatcher for DATA_AID', () => {
    const key = 'value';
    const getSelectorSpy = jest.spyOn(getSelectorWithAttributeType, 'getSelectorWithAttributeType');
    getDataAidSelector(key);
    expect(getSelectorSpy).toHaveBeenCalledWith(key, AttributeType.DATA_AID, undefined);
    getSelectorSpy.mockRestore();
  });
});

spyOn doesn't fit in this case because getSelectorWithAttributeType is not an object.

Let's adjust the task:

ChatGPT Response:

import { getDataAidSelector } from './yourModule'; // import the module containing the function
import { getSelectorWithAttributeType, AttributeType } from './your-file'; // import the module containing getSelectorWithAttributeType

jest.mock('./your-file', () => ({
  ...jest.requireActual('./your-file'),
  getSelectorWithAttributeType: jest.fn(),
}));

describe('getDataAidSelector', () => {
 afterEach(() => {
    jest.clearAllMocks();
  });

  it('should call getSelectorWithAttributeType with the specified parameters', () => {
    const key = 'testKey';
    const attributeMatcher = AttributeMatcher.CONTAINS;

    getDataAidSelector(key, attributeMatcher);

    expect(getSelectorWithAttributeType).toHaveBeenCalledWith(
      key,
      AttributeType.DATA_AID,
      attributeMatcher
    );
  });

  it('should call getSelectorWithAttributeType with only the key parameter', () => {
    const key = 'testKey';

    getDataAidSelector(key);

    expect(getSelectorWithAttributeType).toHaveBeenCalledWith(
      key,
      AttributeType.DATA_AID,
      undefined
    );
  });
});

Now it looks good. The bot correctly wrote the names, guessed the attribute name in the object, and even used a different matcher than in my example, which is good practice.

Now I can generate the entire test suite without reviewing the next cases the bot generates for this scenario. And, of course, I saved a lot of time because all these actions took me about 15-20 minutes.

Sometimes there are problems because ChatGPT often loses context, and the training process has to start from scratch.

Conclusion:

In this topic, I shared my experience of using ChatGPT to optimize the writing of unit tests with specific examples.

Thanks to the help of the bot, you can quickly generate the basic test template and fill it with specific data.

Additionally, ChatGPT can suggest variations of tests and adjust the code based on input data.

It is essential to note that ChatGPT cannot completely replace a developer and their experience in writing tests. However, it can significantly save time by simplifying routine tasks and generating basic test templates.

From Instability to Reliability: Using Puppeteer and Bubanai for e2e stabilization

Yaroslav — Fri, 21 Jul 2023 10:45:39 +0000

Dear Community

I worked as QA Automation / SDET at WiX and I want to share my experience using Puppeteer and stabilising e2e tests in TypeScript.

My last project was EditorX - a platform for creating responsive websites using drag & drop technology. Over the years, we've written a significant number of e2e tests (2000+) in TypeScript, but we consistently faced stability issues in our e2e tests due to most user interaction logic being executed on the frontend, and the consequences of actions may occur some time after the action actually was performed.

Puppeteer, in general, executes it's functions much faster than the system reacts to commands. Initially, we used basic Puppeteer methods and native Jest assertions, but over time, we realised that we needed something more to achieve build stability.

What we decided to change

We have implemented the Continuous Integration process on our project. The entire test suite runs on a ready pull request, and the code can only be merged into the master branch after passing all tests. The requirement for success rate (SR) is not less than 99.6% (excluding failures caused by real bugs or infrastructure issues). If a test does not meet the stability criteria, it is skipped. To address the SR issues and improve the stability of our e2e tests, we developed our own utilities within our project and later noticed that other teams were facing similar stability issues that we had in the past.

So, we decided to extract them into an open-source library called Bubanai.

Bubanai translates from Hebrew as a puppeteer.

Features provided by Bubanai

This open-source library provides the following capabilities:

Simplified Puppeteer methods by adding expectations inside common methods like click and hover, etc.
Fast access to element attributes/properties.
Expectations for elements, for example, waiting for an element to have a certain attribute or part of it; waiting for an element's position not to change, etc.
Expectations for executing functions based on given conditions.
Context handling methods.
Collection handling methods.
Simplified drag & drop.
Logging for high-level test methods.
Scroll utilities.Keyboard interactions.
Console & Network listeners.
Utilities for initialized nested elements (useful for early versions of Puppeteer).
Utilities for dropdowns.
etc.

Typical problems this library solves:

Example 1

Let's say you have a component that changes its color after a certain time delay following a click action. You know exactly the color you're setting it to, but after the change, it does not immediately take on the desired state.

import { getElementBackgroundColor, waitForObjectsToBeEqual } from 'bubanai-ng';

...
const expectedColor = { r: 200, g: 200, b: 200 };
await colorPickerDriver.setColor(expectedColor);
await waitForObjectsToBeEqual(() => 
getBackgroundColor(context, element), expectedColor);
...

With this piece of code, we ensure that we wait for the condition to be met with the minimum possible time interval (default retry interval is 0.5s between function calls).

Example 2

You have a panel from which you add elements to the builder using drag & drop. When you drag an element from the panel and press Enter during the drag, the panel should close. But if you drag an element inside the panel area, it should not close.You need to position the element beneath the panel area (i.e., perform an intermediate drag step and action).

import { getCenter, dragElementToPoint } from 'bubanai-ng';

...
const panelCenter = getCenter(panel);
const stageCenter = getCenter(stageBounding);
await dragElementToPoint(context, element, panelCenter, { tempSteps:
{ point: stageCenter, action: () => pressEnter(), }
});
...

Example 3

You need to write a test for using a hotkey. You have a lasso effect that changes the drag & drop logic when a specific keyboard combination is pressed.

import { getBoundingBox, dragElementBy, holdAndExecute } from 'bubanai-ng';

...
const elementBounding = await getBoundingBox(element);
const action = async () => dragElementBy(context, element, 100, 10);
await holdAndExecute(['Shift', MetaKeys.ControlOrCommand], action);
const boundingAfter = await getBoundingBox(element);

expect(boundingAfter.x).toBe(elementBounding.x + 100);
expect(boundingAfter.y).toBe(elementBounding.y);
...

Example 4

In the CI logs, TypeScript is transpiled into JavaScript, and the failure lines in the test differ from TypeScript (or the assertion is thrown somewhere deep in the lower level). You need to understand which functions were called at the high level and with which arguments in the test file.

import { log } from 'bubanai-ng';

export class ColorModalDriver {

@log
async setColor(color)
{...}

@log
async openModal()
{...}

@log
async closeModal()
{...}
}

Test file:

...
await colorModal.openModal();
await colorModal.setColor({ r: 20, g: 20, b: 20 });
await colorModal.closeModal();
...

Logs:

22/05/23 13:01:43 Calling ColorModalDriver.openModal();
22/05/23 13:01:53 Calling ColorModalDriver.setColor({ "r": "20", "g": "20", "b": "20" });
22/05/23 13:02:15 Calling ColorModalDriver.closeModal();

Example 5

You have a structure that can be either inside an iframe or just located in the general context of the page. The library allows you to manipulate contexts without unnecessary pain.

export class VideoPlayerDriver extends ElementBaseDriver
{...}

Test file:

...
const playerFrame = await getFrameByUrl(page, 'video-player');
let videoPlayer = new VideoPlayerDriver(page, playerFrame);// play is executed in frame context
await videoPlayer.play();
await goToPreview();

videoPlayer = new VideoPlayerDriver(page);// play is executed on page context
await videoPlayer.play();
...

Installation

To start using Bubanai in your project, simply add the dependency to your package.json:

"bubanai-ng": "^2.1.1"

To use the library, just import the necessary methods:

import { click, getText } from 'bubanai-ng';
Most methods work with Xpath, CSS selectors or ElementHandle objects.

const { type, getText, getFrameByName } = require('bubanai-ng');
const puppeteer = require('puppeteer');
const assert = require('assert');

(async () =>
{const browser = await puppeteer.launch();
const page = await browser.newPage();

const newTextValue = 'Additional content. ';
const areaSelector = 'body';
const frameSelector = 'mce_0_ifr';

await page.goto('http://the-internet.herokuapp.com/tinymce');
const frame = await getFrameByName(page, frameSelector);
const currentText = await getText(frame, areaSelector);
await type(newTextValue, frame, areaSelector, {}, { clearInput: false } ,page,);

const newText = await getText(frame, areaSelector);
assert.equal(newText, newTextValue + currentText);
await browser.close();
})();

Bubanai also provides advanced capabilities, allowing you to customize element search and perform more complex actions in Puppeteer-based frameworks. You can find more examples in the library's tests.

Documentation for the library can be found on GitHub.

Note: currently Puppeteer versions higher than 14.3.0 aren't supported.

Bubanai can be useful both for beginners and experienced automation engineers working with Puppeteer. By using this library, you can simplify your test writing process and ensure build stability.

We are actively developing the library and open to collaboration. If you have ideas, suggestions, or additions, please feel free to contribute to our project as a contributor.