Efficient Autotesting with Qase: Playwright and GitLab CI Integration

The approach to maintaining test documentation and the tools chosen for this purpose are crucial parts of the development process that directly affect the product's quality. Keeping test documentation up to date is particularly important. Qase can be an excellent tool for this. Additionally, it helps integrate manual testing with automated testing, and test cases with their execution.

In this article, we will explore the implementation of Qase integration with Playwright and GitLab CI, as used at SmartHead: from project creation to automated testing reports.

Benefits of using Qase, Playwright, and GitLab CI

Qase offers extensive functionality for test management, including test cases, checklists, test runs, and plans, as well as integrations with bug tracking tools. Combined with Playwright and GitLab CI, this provides the following benefits:

• Centralized test management: A user-friendly interface for maintaining all test cases, plans, and runs.
• Automatic test creation and synchronization: Ensure test cases are up to date and minimize the risk of duplication.
• Easy to integrate and use: Playwright simplifies writing and maintaining tests, while GitLab CI automates their execution with every code change.
• Detailed reporting: Qase provides detailed test execution reports to help you quickly identify and fix bugs.
• Run tests locally or in CI: Ability to run tests locally or in a CI/CD Pipeline with automatic synchronization of each run and test case information in Qase.
• Running Pipelines directly from Qase: A more integrated and automated testing process. Reports are available in both Qase and GitLab after each run.
• Attachment support: Ability to attach screenshots, logs, test pass records, and other files to reports for detailed analysis of bugs and issues.

Let's move on to the implementation.

Step 1: Set up a project in Qase

Create a new project in Qase

1. Register or log in to your Qase account.
2. Create a new project, specifying its name and other necessary data.

Setting up automatic creation of test cases

In the project settings (Settings > Run), enable the option to automatically create new test cases when there are no matches. This will help avoid duplication and keep the test suite up to date.

When this option is disabled, the test case name from the code will be displayed in the test run, but it will not be added to the repository. When enabled, if the test case is not marked as existing (no ID from Qase is specified), the new test case will be added to the project repository automatically.

Create test cases in a project

1. Create one or two test cases. We will need them for the next steps.
2. Specify the basic parameters such as the name, description, and steps of the test execution.

Step 2: Customize the Qase + Playwright integration

Install Playwright and playwright-qase-reporter

1. Navigate to the root directory of the project.
2. Following the instructions, install Playwright.
3. Following the instructions, install the reporter for Qase.

Set up the integration in Qase

1. Log in to Qase.
2. Go to the "Apps" section.
3. Locate and select Playwright.
4. Set up the integration and save the token you created. You can give it any meaningful name.

Set up the configuration in Playwright

Create an .env file. This file will store our sensitive data for local launch. We will add the copied token from Qase there.

After installing Playwright, playwright.config.ts will appear in the root directory of the project. Open the file and add the following configuration:

import { defineConfig, devices } from '@playwright/test';
require('dotenv').config({ path: '.env' }); // Required for using variables from .env

export default defineConfig({
  testDir: './tests', // Your test repository
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,

  reporter: [
    ['list'],
    ['playwright-qase-reporter',
      {
        testops: {
          project: 'TEST', // Your project code from Qase
          api: {
            token: process.env.QASE_TOKEN, // Token from .env
          },
          run: {
            id: process.env.QASE_RUN_ID,
            complete: true,
          },
          uploadAttachments: true,
        },
      }],
  ],

  use: {
    trace: 'on-first-retry',
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    }
  ],

});

Don't forget to add the .env file to .gitignore so you don't accidentally commit the token to the repository.

About ID in Qase runs

id: process.env.QASE_RUN_ID

Specifying QASE_RUN_ID in the configuration is necessary to prevent the creation of different runs in Qase. Without this setting, the automatic run created in Qase and the run that starts the pipeline will be assigned different IDs. By specifying QASE_RUN_ID, we ensure that the tests from the pipeline are added to the existing run in Qase, if it already exists.

Specify the IDs of the cases in the tests

To link tests from Playwright with tests from the Qase repository, we need to specify their IDs in the tests. For convenience, it's better to use names identical to those in Qase. Regardless, the test run will contain the name of the case whose ID you specified.

Example code of a case test with qase.id():

// test.spec.ts
import { test, expect } from '@playwright/test';
import { qase } from 'playwright-qase-reporter';

test('Display title of Playwright main page', async ({ page }) => {
  qase.id(1); // ID of the test case in Qase

  // Step #1
  await page.goto('https://playwright.dev/');
  await expect(page).toHaveURL('https://playwright.dev/');
  // Expected result: The page opens

  // Step #2
  const title = await page.getByText('Playwright enables reliable end-to-end testing for modern web apps.');
  await expect(title).toBeVisible();
  // Expected result: The title is displayed
});

Also for the sake of example, let's write a test where we don't specify qase.id():

test('Test that is not in the repository', async ({ page }) => {
  // Step #1
  await page.goto('https://playwright.dev/');
  await expect(page).toHaveURL('https://playwright.dev/');
  // Expected result: The page opens

  // Step #2
  const title = await page.getByText('Playwright enables reliable end-to-end testing for modern web apps.');
  await expect(title).toBeVisible();
  // Expected result: The title is displayed
});

Running tests

Run the tests using the command, specifying your path to the tests:

QASE_MODE=testops npx playwright test tests/test.spec.ts

Checking your results in Qase

After successfully running the tests, you will get a link to your run in Qase in the console.

Directly from the console, we can follow the link to view the test run we just executed. You can share this link by making it public or keeping it private. Additionally, you can see detailed information about all the steps of each individual test case, the duration of its execution, and follow the link to view the Pipeline, and so on.

The case specified with the ID will pull the existing case from Qase. It will include the steps you have provided and all other information.

A case without specifying an ID will be pulled up by default in this state:

The project repository will also include any new test cases that were not previously present. These will be created thanks to the setting we added earlier. If you disable this setting, the tests will still run, but no new test cases will be added to the repository.

A new test case will not be created in subsequent runs. However, it's advisable to include the ID in the code of each test case to prevent confusion.

Step 3: Integrate Qase with GitLab CI

Create a project on GitLab if it hasn't been created already. Ensure that CI/CD is enabled under Settings -> General -> Visibility, project features, permissions.

Customize variables in GitLab

With the .env file created, we can store configuration data locally. However, to run tests in CI, this data needs to be shared with GitLab. GitLab provides CI/CD Variables for this purpose.

To add your own variable, you need to:

1. Go to Settings -> CI/CD -> Variables. Here we can save sensitive data similar to the .env file to use later.
2. Click Add variable and in the window that opens, enter QASE_TOKEN in Key and token values in Value.
3. Save the data :)

Customize the .gitlab-ci.yml file in a project

There are numerous articles available on CI/CD that can assist you in selecting the optimal approach. Here, I’ll cover only what is needed for integration with Qase.

Add the .gitlab-ci.yml file to the project root. A minimal configuration example for .gitlab-ci.yml might look like this:

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.42.1-jammy
  script:
    - npm ci
    - npx playwright install
    - QASE_MODE=testops npx playwright test tests/test.spec.ts

1. Using the official Docker image from Playwright.
2. Installing dependencies.
3. Running tests with reports sent to Qase.

After configuring your .gitlab-ci.yml file, remember to push the code to your GitLab repository. Once pushed, the pipeline will execute, and Qase will generate a test run report similar to the one generated previously.

Create an automatic test-run in Qase

After all the settings are done, we can start a run in Qase, which will create a Pipeline in GitLab and run the tests.

1. Go to Qase.
2. Open the Test Runs section and click Start new test run.
3. In the test run settings, enter the required data, select GitLab integration, specify the required project, branch, and run the tests.

A report will also be generated after the run is completed.

Thus, we executed the automated tests in three ways: locally, by committing code changes to the repository, and by initiating an automated test run in Qase. In each scenario, a report was submitted to Qase.

Conclusion

Building software effectively relies on leveraging modern automation and testing tools. Integrating Qase, Playwright, and GitLab CI forms a powerful combination that enhances the development and testing processes.

Playwright provides rapid and reliable test automation, allowing thorough testing of application functionality across various platforms and browsers, thereby reducing testing durations. GitLab CI ensures that tests are automatically run upon each code update, facilitating early detection and resolution of bugs. Integrating with Qase provides a comprehensive ecosystem within a single platform—from test case creation to automated execution and detailed reporting.