DEV Community: SmartHead

Efficient Autotesting with Qase: Playwright and GitLab CI Integration

Natalia Kudrachinskaya — Mon, 15 Jul 2024 10:58:21 +0000

The approach to maintaining test documentation and the tools chosen for this purpose are crucial parts of the development process that directly affect the product's quality. Keeping test documentation up to date is particularly important. Qase can be an excellent tool for this. Additionally, it helps integrate manual testing with automated testing, and test cases with their execution.

In this article, we will explore the implementation of Qase integration with Playwright and GitLab CI, as used at SmartHead: from project creation to automated testing reports.

Benefits of using Qase, Playwright, and GitLab CI

Qase offers extensive functionality for test management, including test cases, checklists, test runs, and plans, as well as integrations with bug tracking tools. Combined with Playwright and GitLab CI, this provides the following benefits:

Centralized test management: A user-friendly interface for maintaining all test cases, plans, and runs.
Automatic test creation and synchronization: Ensure test cases are up to date and minimize the risk of duplication.
Easy to integrate and use: Playwright simplifies writing and maintaining tests, while GitLab CI automates their execution with every code change.
Detailed reporting: Qase provides detailed test execution reports to help you quickly identify and fix bugs.
Run tests locally or in CI: Ability to run tests locally or in a CI/CD Pipeline with automatic synchronization of each run and test case information in Qase.
Running Pipelines directly from Qase: A more integrated and automated testing process. Reports are available in both Qase and GitLab after each run.
Attachment support: Ability to attach screenshots, logs, test pass records, and other files to reports for detailed analysis of bugs and issues.

Let's move on to the implementation.

Step 1: Set up a project in Qase

Create a new project in Qase

Register or log in to your Qase account.
Create a new project, specifying its name and other necessary data.

Setting up automatic creation of test cases

In the project settings (Settings > Run), enable the option to automatically create new test cases when there are no matches. This will help avoid duplication and keep the test suite up to date.

When this option is disabled, the test case name from the code will be displayed in the test run, but it will not be added to the repository. When enabled, if the test case is not marked as existing (no ID from Qase is specified), the new test case will be added to the project repository automatically.

Create test cases in a project

Create one or two test cases. We will need them for the next steps.
Specify the basic parameters such as the name, description, and steps of the test execution.

Step 2: Customize the Qase + Playwright integration

Install Playwright and playwright-qase-reporter

Navigate to the root directory of the project.
Following the instructions, install Playwright.
Following the instructions, install the reporter for Qase.

Set up the integration in Qase

Log in to Qase.
Go to the "Apps" section.
Locate and select Playwright.
Set up the integration and save the token you created. You can give it any meaningful name.

Set up the configuration in Playwright

Create an .env file. This file will store our sensitive data for local launch. We will add the copied token from Qase there.

After installing Playwright, playwright.config.ts will appear in the root directory of the project. Open the file and add the following configuration:


 typescript
import { defineConfig, devices } from '@playwright/test';
require('dotenv').config({ path: '.env' }); // Required for using variables from .env

export default defineConfig({
  testDir: './tests', // Your test repository
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,

  reporter: [
    ['list'],
    ['playwright-qase-reporter',
      {
        testops: {
          project: 'TEST', // Your project code from Qase
          api: {
            token: process.env.QASE_TOKEN, // Token from .env
          },
          run: {
            id: process.env.QASE_RUN_ID,
            complete: true,
          },
          uploadAttachments: true,
        },
      }],
  ],

  use: {
    trace: 'on-first-retry',
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    }
  ],

});

Don't forget to add the .env file to .gitignore so you don't accidentally commit the token to the repository.

About ID in Qase runs

id: process.env.QASE_RUN_ID

Specifying QASE_RUN_ID in the configuration is necessary to prevent the creation of different runs in Qase. Without this setting, the automatic run created in Qase and the run that starts the pipeline will be assigned different IDs. By specifying QASE_RUN_ID, we ensure that the tests from the pipeline are added to the existing run in Qase, if it already exists.

Specify the IDs of the cases in the tests

To link tests from Playwright with tests from the Qase repository, we need to specify their IDs in the tests. For convenience, it's better to use names identical to those in Qase. Regardless, the test run will contain the name of the case whose ID you specified.

Example code of a case test with qase.id():


 typescript
// test.spec.ts
import { test, expect } from '@playwright/test';
import { qase } from 'playwright-qase-reporter';

test('Display title of Playwright main page', async ({ page }) => {
  qase.id(1); // ID of the test case in Qase

  // Step #1
  await page.goto('https://playwright.dev/');
  await expect(page).toHaveURL('https://playwright.dev/');
  // Expected result: The page opens

  // Step #2
  const title = await page.getByText('Playwright enables reliable end-to-end testing for modern web apps.');
  await expect(title).toBeVisible();
  // Expected result: The title is displayed
});

Also for the sake of example, let's write a test where we don't specify qase.id():



test('Test that is not in the repository', async ({ page }) => {
  // Step #1
  await page.goto('https://playwright.dev/');
  await expect(page).toHaveURL('https://playwright.dev/');
  // Expected result: The page opens

  // Step #2
  const title = await page.getByText('Playwright enables reliable end-to-end testing for modern web apps.');
  await expect(title).toBeVisible();
  // Expected result: The title is displayed
});

Running tests

Run the tests using the command, specifying your path to the tests:


 bash
QASE_MODE=testops npx playwright test tests/test.spec.ts

Checking your results in Qase

After successfully running the tests, you will get a link to your run in Qase in the console.

Directly from the console, we can follow the link to view the test run we just executed. You can share this link by making it public or keeping it private. Additionally, you can see detailed information about all the steps of each individual test case, the duration of its execution, and follow the link to view the Pipeline, and so on.

The case specified with the ID will pull the existing case from Qase. It will include the steps you have provided and all other information.

A case without specifying an ID will be pulled up by default in this state:

The project repository will also include any new test cases that were not previously present. These will be created thanks to the setting we added earlier. If you disable this setting, the tests will still run, but no new test cases will be added to the repository.

A new test case will not be created in subsequent runs. However, it's advisable to include the ID in the code of each test case to prevent confusion.

Step 3: Integrate Qase with GitLab CI

Create a project on GitLab if it hasn't been created already. Ensure that CI/CD is enabled under Settings -> General -> Visibility, project features, permissions.

Customize variables in GitLab

With the .env file created, we can store configuration data locally. However, to run tests in CI, this data needs to be shared with GitLab. GitLab provides CI/CD Variables for this purpose.

To add your own variable, you need to:

Go to Settings -> CI/CD -> Variables. Here we can save sensitive data similar to the .env file to use later.
Click Add variable and in the window that opens, enter QASE_TOKEN in Key and token values in Value.
Save the data :)

Customize the `.gitlab-ci.yml` file in a project

There are numerous articles available on CI/CD that can assist you in selecting the optimal approach. Here, I’ll cover only what is needed for integration with Qase.

Add the .gitlab-ci.yml file to the project root. A minimal configuration example for .gitlab-ci.yml might look like this:


 yaml
stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.42.1-jammy
  script:
    - npm ci
    - npx playwright install
    - QASE_MODE=testops npx playwright test tests/test.spec.ts

Using the official Docker image from Playwright.
Installing dependencies.
Running tests with reports sent to Qase.

After configuring your .gitlab-ci.yml file, remember to push the code to your GitLab repository. Once pushed, the pipeline will execute, and Qase will generate a test run report similar to the one generated previously.

Create an automatic test-run in Qase

After all the settings are done, we can start a run in Qase, which will create a Pipeline in GitLab and run the tests.

Go to Qase.
Open the Test Runs section and click Start new test run.
In the test run settings, enter the required data, select GitLab integration, specify the required project, branch, and run the tests.

A report will also be generated after the run is completed.

Thus, we executed the automated tests in three ways: locally, by committing code changes to the repository, and by initiating an automated test run in Qase. In each scenario, a report was submitted to Qase.

Conclusion

Building software effectively relies on leveraging modern automation and testing tools. Integrating Qase, Playwright, and GitLab CI forms a powerful combination that enhances the development and testing processes.

Playwright provides rapid and reliable test automation, allowing thorough testing of application functionality across various platforms and browsers, thereby reducing testing durations. GitLab CI ensures that tests are automatically run upon each code update, facilitating early detection and resolution of bugs. Integrating with Qase provides a comprehensive ecosystem within a single platform—from test case creation to automated execution and detailed reporting.

Shifting Frontend to the Server

Ramil Aminov — Mon, 18 Sep 2023 13:44:39 +0000

In the past, web applications used to be backend-centric, where HTML pages were generated on the server, and templates were used to add dynamic data.

As time passed, the demand for interactivity on web pages grew, leading to active advancements in frontend technologies, notably through the adoption of the Single Page Application (SPA) approach. Frameworks evolved to the point where the frontend became a separate application, designed and developed independently by separate teams and hosted separately as well.

This separation, while having its advantages, leads to several problems:

The code sent to the browser becomes heavy and grows as the application's functionality increases.
Much of the logic is moved to the client-side. For example, routing is implemented from scratch and bypasses browser mechanisms.
Much of the logic is duplicated on the frontend and backend. For example, validation and access control.
It's necessary to design and develop an API, an additional layer to synchronize frontend and backend, even where the backend only serves the interface.
The division of specialists into frontend and backend developers in teams creates overhead costs, dilutes the understanding of the final task, and responsibilities.

However, the principles of how networks and browsers work have not changed: from the user's perspective, web services still appear as complete applications. Frameworks like Next.js are well-suited for such "holistic" development. Their capabilities were significantly limited until recently, but the latest versions of Next.js and React are changing the game.

React introduced server components, and Next.js implemented them along with a new approach to routing called App Router. Essentially, now some React components run on the client, while others run on the server, together forming a single web page. Server components can interact directly with the database, and the API layer becomes optional, replaced by Next.js routing mechanisms. It's now possible to create a full-fledged web application without a backend in the traditional sense. Nevertheless, of course, this does not mean that the backend is not needed at all times. On the contrary, the backend can now focus on its more backend-specific tasks without mixing them with presentation logic.

By shifting some of the frontend processing and rendering logic to the server-side, we can attain the following benefits:

Accelerated initial page loading times, as the server can pre-render content before delivering it to the client.
Enhanced search engine optimization (SEO).
Mitigation of security risks associated with client-side code, as sensitive logic can be safeguarded on the server.
Increased code reusability between frontend and backend components, streamlining the development process.

Interestingly, for instance, Ruby On Rails, primarily a backend framework, also pushes the boundary of separation, but in a slightly different way. Its authors propose Hotwire as an alternative to SPA, which involves decomposing and streaming individual parts of HTML.

Both approaches converge on one point: a significant portion of frontend logic is executed on the server, and we don't lose the benefits of SPA. Thus, the division between the client-side and server-side is not equivalent to the division between frontend and backend. Frontend is a part of the application responsible for interacting with the user and servicing that interaction, including on the server.

History is progressing in a spiral. More code is executed on the server again, but now we can build client applications of any complexity and beauty with less effort and fewer side effects.

Anchor Links in SPA

Nikita Tikhonov — Tue, 18 Jul 2023 14:00:00 +0000

Single-page applications (SPA) are a popular approach to building modern websites. And it is not just a passing trend: SPAs have a number of advantages over traditional applications.

The disadvantages of this approach are also known. For example, SPAs may experience search engine optimization (SEO) issues, applications may take a long time to open on the first launch, and can burden users' devices.

But there are less obvious problems that can arise when developing a SPA. One such problem is broken anchor links.

Anchor links are used to navigate within a single page. They also allow you to link to a specific section of an external page. For instance, you can share a link to your landing page and immediately scroll the user to the section with information about the company.

If you try to implement anchor links in a simple SPA, they simply will not work. When following such a link, the browser will display the very beginning of the page and will not scroll to the target section.

Why is this happening?

When directly following an anchor link from external resources, the browser scrolls the page to the desired section immediately after the DOMContentLoaded event occurs. This means that the document has been analyzed, the DOM is ready, and the synchronous and deferred scripts have been loaded and have just started executing. This is how anchor links work in Chrome and Safari, at least. At this moment, it is possible that the required element is simply not present in the document because not all resources have been loaded yet and not all scripts have had a chance to execute.

And during transitions between pages within a single application (that is, when using, for example, react-router in a single-page application), the browser attempts to navigate to the section by changing the URI before the new page has time to render.

If you follow the anchor link within the same page, then most likely there will be no difficulties, since in this case all page elements have already been drawn.

To solve this problem, you can write a script that will scroll the page to the intended element after the DOM has undergone all changes.

You can also take a ready-made package, for instance, react-router-hash-link for react-router. It fixes a number of problems related to the work of anchors in SPA on React. There are similar implementations for other popular frameworks.

However, the best option would be to render a full HTML page from the server using server-side rendering (SSR) or a static site generator (SSG). This can be done using Next.js, Gatsby, etc. In this case, when the browser tries to navigate to the desired section, the DOM will already be ready and the target element will be available.

This way looks more natural. We are not inventing anything new, but using existing mechanisms implemented in browsers long before single-page applications. With the help of modern solutions, we can 'restore' the original working principle of our websites, while maintaining the benefits of the SPA.

Decomposition can be excessive

Ramil Aminov — Fri, 24 Mar 2023 15:37:11 +0000

It is commonly believed that decomposition is a good thing. It is indeed useful: it allows us to eliminate uncertainty by breaking down a large, unclear problem into smaller, manageable parts. During the decomposition process, we can better understand the scope of work and even design along the way.

However, excessive decomposition can be dangerous. It creates the illusion that a feature consists solely of the technical tasks we have broken it down into.

Decomposition incurs overhead costs, not only for creating and working on individual tasks but also for maintaining separate lifecycles and implementing and rolling out each task separately.
The requirement for decomposition can lead to premature design and planning. In order to decompose, design decisions are made without all the necessary information. And when new information appears, it may not be corrected. We may feel compelled to implement the previously decomposed plan.
Decomposition creates an illusion of control. Doing half of the subtasks means you’ve completed half of the feature. Completing all the subtasks means you’ve completed the entire feature. This is not the case. To complete a feature, it is not enough to complete all of its subtasks. The feature is usually more than just the sum of its parts. There is emergent behavior, a value that is given by the combination of the results of the subtasks and which is not present in the simple sum of these results. To complete the feature, this value must also be implemented.
Often, decomposition simply doesn’t make sense. For example, if a feature is simple enough and can be completed in one go.

Feature decomposition is relevant when we break it down by delivery: into pieces that are themselves the end result in terms of product or a use case.
There is no need to decompose a feature into works (for example, “develop a frontend”, “develop a backend”, or “write a migration”). It is better to decompose it into separate “pieces of value”.

Decomposition should not be a substitute for documentation or a task description. The description should make it clear what should be the result and what should not be forgotten. But not necessarily each of these items should be a separate subtask.

Another case is when the task is formulated not in terms of value, but in terms of work to be done. In this case, it may be appropriate to decompose it into independent sub-works.

With the seemingly obvious benefits of decomposition, it can easily become a “fifth wheel”. Like any other management tool, it should be used consciously.

Quality instead of Quality Control

Ramil Aminov — Thu, 18 Aug 2022 06:46:00 +0000

When discussing quality, complying with a particular set of requirements usually comes to mind. A client, analyst, or someone else sets tasks and people often take those as the requirements. The worst case scenario is that these requirements are taken as non-negotiable, and lead to the wrong results if followed to the letter. This often happens with outsourced development, and it can have a negative impact on the end user’s experience.

At SmartHead, we believe that there is more to quality than just meeting client demands. It’s not purely complying with requirements, but also the requirements themselves. We do not take only the client’s expectations into consideration, but the end user’s experience too. This means that our concept of quality also includes components such as usability, UI texts, responsiveness, and release frequency.

High-quality execution means that different needs are satisfied, namely those of the developer, the client, and the end-user.

How can this be achieved? It’s not easy by any means, but you should adopt a mindset of built-in quality from the start.

Quality is about development, not about testing

Quality is often associated with testing. But why?

Quality is determined by the way in which a product is built, not how it is tested.

The whole team is responsible for quality — especially the developer — who must be sure that the feature under development works correctly. It is wrong to say that a task is done when there is still uncertainty surrounding its completion. You can be confident that everything works well in two different ways: by how you write the code, and by how you organize the whole development process, including testing. However, it is crucial that quality assurance comes down to more than just testing. Moreover, the post-development manual testing stage should not be a bottleneck, and ideally should not be needed at all.

So, quality engineering should be included in the development process. To say that everything is “done” means that quality is assured.

Resilience

In products with long life cycles, changes are inevitable. So alongside its quality at the moment of feature development, “system quality” and its level of resilience are also important.

Making regular changes is completely normal, and products are likely to require much more than just one-off correction. This is only possible if it is certain that changes won’t break anything, and if this is the case, we can say that the system is resilient to change.

There are issues with resilience or quality if you do not want to touch anything while it is operating, or if the developers are reluctant to discard code they have already written.

The codebase is not something that we build and hand over. It is our environment that we should make comfortable to work in. It should be easy to add, edit and delete code.

No detailed requirements — no problem

If there are no detailed requirements, this does not mean you need to run to the client to ask “where in the UI should this button be?” Nor does it mean that it can go anywhere, however.

Apart from the incoming requirements (or lack thereof) it’s important to rely on common sense, experience, and taste — and high levels of each of these must be maintained.

The need to write detailed requirements can put pressure on clients to make important decisions regarding the product despite the fact that they lack either the knowledge or information to do so.

The absence of detailed requirements is not a problem, but rather an opportunity to do the job as well as possible and adopt a flexible approach towards the project’s scope, taking into consideration the available time and budget. In other words, focusing on the most important elements and ignoring the less significant ones is the best strategy.

Of course, we should understand what we are doing and why we are doing it, but a detailed description of everything is not necessary.

Bugs are inevitable, but not in excess

“Bugs are inevitable” is often said to justify low-quality development. Just because it is impossible to create bugless software does not mean that an excessive amount of bugs is acceptable.

Bugs are OK provided that: they do not prevent you from reaching project goals, they can be fixed quickly, and are not critical for the system’s stability. Severe bugs that break the main use cases occur quite rarely.

However, if every task is reopened after testing, or if fixing one bug causes three new ones to spring up, it means that there is an issue with quality.

Not everything is a bug

Bugs can also be viewed from another perspective. Their quantity depends not only on how the code is written, but also on the definition of a bug. Formally, a bug can be found in software of any quality. But not everything needs to be reported because it can just lead to some extra work.

We are not suggesting that you ignore bugs, although it is nonetheless important that you understand what requires your attention, and when you can move on. Testers can get bogged down in the details to such an extent that they run out of time to focus on what is actually significant. A lot of time can be lost on checking the layout, which then leaves little opportunity to check the main user scenario. Every task can be returned because of insignificant details, but this can mean that you lose sight of the bigger picture when it comes to quality. When all the tasks are reopened, it is not clear what has been done.

Moreover, you should not test the developer’s competence. If you spend more time testing their tasks purely because you don’t trust their ability, then bugs aren’t your problem. Instead, it is likely that there is mistrust in the team or the task is too complex for the developer. Ultimately, you should test software, not people — trust them instead.

In addition, we see little value in testing the design or the written requirements. They themselves, when they are not implemented, bring no value to end-users. You can and should review, discuss, and design them among your team, yet, there is no need to test them formally.

In a world of endless resources, it would be great to test everything, of course. However, in the here and now, there are simply more important tasks to consider.

So what should we test then?

The QA engineer is a product’s “user zero” who knows it well both from outside and inside. They should prioritize the user’s experience rather than checking the developer’s work. They should check what has been done, not what was being done, in other words features, not tasks.

The QA engineer’s objective is not to find as many bugs as possible, but to ensure that fewer of them make it into the final product. From this point of view, the fewer bugs they find the better.

The QA engineer is also responsible for accessibility and usability, including copywriting, animations, etc.

The QA engineer can also improve development quality by contributing their own thought processes. For example, they can help developers to write tests, cover edge cases, and ensure sufficient and reasonable coverage.

Testing is effective when it aims to bring additional value, and not when it compensates for the drawbacks of development.

Summary

Compliance with the requirements that were set initially shouldn’t be the goal. It’s much more important to make a great product.
User experience should be taken into consideration.
The quality level is set during the development. If time is wasted on compensating poor development with testing, then development, and thus quality, is neglected.
Manual testing should not be a critical need but something that adds to the value.