DEV Community: Swikriti Tripathi

"Wondering which test automation framework to choose? 🤔 Here's my detailed blog comparing Cypress and Playwright for speed and efficiency in local and CI environments! 🚀 #Cypress #Playwright #AutomationTesting"

Swikriti Tripathi — Thu, 09 Jan 2025 11:08:11 +0000

Comparing Test Execution Speed of Modern Test Automation Frameworks: Cypress vs Playwright

Swikriti Tripathi ・ Jan 9

#playwright #cypress #testing #automation

Comparing Test Execution Speed of Modern Test Automation Frameworks: Cypress vs Playwright

Swikriti Tripathi — Thu, 09 Jan 2025 10:55:44 +0000

Comparing Test Execution Speed: Cypress vs Playwright for Modern Test Automation Frameworks

As the testing world increasingly shifts towards automation, tests must run efficiently in a CI/CD environment. Quick feedback on the latest code changes is critical—nobody wants to wait an hour or more to see if their pull request passes. Additionally, CI/CD services can be costly, so minimizing execution time directly translates to saving money. Every millisecond counts in this modern testing landscape.

With this in mind, I experimented to compare the test execution speed of two of today’s most popular modern test automation frameworks: Cypress and Playwright.

If you're not already familiar with these frameworks, here's a brief introduction to each:

Cypress is a front-end testing tool designed for modern web applications. Initially built for end-to-end testing, it has since expanded its capabilities to include component testing, accessibility testing, and more. Cypress supports test scripting in JavaScript and TypeScript and offers compatibility with Chromium-based browsers, Firefox, and WebKit (experimental). It operates directly in the browser, providing a seamless developer experience with features like built-in debugging, automatic waiting, and real-time reloading.

Playwright is an end-to-end testing framework for modern web applications that supports multiple programming languages, including JavaScript, TypeScript, Python, C#, and Java. It fully supports all major browser engines—Chromium, WebKit, and Firefox—making it an excellent choice for cross-browser testing. With powerful features like multiple browser context handling, parallel execution, and robust language integrations, Playwright is ideal for scalable and efficient test automation in diverse environments.

For this project, I wrote end-to-end tests in a TDD style using both frameworks for the Swaglabs V1 demo website. You can find the test scripts in my GitHub repository here.

I created three test files for each framework, covering the following functionalities:

Logging in
Adding items to the cart
Checking out

To ensure a fair comparison, I used the same selectors and functions across both frameworks as much as possible. This approach helps maintain consistency and minimizes discrepancies in the results.

File structure

For this project, I created a root directory named swaglabs-tests, which contains two sub-folders: cypress-tests and playwright-tests.

File structure for cypress-tests

cypress-tests
 ┣ cypress
 ┃ ┣ downloads
 ┃ ┣ e2e
 ┃ ┃ ┣ addToCart.cy.js
 ┃ ┃ ┣ checkout.cy.js
 ┃ ┃ ┗ login.cy.js
 ┃ ┗ support
 ┃ ┃ ┣ commands.js
 ┃ ┃ ┣ e2e.js
 ┃ ┃ ┗ helpers.js
 ┣ README
 ┣ cypress.config.js
 ┣ package-lock.json
 ┗ package.json

File structure for playwright-tests

playwright-tests
 ┣ helpers
 ┃ ┗ helpers.js
 ┣ playwright-report
 ┃ ┗ index.html
 ┣ test-results
 ┃ ┗ .last-run.json
 ┣ tests
 ┃ ┣ addToCart.spec.js
 ┃ ┣ checkout.spec.js
 ┃ ┗ login.spec.js
 ┣ package-lock.json
 ┣ package.json
 ┗ playwright.config.js

Test execution

Each framework contains 3 test suites with a total of 9 tests:

Login Suite: 5 tests
Add to Cart Suite: 2 tests
Checkout Suite: 2 tests

In Local Machine

The tests were executed in both headless and headed modes using the Chrome browser.

Cypress Execution in Headless Mode

(Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:09        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:10        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:06        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:26        9        9        -        -        -

Cypress executed the tests in 26 seconds, with repeated runs showing execution times ranging from 21 seconds to 26 seconds, averaging 24 seconds.

Playwright Execution in Headless Mode

By default, Playwright runs tests in parallel across multiple workers. To ensure a fair comparison with Cypress, I configured Playwright to run tests sequentially with a single worker by updating the playwright.config.js file:

//playwright.config.js

export default defineConfig({}
    .....

    fullyParallel: false, // Disable parallel execution
    workers: 1, // Use a single worker
)

Running 9 tests using 1 worker

  ✓  1 [chromium] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (3.9s)
  ✓  2 [chromium] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (1.9s)
  ✓  3 [chromium] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (2.2s)
  ✓  4 [chromium] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (2.1s)
  ✓  5 [chromium] › login.spec.js:9:6 › Login › normal user logs in (1.5s)
  ✓  6 [chromium] › login.spec.js:18:6 › Login › locked out user logs in (435ms)
  ✓  7 [chromium] › login.spec.js:28:6 › Login › user logs in with no password (443ms)
  ✓  8 [chromium] › login.spec.js:37:6 › Login › user submits empty form (394ms)
  ✓  9 [chromium] › login.spec.js:45:6 › Login › user logs in with incorrect password (423ms)

  9 passed (13.8s)

Playwright completed the execution in around 14 seconds, with repeated runs ranging from 13 seconds to 15 seconds, averaging at 14 seconds. This is almost half the time Cypress took.

Cypress Execution in Headed Mode

  (Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:06        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:09        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:06        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:21        9        9        -        -        -

In headed mode, Cypress executed the tests in 21 seconds. The execution time ranged from 20 to 22 seconds in repeated runs, averaging at 21 seconds.

Playwright Execution in Headed Mode

Running 9 tests using 1 worker

  ✓  1 [chromium] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (3.9s)
  ✓  2 [chromium] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (2.0s)
  ✓  3 [chromium] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (3.1s)
  ✓  4 [chromium] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (2.6s)
  ✓  5 [chromium] › login.spec.js:9:6 › Login › normal user logs in (1.6s)
  ✓  6 [chromium] › login.spec.js:18:6 › Login › locked out user logs in (618ms)
  ✓  7 [chromium] › login.spec.js:28:6 › Login › user logs in with no password (580ms)
  ✓  8 [chromium] › login.spec.js:37:6 › Login › user submits empty form (496ms)
  ✓  9 [chromium] › login.spec.js:45:6 › Login › user logs in with incorrect password (587ms)

  9 passed (16.3s)

In headed mode, Playwright completed the tests in around 16.3 seconds. The execution time ranged from 15 to 16 seconds, with an average of 15.5 seconds. This is still quicker than Cypress, despite a slight overhead in headed mode.

Execution Time Summary Table in Local Machine

Mode	Framework	Average Execution Time	Execution Range
Headless	Cypress	24 seconds	21–26 seconds
Headless	Playwright	14 seconds	13–15 seconds
Headed	Cypress	21 seconds	20–22 seconds
Headed	Playwright	15.5 seconds	15–16 seconds

Key Observations

Playwright is faster overall.

In headless mode, Playwright is ~42% faster than Cypress.
In headed mode, Playwright is ~26% faster than Cypress.

Cypress performance is consistent across headed and headless modes.
Playwright exhibits slight overhead in headed mode compared to headless mode but remains faster than Cypress in both modes.

In CI

To evaluate execution times in CI environments, I created two workflows in GitHub Actions for running tests with Cypress and Playwright. Both frameworks were configured to run tests using the Chrome browser. The workflow files can be found here. Below are the results:

Cypress Execution in CI

 (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:05        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:06        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:04        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:17        9        9        -        -        -

Test Execution Time: The tests took a total of 17 seconds to execute.
Full CI Pipeline Duration: The complete CI pipeline, including setup and test execution, took 1 minute and 40 seconds.
Execution Time Range: The test execution time ranged from 15 to 17 seconds, with an average of 16 seconds.

Playwright Execution in CI

Running 9 tests using 1 worker

  ✓  1 [chromium] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (1.5s)
  ✓  2 [chromium] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (1.6s)
  ✓  3 [chromium] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (1.6s)
  ✓  4 [chromium] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (1.7s)
  ✓  5 [chromium] › login.spec.js:9:6 › Login › normal user logs in (1.3s)
  ✓  6 [chromium] › login.spec.js:18:6 › Login › locked out user logs in (188ms)
  ✓  7 [chromium] › login.spec.js:28:6 › Login › user logs in with no password (164ms)
  ✓  8 [chromium] › login.spec.js:37:6 › Login › user submits empty form (162ms)
  ✓  9 [chromium] › login.spec.js:45:6 › Login › user logs in with incorrect password (177ms)

  9 passed (10.0s)

Test Execution Time: The tests were completed in 10 seconds.
Full CI Pipeline Duration: The complete CI pipeline took only 42 seconds.
Execution Time Range: The test execution time ranged from 9–12 seconds, with an average of 10.5 seconds.

Playwright tests run significantly faster in CI as well making them time efficient.

Execution Time Summary Table in CI

Framework	Test Execution Time	Full CI Duration	TestExecution Range	Average Execution Time
Cypress	17 seconds	1 minute 40 seconds	15–17 seconds	16 seconds
Playwright	10 seconds	42 seconds	9–12 seconds	10.5 seconds

Key Observations

Speed: Playwright significantly outperformed Cypress in both test execution and overall CI pipeline duration.
- Playwright's test execution was nearly 1.6x faster than Cypress's (10.5 seconds vs. 16 seconds on average).
- The overall CI pipeline for Playwright was 2.4x faster (42 seconds vs. 1 minute and 40 seconds).
Efficiency: Playwright tests run more efficiently in CI, saving both time and resources.
Time Ranges: Playwright showed lower execution time variability (9–12 seconds) compared to Cypress (15–17 seconds).

Test Execution in Multiple Browser

I executed the test suite in Chrome and Firefox to evaluate cross-browser compatibility and performance. Here is how each framework performed:

Cypress Cross-Browser Test Execution

Cypress does not natively support running tests across multiple browsers in a single command, requiring separate executions for each browser. This means tests had to be run individually for Chrome and Firefox.

Additionally, Cypress uses Playwright's WebKit engine for WebKit testing, which is still in the experimental stage. When attempting to execute tests in WebKit, I encountered an error that I was unable to resolve, so results for WebKit are unavailable.

Chrome headless


  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:06        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:09        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:05        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:21        9        9        -        -        -

The total test execution time in Chrome ranged between 19-22 seconds, with an average execution time of 21 seconds. This time is better than what I observed earlier while executing tests in Chrome.

Firefox headless

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:06        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:08        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:06        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:21        9        9        -        -        -

The total test execution time in Firefox ranged between 21-22 seconds, with an average execution time of 22 seconds.

On average it took around 43 seconds to run tests on both browsers in headless mode.

Chrome headed


  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:05        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:07        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:05        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:19        9        9        -        -        -

The total test execution time in Chrome ranged between 19-21 seconds, with an average execution time of 22 seconds. This time around the Cypress is faster compared to previous runs.

Firefox headed


  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:08        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:09        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:06        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:23        9        9        -        -        -

The total test execution time in Firefox ranged between 21-22 seconds, with an average execution time of 22 seconds.

On average it took around 44 seconds to run tests on both browsers in headed mode.

Playwright Cross-Browser Test Execution

Playwright natively supports running tests across multiple browsers with a single command. By configuring browser-specific projects in the Playwright configuration file, tests can be executed concurrently on all configured browsers. Here’s how I configured and executed tests in Chrome and Firefox.

//playwright.config.js

export default defineConfig({
    .....
    projects: [
    ...
            {
            name: "firefox",
            use: { ...devices["Desktop Firefox"] },
        },
  ]
})

Headless in Chrome and Firefox

Running 18 tests using 1 worker

  ✓  1 [chromium] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (3.4s)
  ✓  2 [chromium] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (1.8s)
  ✓  3 [chromium] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (2.3s)
  ✓  4 [chromium] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (2.3s)
  ✓  5 [chromium] › login.spec.js:9:6 › Login › normal user logs in (1.5s)
  ✓  6 [chromium] › login.spec.js:18:6 › Login › locked out user logs in (443ms)
  ✓  7 [chromium] › login.spec.js:28:6 › Login › user logs in with no password (435ms)
  ✓  8 [chromium] › login.spec.js:37:6 › Login › user submits empty form (434ms)
  ✓  9 [chromium] › login.spec.js:45:6 › Login › user logs in with incorrect password (446ms)
  ✓  10 [firefox] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (2.6s)
  ✓  11 [firefox] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (2.4s)
  ✓  12 [firefox] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (3.0s)
  ✓  13 [firefox] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (3.2s)
  ✓  14 [firefox] › login.spec.js:9:6 › Login › normal user logs in (2.1s)
  ✓  15 [firefox] › login.spec.js:18:6 › Login › locked out user logs in (745ms)
  ✓  16 [firefox] › login.spec.js:28:6 › Login › user logs in with no password (697ms)
  ✓  17 [firefox] › login.spec.js:37:6 › Login › user submits empty form (677ms)
  ✓  18 [firefox] › login.spec.js:45:6 › Login › user logs in with incorrect password (701ms)

  18 passed (31.1s)

Playwright took 31.1 sec to run all the tests in 2 browsers. Execution time ranged from 30 to 33 seconds with an average execution time of 31 seconds.

Headed mode in Chrome and Firefox

Running 18 tests using 1 worker

  ✓  1 [chromium] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (3.3s)
  ✓  2 [chromium] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (2.0s)
  ✓  3 [chromium] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (2.5s)
  ✓  4 [chromium] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (2.7s)
  ✓  5 [chromium] › login.spec.js:9:6 › Login › normal user logs in (1.7s)
  ✓  6 [chromium] › login.spec.js:18:6 › Login › locked out user logs in (546ms)
  ✓  7 [chromium] › login.spec.js:28:6 › Login › user logs in with no password (540ms)
  ✓  8 [chromium] › login.spec.js:37:6 › Login › user submits empty form (517ms)
  ✓  9 [chromium] › login.spec.js:45:6 › Login › user logs in with incorrect password (554ms)
  ✓  10 [firefox] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (3.3s)
  ✓  11 [firefox] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (2.7s)
  ✓  12 [firefox] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (3.6s)
  ✓  13 [firefox] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (3.8s)
  ✓  14 [firefox] › login.spec.js:9:6 › Login › normal user logs in (2.3s)
  ✓  15 [firefox] › login.spec.js:18:6 › Login › locked out user logs in (1.0s)
  ✓  16 [firefox] › login.spec.js:28:6 › Login › user logs in with no password (869ms)
  ✓  17 [firefox] › login.spec.js:37:6 › Login › user submits empty form (824ms)
  ✓  18 [firefox] › login.spec.js:45:6 › Login › user logs in with incorrect password (833ms)

  18 passed (36.3s)

It took around 36 sec for Playwright to run tests in both browsers in headed mode. Execution time ranged from 33 to 36 seconds with an average execution time of 36 seconds.

Summary Table: Test Execution in Multiple Browsers

Framework	Browser	Mode	Execution Time (Avg)	Execution Time Range
Cypress	Chrome	Headless	21 seconds	19-22 seconds
	Firefox	Headless	22 seconds	21-22 seconds
	Both Browsers	Combined	43 seconds	40-44 seconds
	Chrome	Headed	22 seconds	19-22 seconds
	Firefox	Headed	22 seconds	21-22 seconds
	Both Browsers	Combined	44 seconds	40-44 seconds
Playwright	Chrome, Firefox	Headless	31 seconds	30-33 seconds
	Chrome, Firefox	Headed	36 seconds	33-36 seconds

Key Observations:

Cypress:

Lacks native support for multi-browser execution, requiring tests to be run separately for each browser, leading to longer combined execution times.
WebKit support is still experimental, and errors prevented execution in this browser.
Test execution times for Chrome and Firefox are consistent and comparable.

Playwright:

Supports concurrent multi-browser execution natively, saving time and effort.
Significantly faster overall test execution compared to Cypress when testing in both browsers.
Headed mode is slightly slower than headless but still efficient compared to Cypress.

General:
- Playwright is better suited for scenarios requiring multi-browser compatibility testing due to its concurrency capabilities.
- Cypress is reliable but less efficient for multi-browser testing because of its lack of native concurrency.

Multibrowser Test execution in CI

In this section, I have updated the workflow to execute tests in multiple browsers (Chrome and Firefox) using both Cypress and Playwright. Here’s how each framework performed.

Cypress

Test execution in Chrome


  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:05        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:06        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:04        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:16        9        9        -        -        -

It took 16 seconds for the test execution in Chrome, but the test pipeline took 41 seconds due to additional setup steps.

Test execution in Firefox


  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  addToCart.cy.js                          00:05        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  checkout.cy.js                           00:06        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  login.cy.js                              00:05        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:17        9        9        -        -        -

It took 17 seconds for test execution in Firefox, but the test pipeline also took 41 seconds.

Total CI Execution Time: 1 minute 45 seconds

Playwright

Test execution in Chrome and Firefox


Running 18 tests using 1 worker

  ✓  1 [chromium] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (1.6s)
  ✓  2 [chromium] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (1.6s)
  ✓  3 [chromium] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (1.7s)
  ✓  4 [chromium] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (1.8s)
  ✓  5 [chromium] › login.spec.js:9:6 › Login › normal user logs in (1.4s)
  ✓  6 [chromium] › login.spec.js:18:6 › Login › locked out user logs in (205ms)
  ✓  7 [chromium] › login.spec.js:28:6 › Login › user logs in with no password (178ms)
  ✓  8 [chromium] › login.spec.js:37:6 › Login › user submits empty form (163ms)
  ✓  9 [chromium] › login.spec.js:45:6 › Login › user logs in with incorrect password (176ms)
  ✓  10 [firefox] › addToCart.spec.js:9:6 › Add to cart › user adds a item to cart (2.5s)
  ✓  11 [firefox] › addToCart.spec.js:13:6 › Add to cart › user adds multiple items to cart (1.8s)
  ✓  12 [firefox] › checkout.spec.js:9:6 › checkout › user checks-out a item from the cart (2.3s)
  ✓  13 [firefox] › checkout.spec.js:14:6 › checkout › user checks-out multiple items from the cart (2.4s)
  ✓  14 [firefox] › login.spec.js:9:6 › Login › normal user logs in (1.6s)
  ✓  15 [firefox] › login.spec.js:18:6 › Login › locked out user logs in (467ms)
  ✓  16 [firefox] › login.spec.js:28:6 › Login › user logs in with no password (499ms)
  ✓  17 [firefox] › login.spec.js:37:6 › Login › user submits empty form (387ms)
  ✓  18 [firefox] › login.spec.js:45:6 › Login › user logs in with incorrect password (428ms)

  18 passed (23.9s)

Test execution in both Chrome and Firefox took 24 seconds, with the test pipeline taking 25 seconds.

Total CI Execution Time: 45 seconds

Summary Table: Test Execution in Multiple Browsers

Framework	Browser(s)	Test Execution Time	Test Pipeline Time	Total CI Execution Time
Cypress	Chrome	16 seconds	41 seconds	1m 45s
	Firefox	17 seconds	41 seconds
Playwright	Chrome, Firefox	24 seconds	25 seconds	45 seconds

Key Observations

Cypress:
- Test execution times in Chrome and Firefox were very similar (16s and 17s, respectively).
- The test pipeline took longer than the test execution itself due to additional setup steps, taking around 41 seconds for both browsers.
Playwright:
- Test execution times in Chrome and Firefox were less than that of Cypress(43s for combined).
- The Playwright test pipeline was faster than Cypress, taking only 25 seconds in total.
CI Efficiency:
- Playwright tests are more efficient in terms of total CI execution time compared to Cypress, with Playwright completing in 45 seconds versus Cypress taking 1m 45s.

Overall summary table

Framework	Environment	Browser(s)	Mode	Test Execution Time (Avg)	Test Execution Range	Pipeline Time	Total CI Time
Cypress	Local	Chrome	Headless	24 seconds	21–26 seconds	N/A	N/A
	Local	Chrome	Headed	21 seconds	20–22 seconds	N/A	N/A
	CI	Chrome	Headless	16 seconds	15–17 seconds	41 seconds	1m 45s
	CI	Firefox	Headless	17 seconds	15–17 seconds	41 seconds
	Local	Firefox	Headless	22 seconds	21–22 seconds	N/A	N/A
	Local	Firefox	Headed	22 seconds	21–22 seconds	N/A	N/A
	Local	Both Browsers	Combined	44 seconds	40–44 seconds	N/A	N/A
Playwright	Local	Chrome	Headless	14 seconds	13–15 seconds	N/A	N/A
	Local	Chrome	Headed	15.5 seconds	15–16 seconds	N/A	N/A
	CI	Chrome	Headless	10.5 seconds	9–12 seconds	25 seconds	42 seconds
	Local	Chrome, Firefox	Headless	31 seconds	30–33 seconds	N/A	N/A
	Local	Chrome, Firefox	Headed	36 seconds	33–36 seconds	N/A	N/A
	CI	Chrome, Firefox	Headless	22 seconds	20-24 seconds	25 seconds	45 seconds

Overall Observations

Speed:

Playwright consistently outperformed Cypress in test execution time across environments.
In local environments, Playwright completed tests in nearly half the time compared to Cypress in headless mode.
In CI, Playwright's combined browser execution (24s) was significantly faster than Cypress's separate executions (16s + 17s).

Pipeline Efficiency:

Cypress pipelines consumed more time for setup and execution, taking 41 seconds for both Chrome and Firefox individually in CI.
Playwright pipelines were optimized for multi-browser execution, completing setup and execution in just 25 seconds.

Concurrency:

Playwright supports native multi-browser execution, running tests concurrently across browsers with a single command.
Cypress lacks this capability, requiring separate commands for each browser, resulting in increased overall execution time.

Consistency:
- Cypress showed consistent test execution times across Chrome and Firefox in both local and CI environments.
- Playwright maintained fast and reliable execution with minimal time variability in both environments.

Conclusion

Both Cypress and Playwright are reliable testing frameworks for modern automation needs. However, Playwright stands out in terms of speed, efficiency, and multi-browser support. Its ability to run tests concurrently across multiple browsers and its optimized CI pipeline make it a better choice for teams looking to save time and resources. Cypress, while user-friendly and consistent, may not be the best fit for large-scale projects requiring extensive cross-browser testing or time-sensitive CI/CD workflows.

Application Programming Interface (API's) : One Byte Explainer

Swikriti Tripathi — Wed, 19 Jun 2024 09:15:07 +0000

This is a submission for DEV Computer Science Challenge v24.06.12: One Byte Explainer.

Explainer

Imagine APIs as the postman for your digital world. They deliver messages (data requests) between your computer (web client) and websites (web servers). They make sure everything gets where it needs to go, like your own personal courier service for the internet!

Additional Context

How to containerize your web app- a beginner-friendly tutorial for Dockerfile

Swikriti Tripathi — Tue, 18 Jun 2024 18:07:18 +0000

Welcome to part 2 of the series Docker for Dummies in this blog we are going to create an image of a small web app and learn about what each step does. Without any further ado let's get started. For this blog, I'm going to use a simple vue-app quiz app that'll let you guess the name of the book based on the first line. If you want to follow along with the setup you can find the GitHub link to the web app here or you can use your app or you can create a simple hello-world app in node or framework/language of your choosing.

Let's create the image

Note: this tutorial follows ubuntu commands but they should be similar for other OS's as well

Let's clone the app and go inside of it

git clone https://github.com/SwikritiT/Guessthebook-blog
cd Guessthebook-blog

We are going to start by creating a file called Dockerfile in the root of our repository.

touch Dockerfile

Dockerfile

A Dockerfile is a script that contains all the steps necessary to build an image. Dockerfile starts with something called base image. A base image is a pre-configured environment that our images build upon. The base image can be an OS like Linux, alpine, or some application stack. Choosing an appropriate base image is crucial for the overall size and productivity of the image of our application. We will talk more about how to select the appropriate base image in the later part of this series so for now let's keep in mind that a lighter base-image will create a lighter app image(this comes with its limitations which we will talk about in more detail in the later part of this series). So, for this app, we are going to use node:alpine images as a base, you can select your needed version of a base image from an image registry like Docker Hub.

Note: If you're not building the image of the node application, you need to select the image appropriate for your stack

# Start with a base image
FROM node:alpine

FROM: Specifies the base image to use. Every Dockerfile starts with this instruction.

The next step is to set the WORKDIR i.e. working directory for our container where commands will be executed and files will be stored by default. Normally, for web apps, workdir is set to /usr/src/app but you can customize it as per your need.

# Set the working directory
WORKDIR /usr/src/app

Now let's copy the necessary config files to our container, in our case, package.json and lockfile. For this, we will use COPY command

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

COPY or ADD: Copies files from your local filesystem into the container.

The next step is to install dependencies just like we do in our host machine and for this RUN command can be used

# Install the application dependencies
RUN npm install

RUN: Executes commands in the container. These commands are typically used to install software packages.

Now we copy the rest of the files to our container's working directory

# Copy the current directory contents into the container at /usr/src/app
COPY . .

The above command will copy every file and folder that is in your app to the container's working directory. This includes the build file, files created by IDE, or other miscellaneous files/folders that might not be necessary for building the image. So, it is a good practice to either copy only necessary files or create a .dockerignore file with the list of files and folders that you don't want to be copied inside the container. The syntax of the .dockerignore file is similar to that of .gitignore. Learn more here

The next step is to build our application

# Build the application
RUN npm run build

The next step is to Expose the ports on which a containerized application listens for network connections

# Make port 3000 available to the world outside the container
EXPOSE 3000

EXPOSE is a way to document which ports the application running inside the container will use. It does not map the port to the host machine’s ports. It simply indicates which ports are intended to be accessible.

We've come to the final stage where we will run the application. There are two commands that we can use to do this CMD and ENTRYPOINT for this part we will be using CMD and we'll talk about ENTRYPOINT in later parts.

Since we will be running the application in production env we will use the preview command for this. Later we'll also create a dockerized dev env.

# Define the command to run the app
CMD ["npm","run","preview"]

CMD: Provides the command that will be run when a container is started from the image. Only one CMD instruction can be present in a Dockerfile.

Now let's look at the whole Dockerfile

# Start with a base image
FROM node:alpine

# Set the working directory
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install the application dependencies
RUN npm install

# Copy the current directory contents into the container at /usr/src/app
COPY . .

# Build the application
RUN npm run build

# Make port 3000 available to the world outside the container
EXPOSE 3000

# Define the command to run the app
CMD ["npm","run","preview"]

In Docker, each line in your Dockerfile creates a new layer in the final image, like adding ingredients to a sandwich. These layers stack on top of each other, with each layer representing a change or addition, such as copying files or installing software. Docker saves these layers, and if you rebuild your image and some layers haven’t changed, Docker reuses them, speeding up the build process and reducing redundancy. So, in the above dockerfile, we have 8 layers.

Build the image

Now that we've created the image it's time to build it and get it running. We can run the following command in the terminal from the root of our repository to build a docker image

$ docker build . -t guessthebook:v1

[+] Building 1.1s (11/11) FINISHED                               docker:default
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 191B                                       0.0s
 => [internal] load metadata for docker.io/library/node:20-alpine          0.9s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 129B                                          0.0s
 => [1/6] FROM docker.io/library/node:20-alpine@sha256:66c7d989b6dabba6b4  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 1.41kB                                        0.0s
 => CACHED [2/6] WORKDIR /usr/src/app                                      0.0s
 => CACHED [3/6] COPY package*.json ./                                     0.0s
 => CACHED [4/6] RUN npm install                                           0.0s
 => CACHED [5/6] COPY . .                                                  0.0s
 => CACHED [6/6] RUN npm run build                                         0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:08f32d7f583b7e65a844accafff8fc19930849204c5ae  0.0s
 => => naming to docker.io/library/guessthebook:v1                         0.0s

You will get the output like such with the information of each layer being built. The command docker build . -t guessthebook:v1 reads the Dockerfile in the current directory, builds a Docker image according to its instructions, and tags this image as guessthebook with version v1.

Now if you run the following command in your terminal, you should see the relevant info about the image that we just created

docker images

REPOSITORY                                   TAG            IMAGE ID       CREATED         SIZE
guessthebook                                 v1             08f32d7f583b   5 minutes ago   275MB

Running a Docker Container

Once you have built the image, you can run it with the docker run command:

docker run -p 3000:3000 guessthebook:v1

This command runs a container from the guessthebook:v1 image, mapping port 3000 of the container to port 3000 on the host machine. We use -p for port mapping.

We can improvise the above command to use the option -d to run it in detached mode

 docker run  -d -p 3000:3000 guessthebook:v1

After running the docker container you can visit http://localhost:3000 to ensure everything works fine. If everything is setup and working correctly you should be greeted with this screen

You can now take a rest and play the quiz to see how many you get right.

Stop container

You can stop the container with the following command

docker stop <container_name or id> # can run `docker ps` to get the name and id

Publish the image to the docker hub

If we want to take this a step further we can publish the image to the docker hub. For that create an account on Docker Hub if you haven't already. Next, you can create a new repository.

docker login

Tag your local image to match the repo image

# docker tag local-image:tagname new-repo:tagname
docker tag guessthebook:v1 <your-docker-username>/guessthebook

Push the image

# docker push new-repo:tagname
docker push <your-docker-username>/guessthebook

Now, you can go and check if your docker hub has the image that you just pushed.

To test the image you can now run the image by pulling directly from Docker Hub

# let's remove the locally tagged image first
docker rmi <your-docker-username>/guessthebook

# run the container
docker run -p 3000:3000 <your-docker-username>/guessthebook

That's it for this blog. Hope you enjoyed this one and learned something new as well! See you in the next part of this series. If you have any queries or suggestions please comment them below!

Docker for Dummies- Introduction to docker

Swikriti Tripathi — Fri, 14 Jun 2024 08:34:05 +0000

Welcome to Docker for Dummies!

This is going to be an idk-how-many-part series where I'll try to explain docker in as simple a way as possible. I'm also new to the world of containerization so the purpose of this series is to try to teach this technology while learning it myself.

If you are new to the tech industry or have already been a part of this world, you might've probably come across the term Docker now and again, but what exactly is this docker? Are we building a ship or something?

The wonderful world of Docker

Imagine you made a delicious meal and want to share it with your partner but then you realize that you are in a LDR and it won't be possible due to the distance. But what if there was a way to pack your dish, along with all the ingredients and tools you used, into a box that keeps it fresh and ready to eat anywhere, anytime? Well, there isn't one for food but there's one for software that does something like that and that's? You guessed it right Docker

Docker is like a magical box that packages your application, along with everything it needs to run, into a neat, portable container. This container can run on any computer, anywhere, without any worries about compatibility or missing ingredients. Cool, right?

In a more technical term - Docker is a platform that uses containerization to allow developers to package applications along with all their dependencies into a container. This ensures that the application can run consistently across different computing environments.

Why Should You Care About Docker?

There are a lot of reasons to use docker. I'll explain some below:

Consistency: Suppose your team is building an application and you build a feature in your local system and everything is working as expected. But then your team member tries to run the same feature in their local set-up and it crashes and you suddenly find yourself amidst a decade-old problem in software engineering

Docker ensures that your application works the same way
everywhere. No more "it works on my machine" problems but rather
"it works on my container" problem 🫠
Isolated Environments: If you are building 3 different applications that require 3 different versions of node, trying to set it up in your local machine, switching between them, and trying to make them work would be a hassle that I would only wish upon my enemies not that I have any enemies👀. But with docker, you can just switch between the containers and develop everything parallelly without much difficulty.
Portability: Whether you’re developing on your laptop or deploying to a cloud server, Docker containers can run anywhere.
Efficiency: Containers are lightweight and start up in seconds, making them perfect for development and deployment.

These are just a few of the benefits. Docker is a gift that keeps on giving.

Virtual Machines vs Containers

It might be easy to confuse virtual machines with container but they are completely different technologies and solve different problems. Here's a picture to differentiate them

Containers use the host OS's kernel and share it among all containers, making them more lightweight and efficient. VMs emulate an entire physical machine, including its operating system (OS), making it possible to run multiple OS instances on a single physical machine.

Getting Started with Docker

Ready to dip your toes into the Docker waters? Let’s start with a quick example that will get you up and running in no time.

Step 1: Install Docker

First things first, you’ll need to install Docker. Head over to the Docker website and download Docker Desktop for your operating system (Windows, macOS, or Linux). Follow the installation instructions provided.

Step 2: Run Your First Container

Now, let's run your first container. We are going to run hello-world container. Open your terminal (Command Prompt on Windows, Terminal on macOS and Linux) and you'll run the following command:

docker run hello-world

If everything is working fine you should see something like this on your screen

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete 
Digest: sha256:d1b0b5888fbb59111dbf2b3ed698489c41046cb9d6d61743e37ef8d9f3dda06f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Let's break down what that command did, the command pulled an image hello-world from the docker hub and ran it in your host machine. If the image is already present in your system the command will skip the pulling part and just run the existing image.

Now, if you run the command again it will run the previously pulled image.

docker run hello-world

Image and containers

Think of a container as a ready-to-eat meal that you can simply heat up and consume. An image, on the other hand, is the recipe or ingredients for that meal.

So just like how you need a recipe and ingredients to make a meal, you need an image and a container runtime (Docker engine) to create a container. The image provides all the necessary instructions and dependencies for the container to run, just like a recipe provides the steps and ingredients to make a meal.

In short, an image is like a blueprint or template, while a container is an instance of that blueprint or template.

Some useful commands

You can try out these commands and see what the output

#  lists all the running Docker containers on your system
docker ps


# lists all Docker containers on your system, including those that are currently running, as well as those that have stopped or exited.
docker ps -a 

# lists all the Docker images stored on your system, showing details such as the repository name, tag, image ID, creation date, and size

docker images

You can check the official documentation for other commands and try them out.

This is it for this part. In the next part, we will be writing a dockerfile for a small web app and it should be exciting.

Until then take care! Keep learning!

Debugging and Error Tracing in Playwright

Swikriti Tripathi — Wed, 07 Dec 2022 08:22:04 +0000

How many times has it happened that you wrote blocks of code thinking they work but the reality was otherwise? I think it's safe to assume that this has happened quite often. Sometimes it might be easy to find the mistake but it's not always the case, so in this blog, I'm going to try to explain some of the debugging methods in Playwright that we can incorporate into our project to make our life a little bit easier. This might be the right time to mention that you might want to read part one of the series as we are going to use the same code here.

Headed mode

The first method would be to run the tests in headed mode. Playwright by default runs in headless mode. Use headless:false while launching the browser. Additionally, you can also use the slowMo option to slow down the test execution process.



await chromium.launch({ headless: false, slowMo: 100 }); // or firefox, WebKit

Playwright Inspector

Playwright comes with a default GUI tool that we can use to inspect our scripts. Through this tool, you'll be able to step-over each script and evaluate them in real-time. There are a few ways through which we can open Playwright inspector.

1. PWDEBUG

Playwright provides us with an environment variable that'll configure it in debugging mode and opens the inspector. Set the PWDEBUG variable to 1 or console.

In my case, I'm running the e2e test in debug mode by setting PWDEBUG=1 .



PWDEBUG=1 npm run test:e2e tests/acceptance/features/todo.feature

This will open up an inspector like so

As you can see this gives me the test scripts, now I can either step over each script or run everything at once

step over each script

run all at once

With each step-over, the inspector will step through each line of the test highlighting the selector as you go. You can also see the logs that display each action that the tests perform.

Additionally, you'll also be able to access the browser developers' tools.

2. page.pause()

The next method to launch the inspector is to use page.pause() in the script.



When('the user adds {string} to the todo list using the webUI',async function (item) {
   // fill the item that was input from the feature file, to the inputText field in the UI
   await page.fill(todoInput , item)
   // pauses the test execution and launches Playwright inspector
   await page.pause()
   // click the button
   await page.click(todoButton)
})

This will pause the execution of the test and launch Playwright inspector right before clicking the button.

Now, we can perform similar operations as explained above.

Trace Viewer

Trace viewer is another GUI tool to explore the recorded Playwright traces of the tests after the tests have been executed. This is especially essential while running the tests in the Continuous Integration (CI) environment.

Let's see how we can set it up in our end-to-end tests. In the cucumber.conf.js file inside before hook we can add the following configuration



Before(async function () {
   global.context = await global.browser.newContext();
   // start tracing the test execution by enabling the screenshots and snapshots
   await global.context.tracing.start({ screenshots: true, snapshots: true });
   global.page = await global.context.newPage();
});

This will start the tracing of the tests, to know about more options that can be configured while starting the trace you can go through this documentation

Now, in the after hook we can add the following code to stop tracing and store it in a certain specified path.



After(async function () {
   await global.page.close();
   // stop tracing and store it in the given path
   await global.context.tracing.stop({ path: 'tests/acceptance/report/trace.zip' });
   await global.context.close();
});

Now, if we run the tests we should get a trace.



npm run test:e2e tests/acceptance/features/todo.feature

This should create the folder report inside of tests/acceptance and you should see a file called trace.zip.

If you extract the file then inside trace/resources you should be able to see the screenshots of the UI through various steps in test execution.

But the fun part is to view the trace which we can do by running the following command from the root of the project.



npx playwright show-trace tests/acceptance/report/trace.zip

This should open up the Playwright trace viewer GUI for you like this

As you can see in the picture we can access a lot of functionalities through the trace viewer like Actions, Metadata, Console, Network, and so on. These will come in handy if we need to figure out or debug the test failure. If you want to know in detail about each of these functionalities you can go through this documentation

These were the few ways in which we can debug and trace tests with tools provided by Playwright. I hope you found this helpful. In the next part of the series, we'll run the tests on CI and get traces for test failures.

Behavior Driven Development (BDD) using Playwright

Swikriti Tripathi — Mon, 07 Feb 2022 11:44:15 +0000

Playwright is an open-source NodeJS framework for browser automation. It is developed by Microsoft and the development team has members that were involved in developing Puppeteer for Google.

One of the main features of Playwright is that it can automate Chromium, Webkit, and Firefox browsers with a single API. Along with being cross-browser, it is cross-platform and cross-language, supporting the major OS like Windows, Linux, Mac and languages like TypeScript, JavaScript, Python, .NET, Java. Playwright also comes with tools like codgen - which lets you generate automatic code by recording your actions, you can find out more about Playwright on their official website.

For this blog, we will be implementing BDD in Playwright. I have a small to-do web app and I'm going to be setting up Playwright in the same. If you want to follow through you can fork and clone the project from here. If you have your web application you can set up Playwright there as well. Let's get started!

Note: the whole setup is done in Ubuntu 20.04.3 LTS, so some setup steps might differ depending on your platform

Prerequisites

Node.js version 12 or above. If you don't already have node installed in your system you can use this blog as a guide

Note Only Ubuntu 18.04 and Ubuntu 20.04 are officially supported.

Installation

Run from your project's root directory



   npm i -D @playwright/test
   npm i -D playwright 
   npx playwright install

Playwright doesn't come with the built-in support for BDD so we are going to use the help of another tool Cucumber



   npm i -D @cucumber/cucumber@7.3.1 @cucumber/pretty-formatter

After this, devDependencies in your package.json should look something like this



// package.json file

"devDependencies": {
    "@cucumber/cucumber": "^7.3.1",
    "@cucumber/pretty-formatter": "^1.0.0-alpha.1",
    "@playwright/test": "^1.18.0",
    "playwright": "^1.18.1"
  }

Configuration

We are going to use Cucumber to run our tests so we need to have a configuration file for it. At the root level of your project create a file cucumber.conf.js

First of all, we are going to require the following:



// cucumber.conf.js file

const { Before, BeforeAll, AfterAll, After, setDefaultTimeout } = require("@cucumber/cucumber");
// you can choose other browsers like webkit or firefox according to your requirement
const { chromium } = require("playwright");

Set default timeout to some reasonable amount of time



// cucumber.conf.js file

// in milliseconds
setDefaultTimeout(60000)

Add the following code snippet to your file



// cucumber.conf.js file

// launch the browser
BeforeAll(async function () {
   global.browser = await chromium.launch({
       headless: false,
       slowMo: 1000,
   });

});

// close the browser
AfterAll(async function () {

   await global.browser.close();
});

In the above snippet of code, we launch a chrome browser where our tests will be automated. You can launch a different one as per your requirement, just make sure you import the correct browser. We run the browser in the headed mode which can be done by setting headless:false, this means that when the test is running we can see it being automated in the browser. You can set it to true if you don't want to see the test running but where is the fun in that? Another option is slowMo which slows down Playwright operations by the specified amount of milliseconds and can be helpful to watch the test run. There are various options that can be set while launching the browser, you can go through all of them here. After we've finished our operations we will close the browser. This configuration is for before/after all of the tests are run. Now we need to configure what happens when each test scenario is run. For this look at the snippet below:



// cucumber.conf.js file

// Create a new browser context and page per scenario
Before(async function () {
   global.context = await global.browser.newContext();
   global.page = await global.context.newPage();
});

// Cleanup after each scenario
After(async function () {
   await global.page.close();
   await global.context.close();
});

After we've launched our browser we need to create a new browser context. Playwright allows creating incognito browser contexts with browser.newContext([options]) method. Each browser context has its page that provides methods to interact with a single tab in a browser. We can create a page with context.newPage() method. Similar to launching a browser we can set a lot of options while creating a browser context as well like screenshots, record video, geolocation, and more, you can go through all of them here. After we've finished with our operations we close the page and context.

Voila, we're done with the configuration part. The whole cucumber.conf.js file looks like this :



// cucumber.conf.js file

const { Before, BeforeAll, AfterAll, After, setDefaultTimeout } = require("@cucumber/cucumber");
const { chromium } = require("playwright");

setDefaultTimeout(60000)

// launch the browser
BeforeAll(async function () {
   global.browser = await chromium.launch({
       headless: false,
       slowMo: 1000,
   });

});

// close the browser
AfterAll(async function () {

   await global.browser.close();
});

// Create a new browser context and page per scenario
Before(async function () {
   global.context = await global.browser.newContext();
   global.page = await global.context.newPage();
});

// Cleanup after each scenario
After(async function () {
   await global.page.close();
   await global.context.close();
});

Writing Tests

Now some fun stuff, we start writing tests!

Our file structure will look like this



📦tests
┗ 📂acceptance
┃ ┣ 📂features
┃ ┃ ┗ 📜todo.feature
┃ ┗ 📂stepDefinitions
┃ ┃ ┗ 📜todoContext.js

Following the above tree create file tests/acceptance/features/todo.feature. As we are using BDD, we are going to start by writing a feature file and we are going to be using Gherkin language to do so. If you don't know how to write a feature file or what Gherkin is you can take the help of the following blogs as it's out of the scope of this blog and won't be explained in detail.

Here's a basic syntax of what a feature file looks like



Feature: a short description of a software feature
As a user
I want to do this
So I can achieve that

Scenario: name of the scenario
Given [Preconditions or initial context of the system ]
When [Event or action]
Then [Expected output]

Now assuming you've got some knowledge of feature files and how to write them we proceed further.

The application that I'm going to be testing is a todo app and the UI looks like this.

I want to test if the item I've added is displayed on the UI or not. And the feature file looks like this.



// todo.feature

Feature: todo
 As a user
 I want to add an item to the todo list
 So that I can organize tasks

 Scenario: Add item to the todo list
   Given a user has navigated to the homepage
   # the text inside the quote works as a variable that can be passed to a function
   When the user adds "test" to the todo list using the webUI
   Then card "test" should be displayed on the webUI

Now we implement each step of the scenario using Playwright! Create a context file tests/acceptance/stepDefinitions/todoContext.js. We can get a boilerplate for each step in the scenario where we can provide our implementation. For that add the following script in your package.json file.



"test:e2e": "cucumber-js --require cucumber.conf.js --require tests/acceptance/stepDefinitions/**/*.js --format @cucumber/pretty-formatter"

We will be using the test:e2e script for running the test. Now go to your terminal and run the script



npm run test:e2e tests/acceptance/features/todo.feature

This will run your feature file. As the steps aren't implemented yet you will get something like this on your screen.



? Given a user has navigated to the homepage
      Undefined. Implement with the following snippet:

        Given('a user has navigated to the homepage', function () {
          // Write code here that turns the phrase above into concrete actions
          return 'pending';
        });

  ? When the user adds "test" to the todo list using the webUI
      Undefined. Implement with the following snippet:

        When('the user adds {string} to the todo list using the webUI', function (string) {
          // Write code here that turns the phrase above into concrete actions
          return 'pending';
        });

  ? Then card "test" should be displayed on the webUI
      Undefined. Implement with the following snippet:

        Then('card {string} should be displayed on the webUI', function (string) {
          // Write code here that turns the phrase above into concrete actions
          return 'pending';
        });

You can now add the generated snippets into your context file and start implementing them.

Import following



// todoContext.js file

const {Given, When, Then} = require('@cucumber/cucumber')
// import expect for assertion
const { expect } = require("@playwright/test");

Define your launch url and selectors for different UI elements as per need, these are project specific. Playwright supports CSS and Xpath selectors. You can find the detailed information about them here



// todoContext.js file

//launch url
const url = 'http://localhost:3000'

//define selectors
const homepageElement = '.borderTodo'
const todoInput = '.todo-input';
const todoButton = '.todo-button';
const todoItem = '.todo .todo-item ';

Now we can implement the individual test steps, like so



// todoContext.js file

Given('a user has navigated to the homepage', async function () {
   // navigate to the app
   await page.goto(url)
   // locate the element in the webUI
   const locator = await page.locator(homepageElement)
   // assert that it's visible
   await expect(locator).toBeVisible()
})

When('the user adds {string} to the todo list using the webUI',async function (item) {
   // fill the item that was input from the feature file , to the inputText field in the UI
   await page.fill(todoInput , item)
   // click the button
   await page.click(todoButton)
})

Then('card {string} should be displayed on the webUI',async function (item) {
   // get text of the item that is visible in the UI
   const text = await page.innerText(todoItem)
   // assert that its name is similar to what we provided
   await expect(text).toBe(item)
})

You can find different methods available to interact with UI elements like click, fill and so on in Playwright's official documentation, it's very nicely explained how the function works along with the example code.

We use the page that we created in the before hook to interact with various web elements. Playwright performs autowait and performs a range of actionability checks on elements and ensures that elements are ready to perform the expected operation. This is one of its plus points.

This is the whole context file



// todoContext.js file

const {Given, When, Then} = require('@cucumber/cucumber')
// import expect for assertion
const { expect } = require("@playwright/test")

//launch url
const url = 'http://localhost:3000'

//define selectors
const homepageElement = '.borderTodo'
const todoInput = '.todo-input'
const todoButton = '.todo-button'
const todoItem = '.todo .todo-item '


Given('a user has navigated to the homepage', async function () {
   // navigate to the app
   await page.goto(url)
   // locate the element in the webUI
   const locator = page.locator(homepageElement)
   // assert that it's visible
   expect(locator).toBeVisible()
})

When('the user adds {string} to the todo list using the webUI',async function (item) {
   // fill the item that was input from the feature file , to the inputText field in the UI
   await page.fill(todoInput , item)
   // click the button
   await page.click(todoButton)
})

Then('card {string} should be displayed on the webUI',async function (item) {
   // get text of the item that is visible in the UI
   const text = await page.innerText(todoItem)
   // assert that its name is similar to what we provided
   expect(text).toBe(item)
})

Run the test

First of all, you need to run your application, in my case



npm run start

Now run the test and watch it in the browser



npm run test:e2e tests/acceptance/features/todo.feature

You should get a log similar to this one.



Feature: todo # tests/acceptance/features/todo.feature:1

 As a user
 I want to add an item to the todo list
 So that I can organize tasks

 Scenario: Add item to the todo list # tests/acceptance/features/todo.feature:6
   Given a user has navigated to the homepage
   When the user adds "test" to the todo list using the webUI
   Then card "test" should be displayed on the webUI

1 scenario (1 passed)
3 steps (3 passed)
0m04.266s (executing steps: 0m04.010s)

Hopefully, your test also passed like mine and you got to learn about a new library.
You can extend the feature file to add more scenarios or add multiple feature files, implement the Page Object Model as per your requirement and it should all work the same.

You can find the source code of this implementation here

How to automate tests for your website using Nightwatch.js? - Part 2- Writing tests

Swikriti Tripathi — Fri, 03 Dec 2021 12:55:51 +0000

Before starting this blog please make sure you've followed through the previous blog. This blog is the continuation of the previous one.

We will be using BDD for writing tests meaning we will be writing a feature file first. We will be using Gherkin to write feature files.

Through Gherkin, we can write test cases in the native language which will be easier to understand by everyone involved in the project whether they're from the technical field or not. I'm going to explain some of the keywords and their usage, that are used in this blog for further information please visit their official website

Keywords

Feature: a short description about a software feature
Scenario: list of steps to describe a business rule

Syntax

Scenario: name of the scenario
Given [Preconditions or initial context of the system ]
When [Event or action]
Then [Expected output]

Write a feature file

Inside tests/acceptance/ folder make a folder name feature. Inside this folder make a feature file named todo.feature, we will be writing a scenario in this file.

The UI of the application that I'm writing test on looks like this

We can simply add todo items and remove them. For this blog, I'm going to write a test scenario for adding the item and checking if the item is displayed in the UI.

The feature file todo.feature looks like this

Feature: todo
  As a user
  I want to add an item to the todo list
  So that I can organize task

  Scenario: Add item to the todo list
    Given a user has navigated to the homepage
    When the user adds "test" to the todo list using the webUI
    Then card "test" should be displayed on the webUI

You can add multiple scenarios in the same feature file as per your need.

Step Definitions

After writing scenarios we need to implement them, for this, we will create stepDefinitions. Create a folder stepDefinitions inside tests/acceptance/ and inside setDefinitions create a file todo.js.

Add the following script in your package.json in the scripts section

"test:e2e": "cucumber-js --require tests/acceptance/cucumber.conf.js --require tests/acceptance/stepDefinitions"

We will use the test:e2e script for running the test.

Run selenium server

Using docker:
docker run -d --network="host" -v /dev/shm:/dev/shm selenium/standalone-chrome-debug

using jar file: (inside the folder where your selenium server and chromedriver files are)
java -jar <name-of-seleniun-server-standalone>.jar -port 4444

Now go to your terminal and from inside the root directory of your project run the feature file using the following script

npm run test:e2e <path-to-yourfeature-file>

In my case

npm run test:e2e tests/acceptance/feature/todo.feature

The above script will run the whole feature, if you want to run a particular scenario you can and the line number of the scenario at last like so

npm run test:e2e tests/acceptance/feature/todo.feature:6

After running the feature file you will see an output similar to this in your terminal

Failures:

1) Scenario: Add item to todo list # tests/acceptance/feature/todo.feature:6
   ? Given a user has navigated to the homepage
       Undefined. Implement with the following snippet:

         Given('a user has navigated to the homepage', function () {
           // Write code here that turns the phrase above into concrete actions
           return 'pending';
         });

   ? When the user adds "clean room" to the todo list using the webUI
       Undefined. Implement with the following snippet:

         When('the user adds {string} to the todo list using the webUI', function (string) {
           // Write code here that turns the phrase above into concrete actions
           return 'pending';
         });

   ? Then card "clean room" should be displayed on the webUI
       Undefined. Implement with the following snippet:

         Then('card {string} should be displayed on the webUI', function (string) {
           // Write code here that turns the phrase above into concrete actions
           return 'pending';
         });


1 scenario (1 undefined)
3 steps (3 undefined)
0m00.001s (executing steps: 0m00.000s)

You can copy these code templates and paste in your todo.js file. This is where we will write the implementation.

 Given('a user has navigated to the homepage', function () {
           // Write code here that turns the phrase above into concrete actions
           return 'pending';
         });

First of all, import following in the file todo.js

const {Given, When, Then} = require('@cucumber/cucumber')
const {client} = require('nightwatch-api')

Now find the CSS or XPath selector of various elements that will be involved in the test.

//css selectors
const todoInputField = '.todo-input'
const todoCreateButton = '.todo-button'
const todoListItem = '.todo .todo-item'

You can do this by using the developer’s tool in the browser and inspecting each element you need.

You can also check if the element is correct by using console.

Now we implement the steps.

Given('a user has navigated to the homepage', function () {
    return client.url("http://localhost:3000");
});


When('the user adds {string} to the todo list using the webUI', async function (item) {
    await client.waitForElementVisible(todoInputField)
        .click(todoInputField)
        .setValue(todoInputField, item)
        .click(todoCreateButton)
    return client
});

Then('card {string} should be displayed on the webUI', function (item) {
    return client.getText(todoListItem, function (result) {
        this.assert.equal(result.value, item)
    })
});

At the Given step we navigated to the index page of our UI, this is the precondition. Each action that we perform to achieve certain output needs to be specified at the when step and at the Then step we check if the expected output has been achieved or not. The API commands used can be found on the official website of nightwatch.

The whole code together looks like this

const {Given, When, Then} = require('@cucumber/cucumber')
const {client} = require('nightwatch-api')

//css selectors
const todoInputField = '.todo-input'
const todoCreateButton = '.todo-button'
const todoListItem = '.todo .todo-item'

Given('a user has navigated to the homepage', function () {
    return client.url("http://localhost:3000");
});


When('the user adds {string} to the todo list using the webUI', async function (item) {
    await client.waitForElementVisible(todoInputField)
        .click(todoInputField)
        .setValue(todoInputField, item)
        .click(todoCreateButton)
    return client
});

Then('card {string} should be displayed on the webUI', function (item) {
    return client.getText(todoListItem, function (result) {
        this.assert.equal(result.value, item)
    })
});

Now you can run the test again and they should pass.

npm run test:e2e tests/acceptance/feature/todo.feature                                

> todo@0.1.0 test:e2e
> cucumber-js --require tests/acceptance/cucumber.conf.js --require tests/acceptance/stepDefinitions "tests/acceptance/feature/todo.feature"

ℹ Connected to localhost on port 4444 (328ms).
  Using: chrome (87.0.4280.66) on linux platform.

..√ Element <.todo-input> was visible after 69 milliseconds.
.√ Passed [equal]: clean room == clean room
..

1 scenario (1 passed)
3 steps (3 passed)
0m06.385s (executing steps: 0m06.353s)

Voila, you have successfully written and implemented an acceptance test! You can add more tests and features as per your project and need. Hope this blog helped you!

You can find the source code here

How to automate tests for your website using Nightwatch.js ? - Part 1- Setting up nightwatch

Swikriti Tripathi — Fri, 03 Dec 2021 12:45:47 +0000

Testing is an essential part of any software project, it helps you ensure that your product has the best quality and avoid regressions. Manual testing can be effective but, it's time-consuming and has less test coverage. Automated testing, on the other hand, can provide a wide range of coverage in less amount of time and is very applicable for large-scale projects with lots of functionality. In this blog, we're going to write end to end tests for a simple todo app using Nightwatch.js

Powered by Node.js, Nightwatch.js is an open-source end-to-end test automation tool for web-based applications, browser applications, and websites. For further information and guide in Nightwatch.js, you can visit their official website

In this blog, we are going to follow Behaviour Driven Development(BDD) approach, if you don't know what it means or want to learn more about it you can read these blogs

Prerequisites

Note: The commands and setup is Ubuntu-specific

Node.js installed - sudo apt install nodejs
We will be using docker to run the selenium server so you need to set up docker. You can setup docker in ubuntu with the help of this blog
If you don't want to set up docker, alternatively you can run selenium in the following ways, but I highly recommend using docker and this blog will be more focused on running selenium via docker.
- install java sudo apt install default-jdk
- Download the latest stable version of selenium server.
- Download the latest stable version of chrome driver
- Unzip the chromedriver unzip <name-of-zip-file>
- Once the file is unzipped you need to place both selenium server and chromedriver in the same folder.

Setting up Nightwatch

For this blog we are going to use a simple react todo app, you can clone it from github or if you have your own project you can follow this blog to set up tests there too.

Go inside your project and install nightwatch,nightwatch-api and cucumber

    npm install --dev nightwatch-api nightwatch @cucumber/cucumber

Install selenium-server, chromedriver . If you're not using docker and using external selenium-server and chrome-driver you can opt-out of this step.

npm install selenium-server --save-dev  
npm install chromedriver --save-dev

After this, your package.json should look something like this (versions may vary).

 "devDependencies": {
    "@cucumber/cucumber": "^8.0.0-rc.1",
    "chromedriver": "^96.0.0",
    "nightwatch": "^1.7.12",
    "nightwatch-api": "^3.0.2",
    "selenium-server": "^3.141.59"
  }

In the root level create a folder tests. Our folder structure will look something like this. I'll explain more about it later.

tests
 ┗ acceptance
 ┃ ┣ feature
 ┃ ┃ ┗ todo.feature
 ┃ ┣ stepDefinitions
 ┃ ┃ ┗ todoContext.js
 ┃ ┗ cucumber.conf.js

Create a file named nightwatch.conf.js in the root level. In this file we will be adding our configuration. You can configure it as you like by consulting the official documentation of nightwatch or you can use the configuration below

const LAUNCH_URL = process.env.LAUNCH_URL || 'http://localhost:3000';
module.exports = {
    src_folders : ['tests'],
    test_settings: {
        default: {
            launch_url: LAUNCH_URL,
            globals: {},
            selenium: {
                start_process: false,
                host: 'localhost',
                port: 4444,
            },
            desiredCapabilities: {
                browserName: 'chrome',
            },
        },
    },
};

Here LAUNCH_URL is the index URL of your project, in our case
localhost:3000, you can pass this as an environment variable too. We need to specify src_folders which is the folder where your tests reside, in our case tests.

If you're not using docker you can use the following configuration:

const LAUNCH_URL = process.env.LAUNCH_URL || 'http://localhost:3000';
module.exports = {
    src_folders: ['tests'],
    test_settings: {
        default: {
            selenium_host: '127.0.0.1',
            launch_url: LAUNCH_URL,
            globals: {},
            desiredCapabilities: {
                browserName: 'chrome',
                javascriptEnabled: true,
                chromeOptions: {
                    args: ['disable-gpu'],
                    w3c: false
                }
            }
        }
    }
}

Create cucumber.conf.js file inside the tests/acceptance folder and add the following configuration

const {setDefaultTimeout, BeforeAll, Before, AfterAll, After} = require('@cucumber/cucumber')
const {startWebDriver, stopWebDriver, createSession, closeSession} = require('nightwatch-api')

setDefaultTimeout(60000)

BeforeAll(async function (){
    await startWebDriver({})
})

Before((async function(){
    await createSession({})
}))

AfterAll(async function(){
    await stopWebDriver()
})

After(async function(){
    await closeSession()
})

In this file, we specify before and after hooks that will run before and after every test scenario.

We are done setting up nightwatch.js. In the next blog, we will learn how to write tests, run selenium server, and run tests.