Comparison of Automation Frameworks: Selenium, Playwright and Cypress

In modern software development, test automation has become an indispensable element, enabling QA teams and developers to efficiently and effectively verify the quality of their applications. Several tools have emerged to facilitate this process, such as Playwright, Selenium, and Cypress being some prominent options. This comparison will analyze and contrast the three leading tools in test automation, showcasing their strengths, weaknesses, and distinctive features.

Selenium: Widely used, it supports multiple languages and browsers, ideal for end-to-end testing.

Cypress: Modern approach, designed for in-browser testing, stands out for its simplicity, speed and efficiency in integration and unit testing.

Playwright: It is versatile, allowing the automation of advanced tasks, such as emulation of mobile devices. It stands out for its ability to work in multiple browsers and run test in parallel.

General information

In the following table you will learn the general information for each framework:

	Selenium	Playwright	Cypress
Release year	2004	2020	2017
Developer	Jason Huggins -Community	Microsoft	Cypress - Community
License	OpenSource	OpenSource	OpenSource-Cloud license
Written in	Java	TypeScript	JavaScript

Selenium is the oldest and still widely used framework/library for Automation. Playwright, on the other hand, is the newest framework, but their popularity is increasing quickly in the community. Microsoft created Playwright to address common challenges in Selenium, as well as improve integration with DevOps tools and cloud services.

	Selenium	Playwright	Cypress
Languages	JavaScript, Java, Ruby, others	JavaScript, Java, C#, Python	JavaScript only
Browser supported	Almost all of them	Chromium, Firefox, Webkit	Chromium, Firefox, Webkit, Electron
Type	Library	Framework	Framework
Execution time	Fast	Very Fast	Slow
UI extension	No	Yes	Yes

The main feature in Playwright is the performance and support for modern programming languages and frameworks. Selenium has the best support for old browsers and programming languages. Cypress has only support for the JavaScript ecosystem.

Popularity

Let's review the popularity and numbers of downloads in npm. Check the following picture:

In the last months, Playwright has overcome selenium in the number of downloads, stars and contributors on GitHub. This means a great appreciation by the community to this framework.

Selenium is widely regarded as the top choice for Java and Python due to its extensive implementation in major projects over many years.

Demos

In the following sections, we will reproduce the first steps to create our first automation test for these 3 technologies.

For these demos we will use the following web page: https://www.saucedemo.com/

The idea is to perform the login and validate that it was successful on the following screen:

Cypress

Let's review the prerequisites and commands to create our first test in Cypress.

Prerequirements

• Node.js 18 or 20

Steps:

• Go to an empty folder where you want to create the project
• Execute npm install cypress --save-dev
• Then use this command to execute the project: npx cypress open
• Now in the folder cypress/e2e create a new test with the name spec.cy.js
• Add the following code to create a new test for the login.

describe('Sauce demo', () => {
  it('login pass', () => {
    cy.visit('https://www.saucedemo.com/#/')
    cy.get('#user-name').type('standard_user')
    cy.get('#password').type('secret_sauce')
    cy.get('#login-button').click()
    cy.get('.app_logo').should('have.text', 'Swag Labs')
  })
})

In cypress, we type cy to use the API and call the functions to detect elements, set values and navigate in the page. The code is petty simple and easy to understand. We use #element-id cal get elements by Id and .class to get the element by CSS class. We don't need to install extra libraries or components to create our first tests.

Selenium

Let's review the prerequisites and commands to create our first test in Selenium with JavaScript.

Prerequirements

• Node.js 18 or 20

Steps:

• Go to an empty folder where you want to create the project

• Execute the following commands:

  npm install selenium-webdriver

  npm install chromedriver

  npm install mocha

NOTE: selenium-webdriver is used to interact with the browser, chromedriver is used to execute the test in chrome so Selenium can control it and mocha help us to create the test like a simple unit test and capture the result.

• Create a file test.js where the test will be located
• Add the following code to perform the test on the login page with Selenium:

const Chrome = require('selenium-webdriver/chrome');
const {By, Builder} = require('selenium-webdriver');
const assert = require("assert");
const options = new Chrome.Options();
const {suite} = require('selenium-webdriver/testing');


suite(function (env) {
    describe('Should be able to Test Command line arguments', function () {
      it('headless', async function () {
    let driver;

    driver = await new Builder()
            .setChromeOptions()
            .forBrowser('chrome').build();
    await driver.get('https://www.saucedemo.com/');
    await driver.manage().setTimeouts({implicit: 500});
    await driver.findElement(By.id('user-name')).sendKeys('standard_user');
    await driver.findElement(By.id('password')).sendKeys('secret_sauce');
    await driver.findElement(By.id('login-button')).click();
    var textLogo = await driver.findElement(By.className('app_logo')).getText();
    assert.equal('Swag Labs', textLogo);
    await driver.quit();
});
});
});

Selenium has a better experience in other programming languages, like Java, Python and C#. Since Selenium is a library, we need to install other libraries to complement it. And install the drivers to connect with the browser where we want to run the test.

Playwright

Let's review the prerequisites and commands needed to create our first test in Playwright with JavaScript.

Prerequirements

• Node.js 18 or 20

Steps:

• Go to an empty folder where you want to create the project
• Run npm init playwright@latest to create the base project
• Then use this command to execute the tests: npx playwright test
• Create a new file mytest.spec.js in the folder tests
• Add the following code to test the login page using Playwright:

const { test, expect } = require('@playwright/test');

test('Login pass', async ({ page }) => {
  await page.goto('https://www.saucedemo.com/');
  await page.locator("#user-name").fill("standard_user");
  await page.locator("#password").fill("secret_sauce");
  await page.locator('#login-button').click();

  // Expect a title "to contain" a substring.
  await expect(await page.locator('.app_logo')).toHaveText('Swag Labs')
});

In playwright, we use the word page to interact with the browser. We can also use the method .toHaveText to validate the result and create assertions. Playwright is not only incredibly user-friendly for developers, but it also boasts a seamless functionality that requires no additional dependencies or extra libraries in order to work perfectly in a variety of scenarios.

Developer Experience

JavaScript's developers find Cypress easy to understand because the tests are similar to unit tests in JavaScript with libraries like jest. Cypress may be limiting and restrictive for automation QA professionals who prefer to have full control over their workflow and do not rely on the UI for test inspection and execution. Cypress only works with JavaScript, so it's hard to compare it with Playwright, which is popular among other developers.
Selenium has a high flexibility and I can combine it with my favorite libraries. Once the project is established, we can enhance the test structure, streamline the main framework, and optimize code reuse. This approach is favored by many developers.

UltimateQA posted a survey about the user experience for these frameworks. Ultimate QA survey

The results seem inaccurate because Cypress and Playwright are incredibly user-friendly, yet Selenium remains the most widely used automation library and many developers only have familiarity with it.

Possible Scenarios vs Best Option

Let's analyze some common scenarios and which framework is a better fit.

1. A modern application with a small team that uses a new Javascript framework (React.js, Angular, etc..). The team doesn't have experience with automation testing. For this scenario, the best option could be Cypress or Playwright. Developers can easily work with the UI provided for these frameworks and work with JavaScript.

2. An app with a modern backend (Node.js, Python, C#, etc.) and modern frontend (React.js, Angular, etc.) with a big team.
   For this scenario, we can use all the 3 options. But to work with something modern Playwright could be the best one. We can work with different languages according to the team knowledge and Playwright has been adding extensibility with Selenium.

3. A desktop application using Electron (JavaScript) with node.js backend. In this case, all these frameworks are bad options. At the moment, both Playwright and Cypress offer experimental plugins or extensions for electron, but they may not be suitable for production purposes. We can use Spectron, WebDriverIO or Selenium in conjunction with other libraries to perform our tests.

Conclusions

In modern software development, the main test automation tools are Selenium, Cypress, and Playwright. Each of these tools offers unique strengths and weaknesses, catering to different needs within the software testing landscape.

Selenium, as the oldest and widely used framework, excels in supporting multiple languages and browsers, making it ideal for end-to-end testing. Cypress, with its modern approach, stands out for in-browser testing, simplicity, and efficiency in integration and unit testing. Playwright is a versatile automation tool that excels in multiple browser support, parallel test execution, and automating advanced tasks like mobile device emulation.

The popularity metrics, including downloads, stars, and contributors on GitHub, depict Playwright's recent surge, surpassing Selenium in community appreciation. However, Selenium remains a top choice for many teams.

Ultimately, the choice of test automation tool depends on project requirements, team expertise, and the application being tested.

In my opinion, Playwright will continue growing, and it is a modern and fast tool. I see a great projection for Playwright in the future.

If you want to try the demos and access the repo, click on the following link:
https://github.com/Mteheran/AutomationDemos

Comments

[Christine Pinto]

Thank you for this comprehensive overview of Selenium, Playwright, and Cypress! Your comparison provides valuable insights for anyone navigating the options for automation frameworks. I appreciate the effort in putting this together. 🙌

I'd like to add a few points from my experience:

• It's worth noting that Playwright supports TypeScript out of the box, which is a huge advantage for teams looking for strong typing in their test scripts.

• While Cypress offers an excellent testing environment, the cost associated with features like the Cypress Dashboard can be a consideration for some projects.

• It's great that both Selenium and Playwright support testing Chrome extensions, expanding their utility for more comprehensive browser testing.

Having used all three, I'm particularly impressed by Playwright's speed/performance. Being a Microsoft product, its seamless integration with Visual Studio Code is another plus, making it incredibly efficient for development workflows.

Thanks again for shedding light on these frameworks. It's insights like these that make our community so valuable!

[Miguel Teheran]

Thans for sharing your experience. The Playwright framework is addressing the weaknesses of other frameworks, and we can look forward to its evolution in the coming years.