DEV Community: davert

Unleashing AI to Heal Flaky Tests in CodeceptJS with Playwright

davert — Sun, 21 Apr 2024 23:18:18 +0000

Dealing with flaky tests is one of the biggest pain points for anyone doing browser and end-to-end test automation. You know the drill - tests that started to fail without any reason. Intermittent failures that sap confidence in the test suite and prevent stable deployments. Maintaining tests can take up to 50% of the time for QA test automation engineer.

Usually, the cause of failure is quite simple: the element name has changed, or a new modal window appeared. What if we could use AI to identify the root cause and try to fix the test on the fly, instead of distracting engineer from their actual work?

That's what I did.

Rather than just re-running flaky tests hoping they'll pass, or manually debugging every failure, I tried several popular AI models with new AI healing feature of CodeceptJS to fix failing tests.

The results were quite promising...

TLDR: AI CAN FIX YOUR TESTS!

But you still need to read further, to learn which models I used and how to configure them.

Failing Test

I started with a failing test which opens GitHub login page but uses incorrect locators.


Scenario('Github login', ({ I }) => {
  I.amOnPage('https://github.com')
  I.click('Sign in');
  I.fillField('Username', 'davert');
  I.fillField('Password', '123345');
  I.click('Login');
  I.see('Incorrect username or password');
});

This test is executed by CodeceptJS using Playwright as an engine. As an alternative, you can use webdriverio as the engine of choice, but I prefer Playwright as it is faster.

This test fails because there is no "Login" button on the page and no "Username" field. So test won't get to the assertion step and fail on interaction.

The failing test runs ~4 secs. We need to keep that information to see how using AI impacts the speed of a test.

Fixing tests on the fly is possible with heal plugin of CodeceptJS. It should be enabled in codecept.conf.js as well as AI provider to make things work.

ChatGPT

This was my first stop, as OpenAI's ChatGPT has been a game-changer in making AI accessible. My model of choice was gpt-3.5-turbo-0125. To start using it I had to fill my credits balance in OpenAI Console. ChatGPT is still free but if you use it via API, you need to have pre-paid credits. However, it's relatively cheap. I spent only a few cents playing with it.

I updated my codecept.conf.js file to use it as AI provider:

ai: {
  request: async (messages) => {
    const completion = await openai.chat.completions.create({
      model: 'gpt-3.5-turbo-0125',
      messages,
    });
    return completion?.choices[0]?.message?.content;
  }
}

The complete configuration is here

I executed the test few times and with --ai mode and test has passed. Yey!

I received very sane suggestions to fix tests:

1. To fix Github login
  Replace the failed code: (suggested by ai)
- I.fillField("Username", "davert")
+ I.fillField("input#login_field", "davert")
at Test.<anonymous> (./tests/ai_test.js:12:5)

2. To fix Github login
  Replace the failed code: (suggested by ai)
- I.click("Login")
+ I.click("input[value='Sign in']")
at Test.<anonymous> (./tests/ai_test.js:14:5)

Both are correct. However, test duration has risen from 4s to 12s which is not great not terrible. I can assume that every call to OpenAI can take an additional ~4s (or more).

Claude (Anthropic)

I then tried the impressive Claude model from Anthropic. Claude seemed to have stronger reasoning capabilities and was able to provide more nuanced suggestions. I liked Claude more than ChatGPT 3.5 when I had to solve my daily programming tasks. How good Claude is when it comes to fixing tests on the fly?

Claude provides free 5$ credits to start. Nice!

I updated my config to use claude-2.1 model via Anthropic SDK:

ai: {
    request: async(messages) => {
      const resp = await anthropic.messages.create({
        model: 'claude-2.1',
        max_tokens: 1024,
        messages
      });      
      return resp.content.map((c) => c.text).join('\n\n');
    }
}

This model also provided valid results:

1. To fix Github login
  Replace the failed code: (suggested by ai)
- I.fillField("Username", "davert")
+ I.fillField('#login_field', 'davert');
at Test.<anonymous> (./tests/ai_test.js:12:5)

2. To fix Github login
  Replace the failed code: (suggested by ai)
- I.click("Login")
+ I.click('.js-sign-in-button');
at Test.<anonymous> (./tests/ai_test.js:14:5)

But took almost 25 secs! That's too long!

However, I used the legacy Claude 2.1 model, Anthropic recommends Haiku model as a lightweight replacement for it.

So I updated my config:

    request: async(messages) => {
      const resp = await anthropic.messages.create({
        model: 'claude-3-haiku-20240307',
        max_tokens: 1024,
        messages
      });      
      return resp.content.map((c) => c.text).join('\n\n');
    }

However, it was not faster than claude 2.1. It took 25s.

AI assistant took 25s and used ~3K input tokens. Tokens limit: 1000K

===================
Self-Healing Report:
2 steps were healed

Suggested changes:

1. To fix Github login
  Replace the failed code: (suggested by ai)
- I.fillField("Username", "davert")
+ I.fillField('#login_field', 'davert');
at Test.<anonymous> (./tests/ai_test.js:12:5)

2. To fix Github login
  Replace the failed code: (suggested by ai)
- I.click("Login")
+ I.click('Sign in');
at Test.<anonymous> (./tests/ai_test.js:14:5)

I think Claude's general purpose model is not the best choice for healing tests.

Mixtral

Mixtral was something new to me. I never tried to use mostly because it doesn't have common chat interface as Claude or ChatGPT. But recently I discovered service Groq that provides the same experience as ChatGPT and Claude but for opensource models like Mixtral.

Btw, this model is opensource so you can run it on Claudeflare, Google, Amazon, or locally. However, I will use groq due to its simplicity.

And at this moment it's free to use!

I updated my config to use mixtral via groq:

    request: async (messages) => {
      const chatCompletion = await groq.chat.completions.create({
          messages,
          model: "mixtral-8x7b-32768"  // llama2-70b-4096 || gemma-7b-it || llama3-70b-8192 || mixtral-8x7b-32768
      });
      return chatCompletion.choices[0]?.message?.content || "";
    }

Well, this was faaast!

The suggestions were good, and quite surprisingly it took 7s to fix 2 steps!

LLAMA

Another opensource model available from Groq is LLAMA from Meta. Let's try llama2:

    request: async (messages) => {
      const chatCompletion = await groq.chat.completions.create({
          messages,
          model: "llama2-70b-4096"
      });
      return chatCompletion.choices[0]?.message?.content || "";

Llama2 failed to provide a fix.

According to the logs it suggested code fixes for Codeception PHP testing framework, which didn't work in JS:

Here's an example code snippet that demonstrates how to adjust the `click` method to fix the test:

$I->click('Login');

Very bad. Maybe llama3 is better? I also tried llama3-70b-8192 model:

It failed

but after a few tries it resulted in fixing one step

Also, I don't like the suggested locator

1. To fix Github login
  Replace the failed code: (suggested by ai)
- I.click("Login")
+ I.click('a[href*="login"]')

Llama seems like very fast (7s) but very unstable for tasks like this. Maybe another day?

Gemma

Groq provides access to Gemma model from Google. Why not adding model gemma-7b-it to competition?

It worked and worked fast!

But after a few tries, I didn't manage to reproduce the result. Sometimes its response is not valid JS code so it can't be taken into action:

**Solution:**

The adjusted locator `'.auth-form-body form button[type="submit"]:contains("Sign in")'` addresses the issue by:

Unfortunately, CodeceptJS can't run the code from its words.

Conclusion

AI can improve the way we keep our end-2-end tests up to date. If you run a lot of browser tests on CI you should already try to use AI assistant to reduce flakiness of your tests. Just why not?

If it fixes a test, it will use much less time then you would do reproducing and re-running this test.

If it fails, you lose nothing. AI is quite cheap, esp comparing to QA engineers rates.

As of April 2024 the best model you can use to heal flaky tests is Mixtral mixtral-8x7b-32768. True winner! It took about 1s for request. You can enable it today with Groq in your CodeceptJS setup and have your tests fixed up on the fly!

ChatGPT-Turbo gpt-3.5-turbo-0125 also showed a good result. Last year on same task it took ~20-30s per request. Serious improvement, as the request time was reduced to 4s.

P.S. Sure, general LLM are not stable and can't guarantee the same results. But even if they can fix 10% of failures for you, that can make huge impact!

Source code:
https://github.com/DavertMik/codecept-ai-demo/tree/demo

How to fake time inside Docker container

davert — Mon, 23 Oct 2023 16:46:00 +0000

Let me share a working advice you can use in case you need to jump in time in a Docker container. You may need this for testing purposes, for instance, to test some events in the future.

You will need a Docker Image of the app you need want to launch.

Create Docker file:

FROM my-image
RUN apt update -y && apt install -y faketime

Build a container naming it fake-time:

docker build -t fake-time .

Run new container

docker run fake-time:latest faketime '2030-12-18 19:46:00' ./your-app

Or use it inside docker-compose:

  service:
    image: "fake-time:latest"
    command: "faketime '2030-12-18 19:46:00' ./your-app

Follow me on X for more development tips.

Practical Introduction to CodeceptJS: Writing End-to-end Test for Checkout Form

davert — Fri, 02 Dec 2022 03:51:00 +0000

The heart of every ecommerce website is its 'checkout' process. Whatever happens to a shop this should work to ensure the cashflow. That's why the checkout should be regularly tested. To reduce human factor and establish stability of the process we recommend using test automation framework. The simplest one to start is CodeceptJS

What-what? Did you say CodeceptJS?

Yes! CodeceptJS is a popular open-source testing framework for JavaScript. It is designed to make it easy to write and maintain end-to-end tests for web applications, using a readable and intuitive syntax. To run tests in browser it uses Playwright library from Microsoft.

CodeceptJS was first released in 2015, and it has since become a popular testing framework for JavaScript developers. It is widely used by organizations of all sizes, from startups to large enterprises.

Let's get CodeceptJS installed!

To install CodeceptJS, you will need to have Node.js and npm (the Node.js package manager) installed on your system. You can check if you already have these tools installed by running the following commands in a terminal:

node --version
npm --version

If either of these commands return an error, you will need to install Node.js and npm before you can install CodeceptJS. You can download and install the latest version of Node.js from the official website, which includes npm.

To install CodeceptJS create a new folder and run command form terminal:

npx create-codeceptjs .

If you run the npx create-codeceptjs . command, it will install CodeceptJS with Playwright in the current directory.

The npx command is a tool that comes with npm (the Node.js package manager) and it allows you to run npm packages without having to install them globally on your system.

It may take some time as it downloads browsers: Chrome, Firefox and Safari and creates a demo project.

But we are here to write a checkout test, right?

Let's initialize a new project for that!

Run

npx codeceptjs init

Agree on defaults (press Enter for every question asked). When asked for base site URL, provide a URL of a ecommerce website you are testing. For instance, it could be: https://myshop.com if you test already published website or http://localhost if you run the website locally.

When asked for a test name and suite name write "Checkout". It will create the following dirctory structure:

.
├── codecept.conf.js
├── package.json
└── Checkout_test.js

The codecept.conf.js file in the root of the project directory contains the global configuration settings for CodeceptJS.

Now open a test:

Feature('Checkout');

Scenario('test something', ({ I }) => {
});

Inside the Scenario block you write a test.

Add I.amOnPage('/') into it. It will open the browser on a URL you specified as a base.

Feature('Checkout');

Scenario('test something', ({ I }) => {
  I.amOnPage('/')
});

But you may want to ask...

What is I?

Glad you asked!

In CodeceptJS, the I object is used to represent the user performing actions in a test scenario. It provides a number of methods (also known as actions) that can be used to simulate user interactions with the application under test.

Some of the most popular actions of the I object are:

I.amOnPage(url): This action navigates the user to the specified URL.
I.click(locator): This action simulates a click on the element identified by the given locator.
I.fillField(field, value): This action fills the specified field with the given value.
I.see(text, context): This action checks that the given text is visible on the page (or in the specified context).
I.selectOption(select, option): This action selects the specified option from the given select dropdown.
I.waitForElement(locator, timeout): This action waits for the specified element to appear on the page, up to the given timeout.
I.waitForText(text, timeout, context): This action waits for the given text to appear on the page (or in the specified context), up to the given timeout.

We will need to use them to navigate into Checkout process. How do we navigate web? Sure by clicking on links!

Let's use I.click() for that.

But how we can access elements on a webpage?

CodeceptJS is smart enough to locate clickable elements by their visible text. For instance, if on your ecommerce website you have a product 'Coffee Cup' with that exact name you can use

I.click('Coffee Cup');

But sometimes elements are not as easy to locate, so you can use CSS or XPath locators to locate them.

For instance, locating Coffee Cup via CSS can take into accont HTML structure of a page and element attributes. For instance, it can be like this:

I.click('div.products a.product-name[title="Coffee Cup"]');

In this example, the div.products part of the selector specifies a div element with the products class, and the a.product-name[title="Coffee Cup"] part specifies an a element with the product-name class and the title attribute set to Coffee Cup.

You can read more about HTML and CSS locators, and basically that's all what you need to know to start writing a checkout test!

Get back to Checkout

Let's see how a regular checkout script may look in CodeceptJS:

Scenario('test the checkout form', async ({ I }) => {
  // we select one product and switched to checkout project
  I.amOnPage('/');
  I.click('Coffee Cup');
  I.click('Purchase');
  I.click('Checkout');

  // fill in the shipping address
  I.fillField('First Name', 'John');
  I.fillField('Last Name', 'Doe');
  I.fillField('Address', '123 Main St.');
  I.fillField('City', 'New York');
  I.selectOption('State', 'New York');
  I.fillField('Zip Code', '10001');

  // select a payment method
  I.click('#credit-card-option');
  I.fillField('Card Number', '1234-5678-9012-3456');
  I.fillField('Expiration Date', '12/22');
  I.fillField('Security Code', '123');

  // click the checkout button
  I.click('Checkout');

  // verify that the checkout was successful
  I.see('Your order has been placed successfully!');
});

Sure, in relaity your script might be more complicated. As you have noticed, we used CSS locator '#credit-card-option' to get select a payment option. However, the test is simple and you can follow user steps through it.

Please note, that you shouldn't use a real credit card number here. Good news, you don't need to. Payment providers like Strip provide dummy card numbers for testing purposes.

Run the test with next command:

npx codeceptjs run --debug -p pauseOnFail

What are special options here?

--debug flag is used to output additional information to the console, such as the details of each step in the test, the values of variables, and the results of test assertions. This can help you to identify and fix any issues in your tests.
-p pauseOnFail option is also used to keep the browser opened even if a test fails. It will help us to identify to which point test was executed and what can be improved.

Add more test steps if needed, update locators, and notify business owners that all that purchases are made by you so your collegues won't call you in the night asking when you want to get a coffee cup 😀 Also the good idea is to run tests on staging website, to not interfere with business process.

What a test is complete you can run it with:

npx codeceptjs run

If you are annoyed to see a browser window you can use HEADLESS environment variable:

HEADLESS=true codeceptjs run

for Windows users HEADLESS should be set in a different manner:

set HEADLESS=true&& codeceptjs run

The tests will pass but no browser is shown, so you can watch YouTube videos while it goes!

Refactoring

What if you need to check more purchases? Should you copy paste your code for that?

No! You can use Page Object pattern to put repeating interactions into the reusable functions.

You can create a page object via next command:

npx codeceptjs gpo

Sure, we will call it Checkout. It will be created in ./pages/Checkout.js file. You should enable it in codecept.conf.js inside include section:

    include: {
    ...
      checkoutPage: './pages/Checkout.js',
    },

Now open this file:

const { I } = inject();

module.exports = {

  // insert your locators and methods here
}

Feels really empty. What should we do about it? Should we write more code? No, we already have it. Let's copy code blocks from a test we have it and place them under a corredponnding function names:

connst { I } = inject();

module.exports = {

  fillShippingAddress(name, address, city, state, zip) {
    I.fillField('Name', name);
    I.fillField('Address', address);
    I.fillField('City', city);
    I.fillField('State', state);
    I.fillField('Zip', zip);
  },

  fillValidCreditCard() {
    I.click('#credit-card-option');
    I.fillField('Card Number', '1234-5678-9012-3456');
    I.fillField('Expiration Date', '12/22');
    I.fillField('Security Code', '123');
  },

  checkout() {
    I.click('Checkout');
  },
}

After that we can update our test to use the created page object. Note, that we import Checkout PageObject by its name checkoutPage we previously defined in a config.

Scenario('test the checkout form', async ({I, checkoutPage}) => {
  I.amOnPage('/');
  I.click('Coffee Cup');
  I.click('Purchase');
  I.click('Checkout');

  // fill in the shipping address using the page object
  checkoutPage.fillShippingAddress('John', 'Doe', '123 Main St.', 'New York', 'New York', '10001');
  checkoutPage.fillValidCreditCard();
  checkoutPage.checkout();

  // verify that the checkout was successful
  I.see('Your order has been placed successfully!');
});

As you see the code of a test was reduced. And we can write the similar tests on the same manner.

By applying more and more cases you can test a website to all behaviors.

Summary

This was a deep dive! If you think on just starting test automation, CodeceptJS is the best choice for you as it uses native language to pass commands to browser.

If you already skilled in JavaScript, with CodeceptJS you can focus on business level of your test, instead of writing code for browser. This way you can keep your tests stable and maintainable.

To provide a day-to-day status of a tested website use Testomat.io for test reporting. So you always know when the last checks were executed and which of tests passed and failed.

Enjoy the testing!

Planning Your Next Cypress.io Tests with Testomat.io

davert — Tue, 05 Apr 2022 23:52:03 +0000

It is so easy to start writing tests with Cypress.io. It seems that Cypress opened doors to testing for everyone with basic JavaScript knowledge. However, when the number of tests grows writing new tests becomes harder. And the issue is not just about writing code. Planning new test scenarios, refactoring, configuring pipelines, and getting test run reports. The more people involved in the product development the more visibility the testing process needs.

To keep end 2 end tests organized, following questions should be answered:

what tests do we have?
what parts of an application are covered with tests?
what new test scenarios should be implemented?
what is the current state for passed and failed tests?
if there are flaky tests on a project?

🤔

Cypress.io is just a test runner, but it can be paired with Testomat.io, a test management system for automated tests.
Let's see how Testomat.io can answer the questions risen above. We will use Cypress Realworld App, which is a real-world project (as you might have guessed) with a well-written Cypress tests.

🤔 What tests do we have?

But answering the very first question is not as simple as it seems.

All Cypress.io end 2 end tests are written as JavaScript code so to read them a person should be at least familiar with JavaScript, GitHub, etc. So this already clothes doors for business analysts, product managers, and manual QAs, for all these folks who do not actually code.

That's why we can import a Cypress realworld app into a newly created project on Testomat.io.

Importing happens with a magical npx check-tests command which scans all the tests on a project and imports them into UI without executing them. This way Testomat.io can present those tests to non-tech folks.

To import only UI tests from realworld app project, we need to run check-tests command with --typescript flag (as the project is written in TypeScript) specifying that we use Cypress.io and the path to tests via --dir parameter:

TESTOMATIO=a9ae6798atoj npx check-tests@latest Cypress.io "**/*{.,_}{test,spec}.ts" --dir cypress/tests

a9ae6798atoj is an API key of a project you are importing to, so in your case, it will be a different one.

If you use Import from Source button, you will get this command generated. Take note, that by default it imports tests from cypress/integration folder which doesn't exist in cypress-realworld project. So you need to change path and use cypress/tests instead.

Importing is fast, it takes just a few seconds to get this output.

Then all imported tests can be reviewed.

That's great that all tests are finally visible to team members. But is there a way they could be linked to specifications they are testing? Yes, and that answers the next question:

🤔 What parts of an application are covered with tests?

If the development management happens in Jira, Testomatio can link automated tests to Jira issue. You can link tests from a Cypress spec to an Issue:

Now all tests will be available within Jira, so can be taken into the development process. Revealing automated tests to Jira Issues helps managers and business analysts to understand more information on quality control. They don't need to leave Jira to see those automated tests!

The only thing needed, is a Testomatio Jira plugin should be installed from a marketplace and a Jira Project to be connected to Testomatio project.

And we are almost ready to answer the next question:

🤔 What new test scenarios should be implemented?

A project manager or a business analyst knows what features are coming next so they are the right person to propose the next tests.

As they can't be asked to write a code, or open an issue for each test, they could create in Testomat.io test directly from Jira. After that, a developer can take the written scenario and implement it using Cypress.io.

As you see on the screenshot, step autocompletion can be used to help non-tech folks write scenarios efficiently.

The test with a description is treated as a manual test, but once it is implemented as Cypress test, and re-imported it will be marked as automated.

🤔 What is the current state for passed and failed tests

Testomat.io can display and store Cypress reports. To send run reports to Testomat.io a @testomatio/reporter package should be installed and enabled in cypress/plugins/index.ts.

// inside cypress/plugins/index.ts
// require testomatio reporter package:
const testomatioReporter = require('@testomatio/reporter/lib/adapter/cypress-plugin');


export default (on, config) => {
  testomatioReporter(on, config);

  // ....

This reporter can be added to any Cypress.io project, following instructions from Testomat.io

Then tests should be executed via npx cypress run with few extra parameters:

TESTOMATIO=a9ae6798atoj npx cypress run

a9ae6798atoj is an API key of a project to which reports will be sent.

A report will contain a list of passed and failed tests, with videos and screenshots. External S3 storage from one of the cloud providers (AWS, DigitalOcean, Google Cloud, etc) can be used to keep unlimited screenshots and videos.

Videos and screenshots can be enabled in Testomatio by adding S3 credentials into .env file of Cypress Realworld project:

S3_ACCESS_KEY_ID=aws-key
S3_SECRET_ACCESS_KEY=aws-secret
S3_REGION=ap-south-1
S3_BUCKET=testomatio-test

A complete historical data of all Cypress.io test executions are also available:

If we have this data the final question can be answered:

🤔 Are there flaky tests on a project?

Testomat.io has an analytics dashboard where all tests are analyzed to provide valuable insights. For instance, it is important to keep track of flaky tests, and slowest tests, and check for the most common defects across tests.

Final Thoughts

Developing end-2-end tests with Cypress.io is fun. But making them first-class citizens in project development is also important. Cypress End 2 end tests must be visible to the whole team, so even team members who are not directly involved in testing could track the progress, propose new changes to tests, and see the actual test coverage.

Testing is a crucial part of the development process. The more clear it is and the more team members are aware of it, the more quality software can be delivered. While Cypress.io helps build efficient and fast tests for the modern web, Testomat.io provides a feature-rich test management solution for all kinds of automated tests.

P.S. Testomat.io is absolutely free to use for personal projects. Try it out with Cypress realworld app, or any other Cypress project.

5 reasons you should not use Protractor in 2020

davert — Fri, 27 Sep 2019 12:49:15 +0000

TLDR: There is a chance that you might still want to use Protractor when version 6 is released. But not today. As of today, it is recommended to switch to modern frameworks like webdriverio or codeceptjs.

Protractor is a very popular end 2 end testing framework for NodeJS. It was started as a testing tool for Angular Framework but then overgrew to become a de facto standard of testing in JavaScript. However, nowadays, it doesn't feel so well. The web has changed, the JavaScript ecosystem has changed, the Selenium has changed, but Protractor didn't. And if you still use it, or you consider using it because of its popularity - stop that. Today is not a good day.

So what happened to Protractor?

1. Protractor is not updated

Protractor was not actively updated for a few years. No major improvements, no bug fixes, no documentation updates. You can see tons of old issues in the main repo and absolutely no movement in commits for a few months.

It also became out of sync with its main dependency Selenium WebDriver JS. Right, Protractor is based on the official Selenium WebDriver library to drive browsers over W3C protocol. However, Selenium WebDriver was not updated for two years and now they are moving to 4.0. Protractor was not actively developed as well for those years, and you may see outdated documentation or long-standing issues. Protractor is almost ready to switch to this version but this still a big issue to end-users. As they will need to rewrite all their tests!

2. Protractor 6 will break compatibility

In Protractor 5 promises were synchronized using control-flow mechanism. This was also taken from the selenium-webdriver library. However, selenium-webdriver is dropping support of control flow in version 4. This means that once Protractor is upgraded to selenium-webdriver 4 it will drop it as well. So all your tests should be rewritten using the async/await pattern.

Using async/await gives you better understanding and more control over promises. However, it can be harder for developers in tests to switch to this new style if they are not familiar with it. Currently, Protractor follows Java-style syntax, so engineers experienced with Java can easily go to Protractor. After a switch, each browser call should be wrapped with await keyword. Missing even one await will lead to broken and unpredictable tests.

So async/await is coming. You will need to update your codebase and train your engineers. The problem is we don't know when exactly it is coming! Protractor 6 was tagged but never released due to some critical issues. And no estimates when a new version will be released!

(from: https://github.com/angular/protractor/issues/5290)

3. Protractor is overcomplicated

The next problem of Protractor is the design which is highly inspired by Java and is very complicated for the JavaScript world. Protractor adds thousands of lines of code on top of a very big selenium-webdriver library and when something goes wrong you don't have any idea why and where the bug has happened.

Protractor tries to tie up selenium-webdriver, Jasmine, Angular but in the end, they produced a very complicated setup with huge configs, and hard to read tests. At least senior JavaScript developer is required to prepare a good Protractor setup.

Protractor exposes WebElements into tests. So instead of controlling a browser, you control only some HTML elements on a page. But you can't get direct access to those elements from a test, as your browser is executed externally.

Wrapping hundreds of elements, calling commands on them, filtering them and mapping brings you out of the scope of testing. You spend most of the time not writing tests but fighting web elements. Congrats! Now you are a senior element controlling engineer! Whatever it means.

4. Protractor not driving a good design

Take a look of this code. This is quite a common picture of what Protractor tests can become. You don't understand what happens and you don't even want to look into this hell!

product.element.all(by.xpath(cons.xpathproductRate())).then(function (products) {
    var i = products.length;
    (function loop() {
        product.sleep(1000);
        var product = cons.xpathproductRate(i);
        product.element(by.xpath(product)).click().then(function () {
            main.waitForElementAndClick(product, cons.linkRemoveproduct).then(function () {
                main.waitForElementAndClick(product, cons.radiobtnRemoveAll).then(function () {
                    main.waitForElementAndClick(product, cons.btnRemoveproduct).then(function () {
                        i = i - 1;
                        if (i > 0) {
                            loop();
                        }
                    });
                });
            });
        });
    })();
});

You can say that bad code can happen everywhere. But this is a result of overcomplicated design in Protractor. The indirect control of web elements with promises everywhere tends to drive design full of chained methods and loops.

This makes tests unstable and not readable to humans.

5 Protractor is not required for Angular

Protractor started as a primary tool for the AngularJS framework. It was the most popular framework for single-page applications when there was no React or VueJS. Angular is still widely popular but the truth is that you don't need Protractor to test Angular application.

The only key feature of Protractor for Angular was synchronization. Protractor waits for Angular to finish rendering before taking any action on-page. This worked fine in the era of AngularJS 1 but it is less and less stable as Angular evolves. Modern Single Page Applications are built around components and there is no single point of truth if a component has finished rendering or not.

So instead of relying on Angular to synchronize components by some magic, it is more reliable to use explicit Selenium waits like browser.wait(EC.visibilityOf($('#abc')), 5000);

So what are alternatives to Protractor today?

Today Cypress.io takes popularity. Should you rewrite your tests to it?

The answer is NO!

Picking a tool by their popularity is a bad idea at first. But Cypress is quite different technology for different tasks than Protractor. It plays well for testing components of a web application, but it can't replace Protractor as Cypress does not support:

iframes
file uploads
native events
any other browser except Chrome
xpath
multiple windows and tabs
testing sites you don't control
page objects

So if you want to have the cross-browser support for a test, rich ecosystem with services like SauceLabs or BrowserStack, you should look for other Selenium tools instead.

webdriverio

Webdriverio is an alternative webdriver implementation not based on selenium-webdriver with elegant and more consistent API. webdriverio also support testing native mobile apps with appium.

As of today, webdriverio is the best webdriver implementation in JavaScript. It has rich functionality including react selectors, shadow dom support, it can even use DevTools protocol for testing!

Unlike, Protractor webdriverio receive constant updates, it has a very active community of developers, and it just rocks!

codeceptjs

Another alternative would be to use CodeceptJS - a framework for supercharged end 2 end testing. Unlike, Protractor CodeceptJS do not expose web elements, its tests are written in a scenario-based manner, where all actions are explained from the eyes of a user:

Scenario('create todo item', (I) => {
  I.amOnPage('/');
  I.dontSeeElement('#todo-count');
  I.fillField({model: 'newTodo'}, 'Write a guide');
  I.pressKey('Enter');
  I.see('Write a guide', {repeater: "todo in todos"});
  I.see('1 item left', '#todo-count');
});

CodeceptJS do not control a browser by its own. Instead, it delegates a browser control to other libraries, like webdriverio or Protractor. So yes, you can still execute tests in Protractor without the pain of maintaining Protractor code!

CodeceptJS takes a different approach in the end to end testing. But because you don't need to think about how to control web elements your tests become easy to follow, write, and debug. CodeceptJS provides you not only a tool to run browsers but best practice architecture to build sustainable end 2 end tests.

If you didn't have a chance to look into CodeceptJS, it's a good time to do so!

Conclusion

Protractor powers lots of tests out there. But it doesn't stand a test of time. Maintaining old code written in the days of Protractor and Angular glory can be more expensive than rewriting all tests from scratch! Nowadays, it is no use to rewrite those tests in Protractor as its future is not clear. However, it's a good day to try modern frameworks like webdriverio or codeceptjs!