DEV Community: TestCafe

Randomize Your End-to-End Tests: How to Generate Input Data for TestCafe

TestCafe — Wed, 13 Jan 2021 10:47:11 +0000

User input can be truly unpredictable, so it's important to use a wide sample of random input data when testing web forms. In this article, we'll take a look at three Node libraries that generate data: nanoid, faker, and generate-password. We'll see how TestCafe can leverage their features to help you improve your test coverage.

The farfurix/generating-input-data git repo contains the custom demo page and the test examples created specifically for this article. Clone the repository to follow along.

The Basics (nanoid)

This example page contains a text input field. We can use the t.typeText method to populate it:

    import { Selector } from 'testcafe';

    fixture `Random Input Data: example 1`
        .page('https://devexpress.github.io/testcafe/example/');

    test('Generate a random name', async t => {
        await t
            .typeText('#developer-name', 'Developer Name');
    });

Nanoid generates random strings when you call the nanoid() method. We can call this method to randomize our input data.

Import the nanoid module in the beginning of the file:

    import { nanoid } from 'nanoid';

Declare a constant with a randomly generated name:

    const randomDeveloperName = 'testuser_' + nanoid();

Replace the 'Developer Name' string with the constant we just declared:

    await t
        .typeText('#developer-name', randomDeveloperName);

Now every time you run the test, the input value will be unique.

Input validation (faker, generate-password)

Imagine a more complicated scenario: testing a sign-up form's ability to validate user passwords. We need to make sure that the password chosen by the user contains:

At least eight symbols
At least one digit
A lowercase letter
An uppercase letter.

First, we're going to need a valid email address to serve as our username. Faker — a Node.js library that specializes in generating realistic-looking data, such as home addresses, business details, and emails — can help us. Import the library and call the faker.internet.email() method to create a valid email address.

    import faker from 'faker';

    const validEmail = faker.internet.email();

The generate-password library will give us the passwords we need. We can set password requirements by passing arguments to the generate method of the generator object. Let's import the library:

    import generator from 'generate-password';

We're ready to create an array of random, yet valid, passwords. The faker library will, once again, prove itself useful — this time, it will help us determine the password length.

    let validPasswords = [];

    for (let i = 0; i < 5; i++) {
        let newRandomPassword = generator.generate({
            length: faker.random.number({ 'min': 10, 'max': 20 }), // 10-20 characters long
            numbers: true,
            uppercase: true,
            lowercase: true,
            strict: true
        });

        validPasswords.push(newRandomPassword);
    };

It's just as easy to generate invalid credentials. Let's use the generateMultiple method to generate invalid passwords of varying length:

    // These passwords are too short
    const shortPasswords = generator.generateMultiple(5, {
        length: 7,
        numbers: true,
        strict: true
    });

    // These passwords lack uppercase characters
    const passwordsWithoutUppercase = generator.generateMultiple(5, {
        length: 8,
        numbers: true,
        uppercase: false,
        strict: true
    });

    // These passwords lack lowercase characters
    const passwordsWithoutLowercase = generator.generateMultiple(5, {
        length: 8,
        numbers: true,
        lowercase: false,
        strict: true
    });

    // These passwords lack digits
    const passwordsWithoutDigits = generator.generateMultiple(5, {
        length: 8,
        strict: true
    });

    const invalidPasswords = shortPasswords.concat(passwordsWithoutUppercase, passwordsWithoutLowercase, passwordsWithoutDigits);

Now that we have our fake credentials, we can test the web form.

The first test will feed the form valid passwords. To do this, let's iterate over the validPasswords array and enter the data it contains into the form. A confirmation of the password's validity should appear every time we click the #submit button.

    test('Successful password validation', async t => {
        for (const validPassword of validPasswords) {
            await t
                .typeText('#email', validEmail, { replace:true })
                .typeText('#password', validPassword, { replace: true })
                .click('#submit')
                .expect(Selector('#password-status').value).eql('Valid password with a length of ' + validPassword.length);
        };
    });

The second test will iterate over the invalidPasswords array. The main difference between this test and the previous is the content of the password status message. The message should read: "Invalid password".

    test('Invalid password warning', async t => {
        for (const invalidPassword of invalidPasswords) {
            await t
                .typeText('#email', validEmail, { replace: true })
                .typeText('#password', invalidPassword, { replace: true })
                .click(submit)
                .expect(Selector('#password-status').value).eql('Invalid password');
        }
    });

The farfurix/generating-input-data git repo contains full versions of the examples above, as well as a demo page to run these tests against.

As we just demonstrated, it's easy to use third-party data generators in conjunction with TestCafe to randomize your tests' user input. Just remember: not all data generators are created equal. Select one based on your website's unique needs.

Client Functions and Script Injection for Page Interaction

TestCafe — Mon, 23 Nov 2020 18:00:14 +0000

You will learn how to run custom JavaScript code on the tested pages, how to use objects like the page model in this code, how to wait for an arbitrary event in the browser, and the fastest way to run a one-liner or an entire JS module.

End-to-end tests emulate how users interact with the application. In theory, this interaction consists of simple actions like a mouse click on an element, a key press, drag-and-drop, or file upload. The TestCafe API allows you to perform these basic actions with a single line of code. However, real-world applications tend to be more complicated than a static entry form. Some applications cannot be tested without closer interaction with tested pages. In this article, we review several examples and show how knowledge of TestCafe's internal architecture allows you to test complex or poorly designed pages with the help of client functions and script injection.

Use Client Functions to Run Code in the Browser

Let's assume you need to click an element, but another element on the same page overlaps the first one due to design or implementation choices. If you simply call t.click, TestCafe waits for the target element to appear in the foreground. When a predefined timeout expires, TestCafe will eventually click the overlapping element. If the test author intended this click, the test completes successfully. However, the entire timeout period is wasted on aimless waiting. You can hide the overlapping element and avoid this pause if you run the following code:

test('Hide element', async t => {
    const hideElement = ClientFunction(() => {
        document.querySelector('#id').style.display = 'none';
    });
  
    await hideElement();
  
    await t.expect(Selector('#id').visible).notOk();
});

You may also want to update that code and store the element ID in a variable:

test('Hide element', async t => {
    const elementId = '#id';
    const hideElement = ClientFunction(() => {
        document.querySelector(elementId).style.display = 'none';
    });
  
    await hideElement();
  
    await t.expect(Selector(elementId).visible).notOk();
});

A variable is a good idea and it can be used, but not in this manner. The test above would fail with an error:

My Fixture
 × Hide element
   1) An error occurred in ClientFunction code:
      ReferenceError: elementId is not defined

On the surface, your JavaScript programming experience may tell you that the test is fine. Let's look into TestCafe's underlying mechanisms in greater detail to see why the test fails.

Under the Hood

The TestCafe framework is based on a client-server architecture. The test code runs in a Node.js process (server) and uses a proxy server to interact with the browser (client).

Considering this fact, you can expect that:

The test code has no direct access to the browser context.
TestCafe interacts with the browser asynchronously (which is why missing await issues are often posted in GitHub and StackOverflow).
The server-side TestCafe process and the client-side browser can only exchange serialized data.

We can simplify TestCafe's client-server interactions as follows:

On the server, the TestCafe API exposes methods that execute actions and user code in the browser. When the API is triggered, TestCafe creates commands and passes them to the proxy: TestCafe Hammerhead.
TestCafe Hammerhead transmits commands from the server to the browser and sends back the results when they are ready.
On the client, TestCafe scripts receive these commands and execute them in the same way the browser handles user actions.

Pass Variables to Client Functions

Create parameterized functions or use page model properties on the client side.

Let's get back to our code that tries and fails to introduce a variable.

test('Hide element', async t => {
    const elementId = '#id';  
    const hideElement = ClientFunction(() => {
        document.querySelector(elementId).style.display = 'none';
    });
  
    await hideElement();
  
    await t.expect(Selector(elementId).visible).notOk();
});

This function has to be executed in the browser. As we mentioned earlier, only serialized data can travel between the client and the server. This is why TestCafe serializes this function (i.e., converts it to a string). This string is then evaluated when received on the browser side. You can now see why closures do not work for client functions.

What's the solution then? Before TestCafe converts the function to a string, a separate serializer module processes arguments passed to ClientFunction(). This is the same module that serializes all data that travel between the client and the server. This means that you can easily pass parameters to the client function and they arrive safely.

In addition, you can also find the dependencies property among the options that can be passed to the ClientFunction constructor.

TestCafe offers two options to access variables declared in test code.

The first option is to pass the variable as an argument:

test('Hide element', async t => {
    const elementId = '#id';
  
    const hideElement = ClientFunction(targetId => {
        document.querySelector(targetId).style.display = 'none';
    });
  
    await hideElement(elementId);
  
    await t.expect(Selector(elementId).visible).notOk();
});

But what if the element has no ID, so that it's not easy to find in the DOM tree? The second option is to pass a selector to the client function. Use the dependencies option in the ClientFunction constructor:

test('Hide element', async t => {
    const elementSelector = Selector('#populate');
  
    const hideElement = ClientFunction(() => {
        elementSelector().style.display = 'none';
    }, { dependencies: { elementSelector } });
  
    await hideElement();
  
    await t.expect(elementSelector.visible).notOk();
});

Note that client functions and selectors have a lot in common. Methods that create and run selectors derive from similar methods of client functions. To execute a complex selector query, you can pass a function to the Selector constructor, as you do for client functions. The main difference between them is the return type. Selectors should return DOMNode, while client functions should return any serializable value, or a Promise that resolves to such value.

Wait for a Custom Condition

You can specify when TestCafe considers a page loaded.

Now let's see why you may want a client function to return a Promise.

TestCafe is designed to wait for the DOMContentLoaded and load events, as well as XHR requests (allowing three seconds for the latter), before tests begin. In rare instances, the wait time may be over before all scripts on the page are initialized. The following client function pauses the test until a custom condition is fulfilled (you can replace it with your own script completion indicator).

test('Wait for scripts', async t => {
    const waitForScripts = ClientFunction(waitTimeout => {
        function checkCondition () {
            // Replace this line with a custom condition.
            return true;
        }
        return new Promise((resolve, reject) => {
            var timeout, interval;
            timeout = setTimeout(() => {
                clearInterval(interval);
                reject('Timeout is exceeded.');
            }, waitTimeout);
            interval = setInterval(() => {
                if (checkCondition()) {
                    clearTimeout(timeout);
                    clearInterval(interval);
                    resolve();
                }
            }, 100);
        });
    });
    await waitForScripts(2000);
});

Even if a browser does not support Promises, this client function still works because TestCafe automatically adds the necessary polyfills when it processes this code.

Inject Scripts to Add Code Permanently

Client functions are useful to interact with the tested page when necessary. You can even run a client function without saving it to a variable. However, everything you change in this function on the tested page is reverted once a redirect or page reload occurs. To modify the page permanently, inject client scripts into the tested page. The injected scripts are added to the <head> tag.

Scripts that Persist Throughout Testing

Should you need to hide a banner, or modify an attribute for some element, you can organize client code that does it in a separate file:

script.js

window.addEventListener('DOMContentLoaded', function () {
    const element = document.querySelector('your-selector');
    element.setAttribute('foo-bar', 'baz');
});

Next, use your favorite API to attach this file in all tests (which eliminates repetitive beforeEach hooks):

Command Line Interface

testcafe chrome test.js --client-scripts script.js

API

runner.clientScripts('script.js');

Configuration file

{  
    "clientScripts": "script.js"
}

Injected Scripts Run First

TestCafe adds the injected scripts to its automation code, so that they run before any other script on the page. You can therefore use injected scripts to alter the page behavior. For instance, you can override the window.open method for all tests as follows:

const mockWindowOpen = "window.open = function () { };";
fixture`My App`
    .page`https://devexpress.github.io/testcafe/example/`
    .clientScripts({ content: mockWindowOpen });
test('My test 1', async t => { /* ... */ });
test('My test 2', async t => { /* ... */ });

Your code remains almost unchanged. TestCafe only wraps it to catch exceptions.

Checking Other <head> Scripts for Errors

Due to their location inside the <head> tags, injected scripts also allow you to access information that is not obtainable by other means.

Consider a scenario where you need to obtain unhandled exceptions thrown in another script in the <head> section. The TestCafe API contains the t.getBrowserConsoleMessages method that returns messages printed in the console. However, this method can only access messages posted by the console.log, console.warn, and console.error methods in page code. The t.getBrowserConsoleMessages method does not fetch errors thrown by the browser. You could try to use a client function to catch the exception, but client functions are executed after all <head> scripts are completed.

With script injection, you can add the following code:

script.js

window.unhandledErrors = [];
window.onerror = message => {
    window.unhandledErrors.push(message);
};

In test code, you can create a client function to access messages in window.unhandledErrors:

test('Get Unhandled Errors', async t => {
    const errors = await t.eval(() => window.unhandledErrors);
    // ...
}).clientScripts('script.js');

Conclusion

Client functions and script injection extend TestCafe's capabilities so that you can interact with the page directly. These features can configure your application at low levels to prepare it for testing, or fetch data about its execution in a custom manner. You can think of even more ways to use client functions and injected scripts. We hope this brief overview of the TestCafe architecture and sample usage scenarios help improve your understanding of these tools so that you can use them more effectively.

The Fastest Way to Your Next Test – TestCafe + TestCafe Studio - Q&A's

TestCafe — Thu, 24 Sep 2020 17:10:24 +0000

Thank you to all that attended the recent TestCafe Studio presentation, the questions raised have been answered by the team and listed below.

Watch the Webinar

Questions & Answers

Will the webinar recording on YouTube include the Q&A section?

We discussed the most popular Q&As at the end of the webinar. This part is included in the recording. The remaining questions are answered in this post.

Can I test Windows Form applications with TestCafe Studio?

TestCafe Studio can test web applications only.

Am I allowed to install TestCafe Studio on our customer's site to test it in their environment?

TestCafe Studio is a desktop application installed on a developer's/tester's local computer, not on a website. You can install TestCafe Studio on any Windows, macOS, or Linux machine. If the machine where TestCafe Studio runs can access the website deployed on your customer's infrastructure, you can simply specify its URL to test it in its native environment.

Does TestCafe Studio support multiple languages?

The TestCafe Studio UI only supports English. If you are asking about programming languages in which you can write tests, TestCafe Studio supports JavaScript, TypeScript, CoffeeScript, and of course tests recorded in TestCafe Studio (*.testcafe files).

Can I include TestCafe Studio in Continuous Integration?

You can use the open-source TestCafe to run tests recorded or written in TestCafe Studio in CI systems. See the Integrate Tests with CI Systems topic for more information.

What value does TestCafe Studio add to an automated CI pipeline in which tests are executed in headless mode?

TestCafe Studio allows you to create and modify tests with ease. In a CI system, the tests will run with an open-source TestCafe runner that doesn't make any difference between TestCafe Studio recorded tests and JavaScript TestCafe tests. See the Integrate Tests with CI Systems topic for more information. So, TestCafe Studio's goal is to facilitate recording and maintenance.

Is there a way to record tests in TestCafe Studio and export them to JavaScript to run individually?

Yes, you can do this. See the Convert Recorded Tests to JavaScript topic for more information.

Does TestCafe Studio have any known issues with SPA apps, specifically Angular?

No, we are not aware of any significant issues with SPAs including Angular applications. You can also make use of the following selector extension that enables you to test Angular applications even easier: DevExpress/testcafe-angular-selectors

Are there any recommendations on the best method for initializing a selector?

We plan to publish an article that details the best practices for TestCafe tests, including suggestions on how to build effective selectors. In the meantime, refer to the Element Selectors topic to learn more about them.

Does the open-source TestCafe have the reporting and screen capturing capabilities?

Yes, it has both. See the Reporters and Screenshots and Videos topics for details.

Can I test "concurrent" scenarios where two users use the app and modify the data simultaneously?

To simulate users interacting with the app concurrently, you need two browser windows. Support for multiple browser windows is currently available in beta for coded tests only. See the details in the open-source TestCafe documentation: Multiple Browser Windows. As an alternative solution, you can try running the same test that edits a data record in different browsers at the same time.

Can I run tests at specific moments in time, e.g. on schedule?

You can integrate your test with any CI service to run them on schedule.

Can I receive an email notification with test results once they are done?

You can search for a TestCafe reporter that sends an email with test run results or create a new custom reporter for your needs.

Can I deliberately slow down the test run in TestCafe Studio for more convenient viewing?

Yes, you can adjust the Speed option to slow down test execution.

How do you use the GUI to create a custom attribute for a webpage?

You can customize the selector generation mechanism to take into account custom attributes. Add a custom attribute name to the list of selector types. TestCafe Studio will then use this attribute to create selectors. On your website end, you should specify the custom attribute for page elements manually.

Is it possible to write a text assertion with wildcards in it, so that it matches multiple text values?

Yes, you can use regular expressions for this purpose: Match Assertion

Is it possible to load user credentials from a file to use them in tests?

You can use the Run TestCafe Script action. Write a script that imports the fs Node.js module, reads the file content and saves the credentials in the test context. Now you can read these values from t.ctx in other scripts within Run TestCafe Script actions. Currently, you cannot pass the test context to recorded action parameters.

Can I create variables to store secrets or URLs for different environments?

TestCafe Studio can save selectors and functions to variables. In Run TestCafe Script action code, you can create other types of variables and use them according to your needs. To share variables between different scripts, store them in the test context.

Can I use conditions, add conditional jumps or go-to statements?

In recorded tests, you cannot use conditions and go-to statements. You can record test scenarios and then convert them to JavaScript to add conditions.

How to organize tests in TestCafe Studio? How can I tag Smoke and Regression tests to distinguish between them? I tried metadata but found it ineffective. Is there an alternative way in TestCafe Studio?

This is no such capability built in so far. You can split your tests across different directories or convert the recorded tests into JavaScript and use metadata with the open-source TestCafe test runner.

Can I use data from external sources (text files or databases) in my tests?

Yes, but not in the Test Editor UI. If you convert the recorded tests into JavaScript, you can use any Node.js or npm module to read data from your files or load from a database to use it in your test cases.

How can I ensure temporary objects that pop up briefly after a button is pushed or the mouse hovers over something are displayed? The display doesn't last long enough for me to click on it after activating the "=" assertion.

In recoded tests, you can add the Define Function action that checks if the target element exists every n milliseconds and returns the result when the element is found (you can use a Promise). Then add an action for click. After that, use the client function's return value in an assertion.

How would I test visuals that indicate drag-and-drop, since that visuals only exist while you are actively dragging?

TestCafe cannot execute an assertion while you are dragging. We may suggest a multi-step workaround where you inject client-side code that is triggered during the drag, checks for the visuals, and stores the result. When drag is finished, you can obtain this value via a client function and verify it in an assertion. For manual verification, you can use the open-source TestCafe to record a video of a test run to ensure that the drag-and-drop indicator is displayed.

How do I run the same test against my app versions deployed in different environments (at different URLs), e.g., FAT/UAT/production?

In coded tests, you can specify the test's start page programmatically. To test multiple versions of your app with the same test suite, start the tests as many times as you need while passing a different URL each time.

In recorded tests, you can use the same approach, but instead of setting the start page (which is not possible programmatically), begin tests with a custom script that accesses the passed URL and navigates to the required page with the t.navigateTo method.

Can variables be parameterized?

Currently, you cannot use parameters for recorded test steps but you can write any custom code in the Run TestCafe Script action.

Can the JS scripts act as a REST client to obtain assertion values (via a service call to a helper endpoint)?

Yes, you can use a Node.js HTTP client like got.

In our tests, selectors throw errors occasionally. How can we get rid of this irregular behavior?

Let us know how we can replicate this behavior. Please submit an issue in the Support Center.

Can I record accessibility tests with TestCafe Studio?

No, but you can use axe-testcafe in coded tests for accessibility testing.

How do I integrate TestCafe reports with Xray?

Check out the following custom reporter plug-in for Xray: antreyes/testcafe-reporter-xray

Can you show an example of a much more complex real-life test?

We publish TestCafe examples in this GitHub repository: DevExpress/testcafe-examples. You can find both basic examples as well as samples that address some rare scenarios.

However, you wouldn't find a really complex test among our examples because it's not good practice to overengineer end-to-end tests. After all, we created TestCafe to keep your tests simple.

Can you please show an example with the DevExpress Grid Control?

DevExpress Grid Controls can be easily tested with TestCafe like any other web control/page. You can try to record your own test on JS Grid demo pages with TestCafe Studio.

Will TestCafe be able to generate tests in other languages, like C#, or using Selenium API, etc?

Not in the nearest future.

Do you offer a testing framework for desktop applications (specifically, .NET)?

Yes, we offer Coded UI for WinForms applications.

How TestCafe Can Help You to Deliver Faster

TestCafe — Thu, 17 Sep 2020 15:51:10 +0000

DevOps teams must make certain that applications ship on time and meet appropriate quality standards. To achieve the latter objective, DevOps must carefully consider what checks to include in the pipeline and what to leave as smoke tests. In this brief article, we argue that end-to-end tests are crucial to product quality and are worth running each time an organization delivers a build. We will illustrate how TestCafe can streamline this process and help you integrate end-to-end tests into your CI/CD workflow with minimum time and effort.

Unit Tests are not Enough

As you know, some organizations/dev teams require use of unit tests. Unit tests allow developers to verify whether "small" portions of a solution work as intended, but do so independent of the whole. While it may be tempting to use green unit tests as the prime delivery condition within a CI/CD system, unit tests will always remain granular and limited in scope.

Unit tests allow developers to flag problematic code modifications, but unit tests cannot accurately determine whether all parts of a given system are working together flawlessly. Said differently, unit tests cannot establish whether all usage scenarios are fully tested and meet an organization's quality standards.

By contrast, end-to-end tests can check/analyze whether the entire system operates as expected (the whole and not just its parts). End-to-end tests can emulate user actions and determine whether real output matches expected outcomes. Unit tests are important, but we believe that only end-to-end test automation can deliver the reliability users have come to expect from today's top software organizations.

TestCafe – End-to-End Testing Made Easy

TestCafe is an enterprise-ready end-to-end testing framework with an intuitive API. Unlike legacy frameworks such as Selenium, TestCafe is easy to install, run, and integrate within an organization's CI/CD pipeline. You can install the framework via npm with a single command. Since TestCafe does not require tedious configuration or plugins, it can be wound up in a new CI system within minutes. And yes, with TestCafe, you do not need to locate, install, and configure additional testing software/modules. TestCafe is self-contained and only requires Node.js and your preferred browser (no browser plugins are needed).

In addition, TestCafe allows you to test next-gen web features with absolute ease. TestCafe supports all major modern browsers (either locally or cloud-based) – across a variety of mobile or desktop devices. TestCafe can execute its tests on macOS, Windows, Linux, iOS, and Android. You can use Chrome, Firefox, Safari, Edge and Internet Explorer when and where appropriate – you can even use headless browsers (a great way to decrease testing time and to execute browser tests on VMs with no graphical system).

As you'll see below, we've taken care of everything under the hood so you and your team can avoid configuration hassles and the steep learning curve associated with legacy testing frameworks.

How to Integrate TestCafe in Your CI/CD

TestCafe can be incorporated into a pipeline with a few simple steps:

Install Node.js
Install TestCafe. As mentioned earlier, TestCafe can be installed with a simple command: npm install -g testcafe.
Run TestCafe tests. Only a single command is required to execute a test (for instance: testcafe chrome:headless ./tests). With this simple command, TestCafe will auto detect the browser (if installed), launch it, and run your tests automatically.

TestCafe ships with five reporters – modules that can generate test execution reports for your CI/CD. JSON, JUnit, xUnit, and console output (with rich formatting) are built-in. You can also use reporters developed by the TestCafe user community for NUnit, TeamCity, Slack, etc. Should you require these community plugins, you can:

Install the desired third-party reporter via npm: npm install testcafe testcafe-reporter-nunit;
Run TestCafe tests much like step 3: testcafe chrome:headless ./tests -r nunit:report.xml

You can learn more and about TestCafe and CI system integration via the following links:

Note: For Jenkins, TestCafe can post videos and screenshot it captured during test execution. To enable this functionality, you will need to install the Jenkins plugin and use a dedicated Jenkins reporter.

Docker Image

TestCafe is easy to install and requires a very basic environment. Of course, you can wind it up even faster with Docker. To get started, download a pre-configured image from Docker Hub:

docker pull testcafe/testcafe

The image is based on ArcLinux and includes Node.js, TestCafe, Chrome, and Firefox. It is ready to execute tests as needed:

docker run -v //d/tests:/tests -it testcafe/testcafe firefox:headless /tests/**/*.js

You can also use this image on your computer to perform tests on a host machine or remote machines within your network. See the following help topic for additional Docker-related information: Use TestCafe's Docker Image

GitHub Actions

The GitHub Actions CI service is ideal for projects developed on GitHub. To help you quickly integrate TestCafe into your Actions workflows, we created the Run TestCafe action. This action installs TestCafe and runs tests on your behalf. To begin, you simply need to add TestCafe command line arguments:

- uses: DevExpress/testcafe-action@latest
  with:
    args: "chrome tests"

TestCafe: Adoption Made Easy

Ready to see TestCafe's capabilities in action? Want to see how quickly you can incorporate test automation into your CI/CD workflow?

Get Started Today or Ask Us for more information. We are here to help.

Introducing Multi-window Tests (Beta)

TestCafe — Mon, 31 Aug 2020 10:55:36 +0000

Modern web apps often launch new browser windows to authenticate users through third-party websites or display additional interactive content.

TestCafe v1.9.0 introduces partial support for multi-window tests. You can now use client-side calls to open and close browser windows. The updated API includes additional window management methods. During the 'beta' stage, this functionality is only enabled in local instances of Chrome and Firefox. Keep in mind that this feature is not ready for use in production environments. Both the syntax and capabilities are subject to change.

Client-side window events

The TestCafe browser automation driver detects and automatically activates newly launched windows. After the content is loaded, and all asynchronous requests are resolved, the test continues in the new window. When that window is closed, TestCafe switches back to the parent window and resumes the test.

The sample code below demonstrates an OAuth login scenario. Lines 8 through 10 run in the external OAuth window.

fixture('Login via <Service Name>')
    .page('https://my-page');
test('Should log in via <Service Name>', async t => {
    await t
        .click('#login-popup')
        .typeText('#username', 'username')
        .typeText('#pass', 'mySecret')
        .click('#submit')
        .expect(Selector('#user').textContent).eql('Hello %Username%!'); //Make sure we are logged in
});

You do not need to write extra code to create basic multi-window tests. When you need to arbitrarily open, close and switch between windows, use the TestCafe API.

Switch between windows

The t.switchToPreviousWindow method activates the last active window. If you only have two windows open, this method will cycle between them. This is useful in a variety of scenarios.

Imagine, for example, that you're testing a real estate listings website. You want to make sure that once a new property is added, it appears in the window with all available listings. The following test implements this scenario:

fixture('Property List')
    .page('https://url.com/properties'); // Open the listings page
test('Check the property list', async t => {
    await t
        .openWindow('https://url.com/addnewproperty') // Open the 'add new property' page in a new window
        .click('#make-public') // Publish the listing
        .switchToPreviousWindow() // Go back to the listings page
        .expect(Selector('.property-list li').textContent).eql('New '); // Check if the new listing is displayed
});

Manage multiple windows

Test scenarios with more than two open windows require more precision. The t.switchToWindow method can activate any open browser window if you pass a window descriptor object or a predicate function with the window description.

To obtain a window descriptor, call the t.getCurrentWindow method or save the return value when you open a new window.

const windowDesc = await t.openWindow('https://devexpress.github.io/testcafe/');
await t.switchToWindow(windowDesc);

The predicate function must contain a description of the window's URL or title. The URL object has the same structure as its Node.JS counterpart.

await t
    .openWindow('https://devexpress.github.io/testcafe/')
    .switchToWindow(({url}) => url.pathname === '/testcafe');

Imagine you are debugging a task manager app. To ensure that both the task list and the notification feed are updated in real time, use the t.switchToWindow method.

fixture('Tasks View')
    .page('https://url.com/add-task');  

test('Add a new task', async t => {  
    await t
        .openWindow('https://url.com/tasks') // Open a new window with the task list
        .openWindow('https://url.com/feed') // Open the notification feed
        .switchToWindow(({url}) => url.pathname === '/add-task') // Go back to the new task form
        .typeText('#task-description', 'Redesign the landing page by 1 Feb') // Fill in the new task form
        .click('#submit-task') // Submit the task
        .switchToWindow(({url}) => url.pathname === '/tasks') // Switch back to the task list
        .expect(Selector('.task-item').count).eql(1) // Check if the new task is displayed
        .switchToWindow(({url}) => url.pathname === 'feed') // Switch to the notification feed
        .expect(Selector('.message').textContent).eql('Redesign the landing page by 1 Feb'); // Check for the corresponding notification
});

API overview

The updated API includes a number of useful window management methods.

t.openWindow(url) opens a new window and points it to the specified URL.
t.getCurrentWindow() obtains the window descriptor that corresponds to the active window.
t.switchToWindow(windowDescriptor) activates the window that corresponds to the window descriptor.
t.switchToWindow(predicate) uses the predicate function to find a matching window, and activates it. The predicate can include the window's title and URL.
t.switchToParentWindow() activates the parent of the active window.
t.switchToPreviousWindow() activates the last active window.
t.closeWindow() closes the active window.
t.closeWindow(windowDescriptor) closes the window that corresponds to the window descriptor.

Try the new API and let us know what you think

To try the functionality described in this article:

Install TestCafe version 1.9.0 or later;
Create a test scenario that incorporates more than one browser window;
Include the window management methods from this article in the test.

The TestCafe team is proud to create API that realistically model user behavior. If you find that the new window management capabilities can be modified to better serve your needs, please let us know. Submit your feedback and bug reports to our GitHub repository.

Refer to the TestCafe help topics for additional information about these methods.

Upcoming Webinar: The Fastest Way to Your Next Test – TestCafe + TestCafe Studio

TestCafe — Tue, 25 Aug 2020 09:26:00 +0000

In this webinar, our Technical Evangelist Paul Usher will demonstrate the difference between TestCafe and TestCafe Studio. Paul will show you how to write code-based tests more quickly and he'll explore the power of TestCafe Studio's integrated visual test recorder (for those who don't want to manually write test code).

Paul's topics will include:

How to create and debug tests in TestCafe Studio without writing any code.
How to create stable selectors for your tests.
The advantages of TestCafe's Smart Assertion mechanism.
How to leverage both TestCafe and TestCafe Studio within your organization.

End-to-End web testing made easy

TestCafe Open Source and TestCafe Studio are driver free and do not require you to manage complex plug-ins. If you're ready to test your web apps and want to deliver more reliable solutions to your end-users, be sure to click the link below and register your interest in this live webinar.

Webinar Date: Tuesday, September 8, 2020

Start Time: 10AM Pacific Time

Register

TestCafe Webinar - Your Questions Answered

TestCafe — Mon, 24 Aug 2020 10:11:42 +0000

We recently hosted a webinar on TestCafe and found the response overwhelming. During the presentation a large number of questions were raised, and as promised, the team have been through and answered them all.

Watch the Webinar

Webinar Resources

Does the book (End-to-End Testing with TestCafe by Dennis Martinez) go over the examples from the Webinar?

This webinar is based on the book, so you will certainly find details about the examples we showed.

Is the webinar available in other languages?

We currently hold online events in English only.

In what language the application you show is written?

It is JavaScript, the ES6 standard.

TestCafe Benefits

What are the advantages of TestCafe over Selenium/Cypress/Nightwatch.js/Robot Framework/WebdriverIO?

TestCafe does not require browser plugins, SDKs or other tools and libraries - just Node.js and browsers.
You don't need to configure testing environment. Install TestCafe from npm and you are ready to test.
TestCafe is truly cross-browser and cross-platform. It can run tests on Windows, macOS, Linux, iOS, and Android. Supported browsers include Chrome, Firefox, Safari, Edge, and Internet Explorer.
TestCafe features a built-in wait mechanism. You don't need to insert manual waiting in your tests. TestCafe waits for all kinds of loadings and animations automatically, so your tests are stable even when they slow down for whatever reason.
TestCafe allows you to test even the most complex scenarios like those with cross-domain navigation, file upload, sophisticated iframe use cases, secure services (payment, geolocation) or multiple windows (in beta).

License

Do I need to purchase a separate license for TestCafe? Is it included in any of the DevExpress subscriptions?

TestCafe is open-source and free. It is licensed under the MIT license.

Desktop Application Testing

Does TestCafe only work in a browser? Can I use it to test desktop applications, like WinForms?

TestCafe is designed for Web applications only. Thankfully, if you use Electron.js for your desktop application, you can easily test it with TestCafe.

Parallelization

Can I start 100 TestCafe tests to simulate high load?

You can start 100 parallel browser instances if your computer has sufficient resources to run such a number of browsers. But we recommend using TestCafe for end-to-end rather than load testing.

Can I run the tests in parallel in multiple browser windows?

Yes, TestCafe allows you to execute tests concurrently. In concurrent mode, TestCafe invokes multiple instances of each browser. These instances constitute the pool of browsers against which tests run concurrently, i.e. each test runs in the first available instance. You can find information how to run tests in parallel in this topic.

How well does TestCafe scale when it comes to parallelization? What is the ideal number of tests that can be run in parallel?

You can run any number of tests in concurrent mode. Regarding the number of browser instances that can be used to run these test, it's hard to name an optimal number. It all depends on the resources available on your machine and the tests themselves. It is best to choose this number on a case-by-case basis.

Can I have multiple TestCafe instances running tests?

Yes, you can use several TestCafe instances. There are no special restrictions here. But in this case you will have to make sure that the instances do not conflict for resources (open ports, access to files, etc.).

Is there a way to start more parallel tests in memory without running browsers?

Concurrent mode implies launching the specified number of browser instances. All instances will be launched even if the number of tests is not enough to run at least one on each of them. Therefore, now there is no way to use this mode without launching browsers.

Integrations

Does TestCafe integrate with BrowserStack?

Yes, you can run tests in Cloud Testing Services (BrowserStack, Sauce Labs) via the corresponding plugins. See more in the documentation.

Can I run TestCafe on Azure DevOps?

You can find a topic about Azure DevOps integration in the TestCafe documentation.

Can I run TestCafe on Jenkins where there are no local browsers installed? Should I use headless mode on Jenkins?

If the machine that runs Jenkins does not have local browsers installed, you can use portable browsers or cloud testing services (BrowserStack, SauceLabs).

Headless mode is supported for both locally installed and portable browsers. We recommend that you use headless mode to decrease testing time. You can run browsers in headless mode even on machines without graphical system.

This tutorial describes how to integrate TestCafe with Jenkins.

You may also wish to check the TestCafe plugin for Jenkins that attaches screenshots and videos to the Jenkins test results page.

Is TestCafe compatible with Angular applications?

Yes, TestCafe has full support for Angular applications. You can use the Angular selector plugin that adds selector extensions to make it easier to test Angular applications with TestCafe. These extensions allow you to create a Selector to find elements on the page in a way that is native to Angular applications.

Does TestCafe support Vue?

Yes, TestCafe allows you test Vue applications the same way as any other apps. You can use the Vue selector plugin that contains selector extensions to make it easier to test Vue components with TestCafe. These extensions allow you to test Vue component state and result markup alongside.

Can TestCafe pick elements by their React component name?

You can use the TestCafe selector plugin for React components. This plugin adds selector extensions that allow you to select page elements in a way that is native to React. For instance, you can use component names to identify page elements.

Is TestCafe compatible with Knockout?

Yes, TestCafe can test applications based on Knockout. There are no known issues.

Does TestCafe support Bamboo?

You can use "Script Task" in Bamboo setting to write a wrapper that invokes automated tests. Alternatively, you can use the TestCafe Docker image.

Blazor

Can I use TestCafe to test a Blazor application?

TestCafe supports testing for Blazor apps. You can test your Blazor app the same way as any other application. There are no known issues.

Action Chaining

Are there limitations as to what kinds of actions can be chained? Is it the best practice to chain as many actions as I can?

Almost all TestCafe test controller's methods can be chained, except those that return a value and hence break the chain:

You can chain these methods in any desirable manner but we recommend adding blank lines between logical parts of the action chain to improve tests readability, e.g.

await t
    .click('#id1') //some settings
    .click('#id1')
    .click('#id1')
    .expect(Selector('#result').textContent).eql('expected text');//assertion

Can I extend the TestController object with my own chainable methods? Or only helpers and pageobjects are the way to go?

TestController cannot be extended with custom methods at the moment. We may consider adding this feature, but it isn't in our current roadmap. As you correctly suggested, you can use helpers and page objects to implement custom methods.

TestCafe Studio

Can TestCafe record my actions (some tests need many entries and writing them is very time-consuming)?

You can use TestCafe Studio that has a built-in test recorder. It allows you to record your actions without coding them.

Is there a user interface for building tests, or is it all JS-based?

You can use TestCafe Studio which is a UI tool to maintain your tests.

Is TestCafe Studio a desktop application or is it available as a Web service?

TestCafe Studio is a cross-platform desktop application (powered by Electron.js). You can download its installer for Windows, macOS or Linux and install on your machine. Offline registration is available for computers without internet access. Read more in the Installation topic in the Docs.

Does TestCafe Studio require a license?

Yes, this is a commercial product that requires obtaining a license. There are two options available: TestCafe Studio and TestCafe Studio Pro with DevExpress Technical support.

Can TestCafe Studio generate tests steps in a more user-friendly form than code?

TestCafe Studio has the built-in Test Editor designed for codeless test creation.

Can you change the order of tests on the side panel on the left?

The Explorer Panel allows you to view and manage files in your testing directory. For recorded tests, the Explorer Panel displays fixtures and tests. You can create new tests, copy/paste them and more. But currently you can't change the order of the tests.

Are there more formats of report logs in TestCafe Studio?

In the Report View you can see tests run report in the TestCafe Studio format - Grid View

The following built-in reports formats are also available:

JSON
xUnit
List
Minimal
Spec

You can install TestCafe reporter plugins to add more formats: Get More Report Formats.

Is there any guide with comprehensive instructions and examples for TestCafe Studio?

See the guides, videos, and examples in the TestCafe Studio documentation.

What is the element locator strategy?

TestCafe Studio use some characteristic of the DOM element in order to identify it on the page, such as id, class, text etc. You can set the usage priority of these characteristics, switch them off (e.g. for auto generated id) or create new ones based on custom attributes. See the details in the documentation.

Is TestCafe Studio a part of the Universal Subscription?

Yes, TestCafe Studio Pro version is.

Does TestCafe Studio support console.log output? I am currently running tests from the command line so I'd like to see my console.log statements.

You can add the Run TestCafe Script action. In this action, you can use console.log() to print messages directly, or even call t.getBrowserConsoleMessages() to fetch messages from the browser's console.

Can TestCafe Studio generate reports with statistics, graphs, pies, etc., illustrating execution results?

Currently there is no such capability.

In TestCafe Studio, moving a line becomes almost impossible over 100 lines. Is that a memory issue that can be changed via config?

We cannot reproduce this behavior on our side. Please create an inquiry on https://www.devexpress.com/ask and provide more detail about the issue you encounter.

We are using both DevExpress and custom controls in our application. We record tests in TestCafe Studio, but they fail when we run them. Can you tell why this happens?

To resolve the issue, please refer to the Troubleshooting section on our FAQ page. If suggestions described there do not help or apply to your case, create a ticket on https://www.devexpress.com/ask and describe your issue in detail.

Page Model

Do you create the page models manually? Can the page models be dynamically generated?

Page Models should be created manually, there is no built-in way to generate page models automatically. A page model reflects the internal structure of the testing page: what is the container, what is the parent and what is the child, etc., which distinguishing characteristics are best used to identify elements. Any automatically generated model is unlikely to reflect these semantics.

Is it necessary to create selectors with the Selector constructor in a page object class?

No, you can use CSS strings:

class Page {
    constructor(){
        this.submitButton = '#submit';
    }
}

But if you want to extend or chain such a selector with selector's methods, you will need to use the Selector constructor:

class Page {
    constructor () {
        this.button       = Selector('button');
        this.submitButton = this.button.withText('Submit');
        this.cancelButton = this.button.withText('Cancel');
    }
}

Can I add additional methods to page object in TestCafe?

Yes, you can add methods to a page model. Usually these methods are the most frequently repeated actions specific to this page. See an example in the TestCafe docs.

Does TestCafe support the PageObjects library?

If you mean this PageObjects library, then TestCafe does not support it because this library is designed for Selenium.

Database Access

Can test results be sent to a database?

There's no built-in functionality for this, but you can certainly make a plugin that will do this.

Can TestCafe connect to databases to verify that the correct data was written?

TestCafe doesn't have built-in functionality for interaction with databases. But you can use any npm module for it.

Browsers

Does TestCafe support Microsoft Edge? Can TestCafe create mocks and run tests in IE11?

Yes, TestCafe supports both browsers. You can find the list of supported browsers in the TestCafe docs.

Selectors

What type of selectors are available? Is it only HTML element IDs and/or classes?

You can use any pure CSS selectors or TestCafe Selector API that extends capabilities offered by CSS selectors.

Can I target elements based on text that is actually visible to the end-user?

TestCafe Selector's withText method finds elements by text that is visible to the end-user (the element's innerText property). You can also create a selector that uses custom logic to find an element. Pass a function to the selector's constructor.

Is it possible to set the waiting time for an element? What if I mistyped the selector's "name" or the page is loading really slow? How long does TestCafe wait by default?

You can specify the Selector's timeout option. During this timeout, the selector waits for the element. The default timeout duration is 10000 ms.

I use auto-generated names for DevExpress control's elements in my application. Can TestCafe interact with them?

Yes, you can identify these elements by their static parts. See an example in our docs.

Can I use XPath to locate elements?

XPath selectors are not built-in with TestCafe, but you can implement a simple helper method to use them. See an example here.

Can I verify a certain element does not exist or will that cause a failure?

If an element used in some action doesn't exist, a test fails. A test run report then indicates that the test failed because a target element doesn't exist on the page. To check if the element exists before TestCafe attempts to interact with it, use the following approach

if(await Selector('#element').exists)
    await t.click(Selector('#element'));

For instance, you would use this verification to close ad popups (should they appear).

Debug

Are there any approaches that help debug tests? How can I debug TestCafe and the Hammerhead proxy?

Check out this blog post on how to debug your application and tests.

You can also find the instruction about how to debug in other IDEs in our guide.

To debug TestCafe and Hammerhead, enable the development mode. Use the --dev flag when you run tests:

testcafe chrome my-tests --dev

What are the available debug options?

The following capabilities help you determine the cause of errors and failures:

Take screenshots automatically when tests fail,
Capture videos of test runs,
Set the speed of test execution
Enter debug mode automatically when the first test fails,
Execute tests step by step in debug mode.

Can TestCafe capture a screenshot when a test fails?

Absolutely, enable the takeOnFails option to do this.

Authentication

Can I log in once and execute multiple tests under the same account rather than logging in in every test?

Yes, you can use roles to perform actions that log you in only once. In the subsequent tests, you just activate the role you defined earlier to log in without repeating the authentication process each time. See User Roles in the documentation for details.

Do roles work with auth0/openid which has JWT in headers?

Roles can process the JWT if it is sent in the cookies or stored in the local or session storage.

Would we run into any issues if we're using Microsoft authentication tokens for permissions?

You should not have any issue. Please try this and see how it works. If you face any issues, let us know, so we can research it.

How does TestCafe log out the user to switch to a different role? Does it just re-open the page?

When you switch to a role that was active previously in the current session, TestCafe restores cookies and browser's local and session storage where authentication data is located. If the new role wasn't used yet, TestCafe just resets the cookies and storage and then performs authentication actions.

Can you create Roles with API calls instead of functional steps?
If your question is about REST API, then yes, you can call REST API to authenticate a user in a role. Just note that you need to use a client function to call REST API from the browser. You may want to inject a library that can call API into the tested page.

What are the options for smart card logins?
TestCafe only supports web-based authentication. That is, your website should be able to authenticate you by your actions in the web browser, without relying on additional hardware. The only way is to deploy a testing instance of your web app without smart card authentication.

Multiple Apps, Windows, and Tabs

Can TestCafe test integration across multiple applications?

Yes, you can test any number of web applications.

Can you test multiple browser windows at the same time? For example, I may need to access application A, then check application B and come back to A.

Yes, you can do this. We released multiple window support in beta in v1.9.0. Please find more information in the Multiple Browser Windows topic and don't hesitate to leave your feedback. Your opinion helps us make sure you get the best experience when this feature is released.

Does TestCafe support multiple tabs?

You can open and switch between multiple browser windows with this new beta feature. TestCafe will open a new popup window instead of a tab when necessary. This is because tabs can be throttled which interrupts TestCafe scripts.

iframes

Does TestCafe support iframes?

Yes, TestCafe supports iframes and provides two simple methods to interact with their content: switchToIframe to switch the test's browsing context to iframe's context and switchToMainWindow to switch it back to the main window.

Reports

How does TestCafe output test run reports? Where can I find test results?

TestCafe shows the test run report in the console by default. You can redirect output to a file or use dedicated reporters to post reports to a CI system or elsewhere.

What happens when a test fails? What logs are produced?

If a test fails, you will see a comprehensive report in the console. For instance:

Multiple report formats are supported, including the following:

spec
list
minimal
xUnit
JSON

Read more in the TestCafe documentation.

In what format does TestCafe generate the reports? Is it XML or HTML?

The format depends on the reporter. It can be XML, JSON, HTML, and many others. You can explore the available reporter packages on npm.

Test Organization

Can I use logic operators to have different conditional "paths" during a test?

Yes, TestCafe can do conditional testing. You can use standard JavaScript if-clauses to make this happen.

Is it possible to share fixtures between test files?

TestCafe treats fixtures declared in different files separately. If you can share your use case when merging fixtures is useful, please open an issue on GitHub.

How TestCafe Works

Did I miss what the async keyword means in code? Is that to say TestCafe commands can run as other tests are running?

async is a JavaScript keyword that indicates that the function is asynchronous. Technically, an asynchronous function means that this function returns a Promise. TestCafe actions are asynchronous because so are the page events. Page loading, animations, HTTP requests take different amounts of time. The test function is asynchronous as well because it uses asynchronous TestCafe actions.

await means that the next command should not execute until the previous ("awaited") command is completed. This is required to execute test actions in the right sequence (and, in the same manner, TestCafe "awaits" tests in a fixture to run them one by one). Note that, by contrast, tests in different browsers can run in parallel, just like in concurrency mode.

Please provide technical insight about how TestCafe avoids installing special plug-ins in order to interact with the browser. How does it actually interact with browsers?

TestCafe proxies the tested webpage and injects automation scripts right inside the proxied copy. This way, browsers do all the work when they execute the injected scripts. You can find more details in the How It Works article.

Is it possible to access runtimeJS objects from application client libraries loaded in the browser?
TestCafe is designed to conceal itself from application's JavaScript code. This is necessary to ensure that pages proxied by TestCafe run exactly in the same way as they do on the website. For this reason, application code cannot access TestCafe scripts or test code entities.

Waiting and Asynchrony

What if a portion of a page takes a while to render (for instance, the page uses AJAX). Can TestCafe take it into account?

Yes, TestCafe tests wait automatically until the page is rendered completely. You can also insert manual waiting, but it shouldn't be necessary in almost all cases.

I set up my page in a jQuery callback: $(document).ready(function(){}). Will there be JS races with TestCafe?

TestCafe will wait until the page loads completely and run tests only after that. TestCafe wait mechanisms are designed to prevent races with JS code.

It is great that TestCafe handles waiting internally. Would there be a need to override default timeout functionality though? How should a user go about it?

You may need to override the default timeout if your page requires extra long waiting. All TestCafe auto-wait timeouts are customizable. You can find more information in the Built-in Wait Mechanisms topic.

Other Features

Does TestCafe support all ES6 or later features? Do I need plugins to support arrow functions or the async/await syntax?

TestCafe supports all modern JavaScript and TypeScript features. No plugins needed, everything works out of the box.

Can TestCafe work with the Service Worker?

We are currently in the progress of implementing full support for Service Workers. Stay tuned for our updates to be the first to know when we are ready.

Does TestCafe work with Python?

You can write tests in JavaScript/TypeScript/CoffeeScript only.

Does TestCafe support upload ( excel ) data functionality?

TestCafe can upload files, including Excel, to websites if it is a part of the test scenario. However, if you are asking about API to access Excel data sheets from test code, TestCafe does not provide one out of the box. You can search for npm modules that implement the functionality you need and use them from TestCafe tests.

What about functional test coverage? Does TestCafe support it?

Test coverage estimation is not available at the moment.

Miscellaneous

If TestCafe runs in headless mode, is it still possible to grab screenshots on failures?

Yes, TestCafe can take screenshots even in headless mode.

Does this require a test framework like jest/mocha/jasmine?

No additional frameworks are required for TestCafe.

How can I use TestCafe to test different resolutions?

You can use the t.resizeWindow action to change the browser window size and test adaptive UIs. TestCafe also supports Chromium device emulation to test on emulated mobile devices of various sizes.

Can we use Test Cafe to compare two documents? Like documents with logos, disclaimers and other information?

If you mean visual comparison, TestCafe does not provide this functionality out of the box. However, you can find multiple open-source packages that try to solve this task, like testcafe-blink-diff. Note that they are maintained by the community and are not related to the TestCafe team.

If your goal is to compare HTML, you can pass innerHTML to a TestCafe assertion as shown in the example in this topic.

How do you change the base URL to test in different environments?

You can import the base URL from a configuration file or an environment variable, and then use it to build start page URLs in test code. See this example on the FAQ page.

Can I run TestCafe on a Linux computer without the UI?

Yes, you can run your tests on Linux and use headless browsers to run your tests without the UI.

Could you please share ideas about how to do localization testing? Does TestCafe support that?

TestCafe does not have built-in functionality for localization testing. As a suggestion, you can fetch the localization strings from a resource file or database with Node.js means or third-party npm modules. Then, you can create data-driven tests that check if the displayed strings are localized (present in the localization resources).

When do you plan to release version 1.9.0?

1.9.0 was released on August 6th. If you are asking about the multi-window support final release, we do not have an exact date yet.

How does TestCafe handle exceptions?

TestCafe considers the test failed if its code throws an exception. The report will indicate the line of code and call stack where this exception originated. Exceptions in JavaScript code on the tested page also make tests fail. However, this behavior is customizable. You can use the --skip-js-errors flag to ignore page errors. Similarly, the --skip-uncaught-errors flag allows you to continue tests when test code throws an exception.

Can we measure client-side performance using TestCafe (both HTTP and script execution time)?

TestCafe is not intended for this task. In fact, TestCafe impacts webpage performance when it runs tests. We are constantly optimizing our code to minimize this slowdown, but we do not aim to precisely match the original performance.

Are tests executed in sequence within a fixture?

Yes, tests are executed as they are declared in code.

Can I add my custom element library to the context object?

You can add anything to the feature and test context objects. However, keep in mind that test code does not run in the browser, so adding browser libraries to the context objects might not help. If this is the case, you may be more interested in the custom script injection functionality designed for user's client libraries.

Can TestCafe test HTML5 features like fullscreen, camera, audio, or video interaction?

Yes, you can test these HTML 5 features with TestCafe.

Note that you might need to set the --autoplay-policy=no-user-gesture-required browser flag as explained in this article to test the fullscreen mode and audio/video. The camera can be mocked using ClientFunctions.

Does TestCafe support multiple domains in a test?

Yes, navigation and requests between different domains work out of the box. You do not need to care if you visit pages on one or several domains.

What happens if TestCafe loses network connection with a remote device that runs tests?

Tests cannot run without a network connection because TestCafe must be able to send commands to the remote device. If the connection is interrupted, TestCafe will throw an error saying that a browser connection is lost.

Is it possible to run all JS in one command? Could you please specify this command?

If I understand you correctly, yes, you can run all your JS tests using a single console command.
Here is the CLI reference where you can see the testcafe command syntax.

I noticed non-standard code indentation. Is a respective formatter for VS Code available?

We didn't release a specific formatter because you can use any indentation you prefer with TestCafe.

Can I use TestCafe to test the development and production instances of my app? If so, how can I configure different URLs for my instances?

Yes, you can import the base URL from a configuration file or an environment variable, and then use it to build start page URLs in test code. See this example on the FAQ page.

Does TestCafe have support for file download? Can you set the default file download location using TestCafe API?

Yes, if test actions on the page trigger file download, the browser will download it as usually. Then you can access the file from test code to perform verification. Currently, TestCafe API does not allow you to set the default download location.

Introducing the TestCafe Jenkins Plugin

TestCafe — Wed, 05 Aug 2020 10:51:35 +0000

TestCafe can capture videos and screenshots so you can debug your tests and examine page UI. You can record all tests or only those that failed, generate unique video files for each test or record all of them in one take, and single out specific interactions. (For a full list of available options, see the following help topic: Screenshots and Videos).

Many TestCafe users take advantage of Jenkins — a CI/CD solution — to automate their tests. Until recently, they had to manually match the screenshots and videos taken by TestCafe to the individual test reports. The newly implemented TestCafe Jenkins plugin simplifies this process. Links to screenshots and videos taken during the test now automatically appear on the Jenkins test results page. There is no need to keep the testing server running — all required files are stored inside the Jenkins build folder.

With our plugin enabled, the test results page should appear as follows:

Test case: the disappearing button

Let's assume that the 'Add to Cart' button disappears on the mobile version of your website.

TestCafe supports the following mobile device emulation methods:

Resize the browser window during the test
Run the test through a third-party cloud testing service
Use Chromium's built-in emulation mode

The third option offers greater speed and stability. We describe it going forward and recommend that you use it if possible.

First, let's write a simple test that checks if the 'Add to Cart' button exists:

test('`Add To Cart` button should exist', async t => {
    await t
        .expect(Selector('button').withText('Add To Cart').visible).ok();
});

Now let's configure Jenkins to launch this test every time we build our project.

Basic setup

Step 1. Install prerequisites

TestCafe requires a working Node.js installation to run. If your testing environment does not contain node.js, the Node.js Jenkins plugin can install it for you. Also ensure that Chromium is installed: TestCafe cannot function without a browser.

Step 2. Install the TestCafe Jenkins plugin

To install the TestCafe Jenkins plugin, click the "Manage Plugins" link on the Manage Jenkins page, select the "Available" tab, and enter "testcafe" into the search field. Check the box next to our plugin and click the 'Install without restart' button below it.

Step 3. Install the required Node packages

Add the following build step to your Jenkins project:

npm i testcafe testcafe-reporter-jenkins

This command installs the main testcafe library, as well as the testcafe-reporter-jenkins package. The latter is needed to generate Jenkins-friendly JUnit format reports.

Step 4. Configure the tests

You can configure your tests via:

the command line interface
the JavaScript/TypeScript API
the .testcaferc.json configuration file

The last option provides an easy, declarative way to define your test settings. You can store this file in the same folder as your tests. Below is the configuration used in this tutorial:

{
    "browsers": [
        "chromium:headless:emulation:device=iPhone 4",
        "chromium:headless:emulation:device=iPad Mini",
        "chromium:headless"
    ],
    "src": "test/e2e/**/*",
    "screenshots": {
        "takeOnFails": true,
        "fullPage": true
    },
    "videoPath": "videos",
    "reporter": {
        "name": "jenkins",
        "output": "report.xml"
    },
    "appCommand": "node server.js"
}

The Browser array contains the list of browsers TestCafe will use to run our test. The TestCafe browser syntax allows us to specify the mobile devices we want Chromium to emulate.

The name property of the reporter object is set as jenkins, which ensures that the reports generated by TestCafe can be properly parsed by the server.

If you decide not to use a config file, be sure to manually set the report format when you launch the tests as follows:

testcafe chrome test.js -r jenkins:report.xml

Otherwise, simply run the following command:

npx testcafe

Step 5. Configure the reporter

Turn on the JUnit reporter plugin: add the "Publish JUnit test result report" post-build action. To display screenshots and videos alongside your test results, select the "Include links to TestCafe artifacts" option from the "Additional test report features" drop-down menu.

Important: Remember to check the "Retain long standard output/error" box. Otherwise, the reporter plugin will automatically truncate the URLs of your videos and screenshots.

Step 6. Run the build

Save the changes to your project and click the "Build now" button on the project page:

Review the results

After the tests are completed, click the following link on the build page to view the results:

You will see links to screenshots and videos taken during the test.

These artifacts instantly indicate if the Cart button is present on the page.

Jenkins Pipeline integration

If you do not wish to use the Jenkins GUI to launch these tests, you can always use the Jenkins Pipeline. It lets you declaratively configure your CI/CD setup via a Jenkinsfile - a text file checked into the project's source control repository. The following is a sample Jenkinsfile to be used with TestCafe:

pipeline {
    agent any
    stages {
        stage('Run E2E tests') {
            steps {
                sh 'npm install'
                sh 'npm run test-e2e'
            }
            post {
                always {
                          junit keepLongStdio: true,
                          testDataPublishers: [[$class: 'TestCafePublisher']],
                          testResults: '*.xml'
                }
            }
        }
    }
}

Note: The test-e2e npm script in this example launches TestCafe. As mentioned earlier in this topic, be sure to manually set the reporter option. For more information on the TestCafe CLI, see the following help topic: Command Line Interface.

If you enjoyed this guide and would like more tips on debugging, please see the following blog articles: How to Debug Tests in TestCafe: Quick Guide and How to speed up debugging in TestCafe: Tips and Tricks.

Upcoming Webinar: 10 Ways to Simplify Your UI Testing

TestCafe — Tue, 21 Jul 2020 12:31:52 +0000

In this free webinar, our Technical Evangelist Paul Usher will show you how to easily incorporate UI testing in your development workflow. Paul will demonstrate how TestCafe can fit in any organization and how its features can be leveraged to deliver high quality web apps that always meet end-user expectations.

In this webinar, you’ll learn:

How to setup TestCafe and start testing in under 10 minutes.
How to create customizable tests that can easily evolve as business requirements change.
The purpose of Custom Selectors and why they make web testing so much easier.
The purpose of the Page Model approach and why it helps increase productivity.
And you'll learn how to master use of Smart Assertions.

Experience the TestCafe Difference

TestCafe Open Source and TestCafe Studio are driver free and do not require you to manage complex plug-ins. If you’re ready to test your web apps and want to deliver more reliable solutions to your end-users, be sure to click the link below and register your interest in this live webinar.

Webinar Date: Wednesday, July 29, 2020

Start Time: 10AM Pacific Time

Register

How to Speed up Debugging in TestCafe: Tips and Tricks

TestCafe — Mon, 13 Jul 2020 12:48:15 +0000

In the previous post, we talked about the approaches you can follow to debug TestCafe tests. In this post, we will focus on best practices that help you save time while debugging. We will also mention a few things to keep in mind in order to write easy-to-debug tests.

Tips on how to debug tests

If you know which test throws an error, run this test separately.
If you only need to check one test, do not wait until all tests pass. TestCafe can run any single test separately. Add the --test shell argument or use the test.only method in test code. (Be sure to remove .only before you commit changes.)

testcafe chrome ./tests --test "My test name"

test('My first test', async t => {
    /* ... */
}).only;

Use live mode when you edit tests.
If you find yourself running a test repeatedly while you write or debug it, try live mode. In this mode, TestCafe watches the test file and restarts the test automatically after you save changes. To enable live mode, run the test with the --live flag.

testcafe chrome ./tests --live

To identify an issue visually, decrease the test speed.

By default, tests are executed at full speed – with minimum delays between actions and assertions. This makes it difficult to identify problems visually during a test run. To slow the test down, use the -speed flag. The argument value is between 1 (fastest) and 0.01 (slowest).

testcafe chrome ./tests --speed 0.1

Use TestCafe Studio to determine why a selector does not work.
TestCafe may report that a selector does not return any element, although you have written it correctly. The reason may be an invisible DOM element that matches the same selector expression. To get a better understanding of the issue, record a similar test in TestCafe Studio and compare your selector to the selector that is generated.

If a test fails on CI, use screenshots and videos to clarify the reason.
You do not have to run a test locally to determine the reason why a test fails on CI. TestCafe can take a screenshot when the test fails and record a video of the test run. To enable screenshots and video recording, use the --screenshots-on-fail and --video CLI options. See the following help topic for details: Screenshots and Videos.

Tips on how to write tests that are easy to debug

Tests should be easy to run locally.
If it is difficult to run a test locally, developers are likely to delay test fixes, so that they turn 'green' only shortly before the release. These delays may complicate the debug process.

Developers can easily reproduce test scenarios locally and maintain 'green' tests if the following is true:

Tests are isolated from each other.
The workstation environment is similar to the test environment.

Ideally, a developer must be able to run any test locally with a single command, which requires the following:

The product test build is easy to run in a Docker container (or by another approach that emulates the test environment).
Test data is seeded in test code (for example, within the before and after hooks).

Tests should be independent.
If subsequent tests use data produced by previous tests, you can expect the following two issues:

When a previous test fails, all dependent tests will also fail.
You cannot debug a test until all previous tests are complete.

To simplify debugging in the future, be sure to avoid dependencies between tests.

For more best practice tips, see the following help topic: How to Scroll Web Pages Using TestCafe.

TestCafe Studio v1.3.0 - A New Way to Work with Selectors

TestCafe — Mon, 22 Jun 2020 10:29:09 +0000

TestCafe's Visual Test Recorder automatically creates selectors for all elements used in a test. This complicated algorithm balances many different variables, and we just recently added a few options so that you can customize it to meet your specific needs.

This article first covers what we consider to be a reliable selector. We then mention the pros and cons of various selector strategies, and custom attributes in particular. Finally, we'll show you how to customize TestCafe options so that it builds selectors that are optimized for your website.

How TestCafe Studio Generates Selectors

To record an automated test in TestCafe Studio, specify the tested page’s URL and start the recorder. TestCafe Studio launches the selected browser, opens the web page, and records your actions. Once the test scenario is ready, you can run it in any local browser, a mobile device, or as part of your CI pipeline.

TestCafe Studio not only records your actions (element clicks, text input), but it also creates a selector for every affected page element. The selector generation algorithm was developed to adhere to the following requirements:

Selectors shouldn’t be too specific. Otherwise, you might have to rewrite them after each page modification. For instance, Selector(‘body’).find(‘div’).nth(5).find(‘p’).nth(3) must be revised each time the number of elements on the page changes.
Selectors shouldn't be too generic. Otherwise, they may return different elements after each markup change. For example, Selector(‘div > button’) can match multiple elements at once.
Selectors shouldn’t rely on element parameters that are likely to change. For instance, Selector('[style*="background-color: red"]') uses a CSS style that changes frequently.
Selectors should remain readable. Selectors should be easy to understand (by you or another developer) for as long as the test is maintained. For instance, it may be difficult to understand which element corresponds to the following selector: Selector(‘div’).find(‘pre’).nextSibling(-1). If you use Selector(‘#topContainer’).find(‘.child’).withText(‘Add item’) instead, the selector is much easier to read.
Selectors should reflect the user’s point of view. Since TestCafe supports end-to-end testing, it’s a good idea to build selectors that identify elements as an end-user would. For instance, Selector(‘form’).find(‘[name=”btn-foo-123”]’) might be stable, but it is written from the programmer’s perspective rather than from the user’s point of view.

In addition to these general requirements for selectors, page organization is also vital in the testing process. Some pages may have stable selectors that rely on the ‘id’ attribute, while other pages use auto-generated ‘id’ attributes - so it does not make sense to use ‘id’ in a selector. This also applies to other attributes such as ‘class’. Text-based selectors can also be ineffective, depending on changes you introduce to the page over time.

Why Custom Attributes are Required

Modern web component frameworks, like React, complicate DOM element identification. This is why many developers prefer to assign custom attributes to page elements. Custom attributes enable tests to survive changes in HTML structure, script behavior, and styles.

Let’s consider the following Populate button on our example page:

<input type="button" id="populate" data-testid="populate-button" value="Populate">

You can build the following selectors for this button:

HTML-based

Selector: Selector('input').nth(1)
Pros and Cons: Too generic.
Summary: Might match the wrong elements because it is vague.

Selector: Selector('body').find('form').find('div').find('fieldset').find('input').nth(1)
Pros and Cons: Too strict.
Summary: May break due to minor changes in HTML structure.

ID-based

Selector: Selector('#populate')
Pros and Cons: Relies on ID only, which could be introduced for styling or event handling, or may be auto-generated.
Summary: Better options are available.

Selector: Selector('#main-form').find('input').nth(1)
Pros and Cons: Similar to the above selector, but relies on the parent ID.
Summary: Appears to be more specific than the above selector, but it is not.

Class-based

Selector: Selector('.column.col-1').find('input').nth(1)
Pros and Cons: The element has no class, so this selector uses the parent class. Classes can change because they are usually assigned for styling purposes.
Summary: Sensitive to CSS modifications; difficult to maintain.

Text-based (text matches the attribute value in this case)

Selector: Selector('input[value="Populate"]')
Pros and Cons: Mirrors how users perceive the page, so it may be stable. Detects actual issues caused by accidental text changes.
Summary: Satisfactory, but fails even when text changes are intended.

Custom attribute-based

Selector: Selector('[data-testid="populate-button"]')
Pros and Cons: The most stable selector, since it is not affected by changes.
Summary: The preferred selector type for most scenarios.

In summary, custom attributes have the following benefits:

Tests do not break based on markup or code changes. A test will not fail when you refactor HTML or implement new behavior in JavaScript.
Tests do not depend on page styles. You can switch themes as many times as you wish - tests will not fail because of a certain style or theme.
Text content modifications do not affect tests. Tests are not affected by changes to content, including localization changes. Note, however, that if an element’s text is essential to a test scenario, you should verify the text content or use a text-based selector: Selector(‘div’).withText(‘Add to Cart’).
Use a custom attribute to indicate that an element is used in tests.

A New Way to Customize Selectors

When you record a test, TestCafe automatically generates a selector. TestCafe also lets you choose an alternative selector from a drop-down list or create one manually. The latest TestCafe update (v1.3.0) ships with an adjustable selector generation algorithm. You can now configure the selector generator on the fly, based on the tested page:

Prioritize selector types. Use this option if you often need to choose a different selector from the suggestion list. The default selector is generated based on your preferences.
Disable unwanted selector types. For instance, your website can generate id, class, or other element attributes automatically, so that they change each time the page reloads. However, you cannot use these values in selectors because the test cannot be replayed. In this instance, you may want to exclude these attributes from selector generation rules. You can now simply disable the corresponding selector type - and switch back to it at any time - without restarting the recorder.
![Disable unwanted selector types](
Add selector types based on custom attributes. These selector types are preferred because if you configure them, you won’t have to edit selectors due to changes. The only requirement is that you maintain unique attribute values during development.

Recap

TestCafe Studio generates selectors for every page element that takes part in a test. Good selectors must comply with strict guidelines to be stable and reliable. TestCafe algorithms use best practices to generate selectors that meet these criteria. You can also optimize selectors based on your understanding of how the website works. Our latest update (v1.3.0) allows you to instruct TestCafe Studio to use custom attributes in selectors or ignore dynamic IDs. The result is more stable selectors that are easy to read and maintain.

You can read more about the new adjustable selector generation algorithm in our documentation. As always, we are happy to answer your questions in the comments. Happy testing!

How to Debug Tests in TestCafe: Quick Guide

TestCafe — Tue, 02 Jun 2020 12:45:53 +0000

If you've ever written a TestCafe test, you may be familiar with the following errors:

Cannot obtain information about the node because the specified selector does not match any node in the DOM tree.
The specified selector does not match any element in the DOM tree.
The element that matches the specified selector is not visible.

The first two errors are self-explanatory. The last error, however, can easily be misinterpreted. This error frequently occurs if multiple elements match the same selector. The first matching element may be hidden, while the required element is visible.

What should you do next? How do you update the selector so that it produces expected results? What should you do if the test previously passed, but it now fails?

First, check the application to ensure that the test didn't fail due to a bug. If the app runs as expected, then you can begin debugging the test.

Client-side Debugging

TestCafe uses CSS selectors to identify page elements. To see if a selector matches an element, pause test execution in the browser and debug the test.

To stop the test, you can use the --debug-on-fail CLI flag or the t.debug() test action. The difference between these approaches is summarized below:

`t.debug()`	`--debug-on-fail`
Allows you to define the moment when the test stops.	Suspends the test only if it fails.
You only need to modify the test code to use it.	Requires that you edit the test run command (which may take longer than the t.debug() action).

Let's look at a test from the TestCafe Getting Started topic. Assume the Submit button's selector changed and the test failed:

import { Selector } from 'testcafe';

fixture `Getting Started`
    .page `http://devexpress.github.io/testcafe/example`;

test('My first test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button-invalid')

        // Use the assertion to check if the actual header text is equal to the expected one
        .expect(Selector('#article-header').innerText).eql('Thank you, John Smith!');
});

The test now throws the following error when we run it:

Insert t.debug() before the line that fails:

await t
    .typeText('#developer-name', 'John Smith')
    .debug()
    .click('#submit-button-invalid');

The test pauses before the button click. You can now use browser utilities to see if any elements match the selector.

Since document.querySelectorAll returns an empty list, you can see that there is no such element on the page, but the Submit button is visible. Use the Inspect command to see how this button is rendered in markup:

The browser's development tools pane shows that the button's ID is submit-button.

Note. You can use Chrome Dev Tools' shortcuts ($ and $$) to query selectors. However, if your website uses jQuery, the $ shortcut is overridden. TestCafe searches for elements with document.querySelectorAll, which behaves in a slightly different way than jQuery. Please keep this in mind.

To use an element inside an iframe, switch the browsing context with t.switchToIframe before the action.

Hint. Google Chrome lets you pause complex animations that your website might play when elements appear. To stop JavaScript execution, press F8 when the Sources tab is open.

Debugging Client Functions

Another client-side debugging trick in TestCafe allows you to execute code within the browser to obtain client-side data or identify page elements.

const getStoredValue = ClientFunction(key => {
    return window.localStorage.getItem(key);
});

const getCell = Selector((table, row, column) => {
    return table.querySelectorAll('tr')[row].querySelectorAll('td')[column];
});

To debug this code, add the debugger statement. The browser will stop the script and open developer tools.

const getStoredValue = ClientFunction(key => {
    debugger;
    return window.localStorage.getItem(key);
});

const getCell = Selector((table, row, column) => {
    debugger;
    return table.querySelectorAll('tr')[row].querySelectorAll('td')[column];
});

Server-side Debugging

Test flow can be complicated if it includes data preparation, HTTP requests or database access. You can use Node.js to debug this code, along with Visual Studio Code, WebStorm, or Chrome Developer Tools. See the Recipes for details.

Note that it is easier to use client-side debugging to execute chained actions step by step. The server treats the entire chain as a single action, so you must divide it into separate actions to debug.

Stay Tuned for More Information on Test Debugging

As explained in this blog post, TestCafe and modern browsers offer all the tools necessary to debug your application or test. Note, however, that tools do not always locate errors quickly and easily. We've often spent more time debugging than desired. In our next blog post, I'll share a few techniques that may save time when you debug a test.

If you have any questions or want to share your test debugging story – write to us directly at testcafeteam@devexpress.com.

DEV Community: TestCafe

Randomize Your End-to-End Tests: How to Generate Input Data for TestCafe

The Basics (nanoid)

Input validation (faker, generate-password)

Related Topics:

Client Functions and Script Injection for Page Interaction

Use Client Functions to Run Code in the Browser

Under the Hood

Pass Variables to Client Functions

Wait for a Custom Condition

Inject Scripts to Add Code Permanently

Scripts that Persist Throughout Testing

Injected Scripts Run First

Checking Other <head> Scripts for Errors

Conclusion

Related Topics

The Fastest Way to Your Next Test – TestCafe + TestCafe Studio - Q&A's

Watch the Webinar

Questions & Answers

How TestCafe Can Help You to Deliver Faster

Unit Tests are not Enough

TestCafe – End-to-End Testing Made Easy

How to Integrate TestCafe in Your CI/CD

Docker Image

GitHub Actions

TestCafe: Adoption Made Easy

Introducing Multi-window Tests (Beta)

Client-side window events

Switch between windows

Manage multiple windows

API overview

Try the new API and let us know what you think

Upcoming Webinar: The Fastest Way to Your Next Test – TestCafe + TestCafe Studio

End-to-End web testing made easy

TestCafe Webinar - Your Questions Answered

Watch the Webinar

Webinar Resources

TestCafe Benefits

License

Desktop Application Testing

Parallelization

Integrations

Blazor

Action Chaining

TestCafe Studio

Page Model

Database Access

Browsers

Selectors

Debug

Authentication

Multiple Apps, Windows, and Tabs

iframes

Reports

Test Organization

How TestCafe Works

Waiting and Asynchrony

Other Features

Miscellaneous

Introducing the TestCafe Jenkins Plugin

Test case: the disappearing button

Basic setup

Step 1. Install prerequisites

Step 2. Install the TestCafe Jenkins plugin

Step 3. Install the required Node packages

Step 4. Configure the tests

Step 5. Configure the reporter

Step 6. Run the build

Review the results

Jenkins Pipeline integration

Upcoming Webinar: 10 Ways to Simplify Your UI Testing

Experience the TestCafe Difference

How to Speed up Debugging in TestCafe: Tips and Tricks

Tips on how to debug tests

Tips on how to write tests that are easy to debug

TestCafe Studio v1.3.0 - A New Way to Work with Selectors

How TestCafe Studio Generates Selectors

Why Custom Attributes are Required

HTML-based

ID-based