DEV Community: Mateusz Garbaciak

Choosing e2e testing framework. Cypress vs Playwright case

Mateusz Garbaciak — Fri, 27 Sep 2024 14:00:31 +0000

In one of our client projects, we had to decide on an end-to-end (e2e) testing framework. While we were familiar with Cypress, we wanted to explore a newer option, Playwright. We compared how both frameworks addressed the issues that were most important to us, allowing us to establish criteria for making an informed decision. I'd like to summarize some of the key points we discussed and the conclusions we reached.

1. Installation

Both frameworks provide npx scripts that manage package installation and create example tests to help users get a feel for the framework. They also offer Docker images that can be used for CI integration. While all necessary dependencies for Playwright are stored in node_modules (with browsers included in the binary), Cypress relies on what is provided in the system (or Docker image). It's worth mentioning that the Cypress binary is stored globally in the system, which can complicate installation if you're working on a corporate PC with restricted rights to install software. This has another implication: if you wish to cache dependencies between CI jobs, you'll need to set the caching directory and store it next to node_modules.

2. Test Explorer

A comprehensive test explorer is essential for navigating and executing tests seamlessly. Both frameworks offer this functionality, but the developer experience (DX) varies. Cypress presents the real application in its explorer, allowing developers to click through the app and explore the tests once they finish execution. Playwright offers similar functionality, but instead of presenting the real application, it displays app screenshots on a timeline. This might feel like a limitation, but fortunately, there is a VSCode extension that helps overcome it.

3. Syntax and Execution Environment

Both testing frameworks support TypeScript. Cypress handles asynchronous events with a wrapper that looks similar to Promises, while Playwright uses native Promises and supports async/await syntax. It's important to note that Cypress code is executed in the browser, whereas Playwright runs in a 'Node' environment and communicates with the browser through the DevTools protocol. As a result, environment variables can be accessed directly at runtime, which might be useful for passing test-user credentials or other test-related data.

Below you'll find code snippets of what tests look like for both frameworks. Bear in mind that all selectors have been picked up automatically by the locator picker tool (available in both frameworks).

Example test written in Cypress. Cypress manages asynchronous tasks internally, which is why one of the console logs is logged out of order, right at the start of the test. This might be surprising at first. To execute it after some action, you need to wrap it around the then method.

Example Playwright test. The framework uses native async/await syntax, so each statement is executed in order, as one would expect.

4. Parallel Test Execution

Both frameworks support parallel test execution, but it's worth noting that this option is available in Playwright out of the box and for free. With Cypress, however, you need to subscribe to Cypress Cloud, which is only free for small teams.

5. API Testing

In our project, we wanted to cover some functionality with API tests. Our initial idea was to use 'Jest' for this purpose, but it would require us to implement a mechanism to manage auth tokens. With either framework (Playwright or Cypress), this process is simplified. The token can be acquired at the beginning of a test session using application logic (since we have access to the real application) and stored. Later, it can be reused for each API test.

Example API test using Cypress. Cypress uses the Chai assertion library. You might want to extend it with additional modules, allowing for deep checking arrays or checking the subset of properties in the result object.

Example API test using Playwright. For those familiar with Jest, Playwright's syntax looks familiar. It has built-in generic assertions, a class that lets us check if a received object contains a specific value, if a value is close to another, or if a value is of a specific type, and so on.

6. Different Browser Tabs, Changing Origin During Test

Simulating real-world scenarios often involves testing across multiple browser tabs or changing the origin during test execution. While Cypress has a workaround for the latter, it's impossible to execute tests involving changing browser tabs. Playwright, on the other hand, handles such tasks effortlessly due to its method of communicating with the browser. You can open new tabs and switch between them within a test case easily.

7. Mobile Viewports

Both Cypress and Playwright support testing on mobile viewports, offering predefined resolutions for various devices. Playwright's list seems to be longer than Cypress'. In both cases, the resolution can be set either globally (or per project in the case of Playwright) or within a test case. However, it's important to note that this is not about emulating a device, but a device resolution. For thorough testing in real-life scenarios, external tools like Browserstack or Lambdatest are necessary.

8. Visual Comparisons

If you want to ensure that your CSS isn't broken after a change, visual comparison tests are a good idea. Both frameworks support this option, but with different levels of adoption simplicity. In Playwright, it works out of the box; you simply call await expect(page).toHaveScreenshot();. This function generates a new screenshot if one isn't already defined; otherwise, it compares the current snapshot to what's stored in the project, with no additional setup required. Cypress, on the other hand, does not support this natively, so you need to rely on external plugins. I recommend checking the official Cypress docs to see what tools are recommended.

Summary

Taking everything into consideration, we decided to go with Playwright. Its ability to control the browser via the DevTools protocol is a significant advantage, as it allows for testing on different browser tabs without issues. Additionally, its syntax is more explicit, using native Promises that can be consumed with standard await, compared to the promise-like syntax of Cypress. We also appreciated how easy it is to perform visual testing with Playwright.

Miscellaneous

In our case, running tests on CI turned out to be quite slow. After some investigation, we discovered that the issue was a misconfiguration of our Playwright setup specifically, the trace viewer configuration. We had been using the retain-on-failure setting, which records traces for all the specs. This resulted in a large number of files needing to be stored, which was very time-consuming for our GitLab runner. Switching to the 'on-first-retry' option resolved the issue.

RxJS: retry with exponential backoff

Mateusz Garbaciak — Mon, 17 Apr 2023 09:22:52 +0000

Cover photo by Lucas Andrade on Pexels

Http request may occasionally fail. RxJS provides us with a retry operator that helps us make a call again in the case of a failure. Let's see how to implement retries with exponential backoff, i.e. delay that grows with every request.

Back in the days, exponential backoff could be implemented using zip operator to keep track of an index, or saving error index in closure. Currently it's possible to implement it in a simpler way.

Let's first have a look at function signature.

Type of configOrCount is a union of number or RetryConfig.

Simple retry

If we provide a number argument for a retry, it stands for maximum count of retries after error. The example below will retry for next 3 times, without a delay.

source$.pipe(retry(3));

Retry with a delay

If we want to add some delay, we need to provide a retry config object. Below there is an Observable that is going to be retried 3 times, each attempt with a 200ms delay.

source$.pipe(retry({ count: 3, delay: 200 }));

Retry with backoff

Exponential backoff is about increasing the delay with every failure, to reduce the rate of HTTP queries to server. We might want to make the first retry after 200ms, then after 400ms, third failed attempt should happen after 800ms and so on.

A delay can be either a number, as we have just seen that in the previous example, or a function returning an observable. We will concentrate on the latter now.

source$.pipe(
  retry({
    count: 3,
    delay: (_error, retryIndex) => {
      const interval = 200;
      const delay = Math.pow(2, retryIndex - 1) * interval;
      return timer(delay);
    }
  })
);

In configuration object we define maximum number of retries, setting it's count to 3, same as in previous example, what changed is a delay property, which is now a function. It accepts two parameters, error and current index of a retry. We are ignoring first one and using only retryIndex to use it when calculating an exponent.

Our expected backoff values should look like the following:

attempt | value [ms]
1       | 200
2       | 400
3       | 800

To calculate the dalay we are going to use this formula:

delay = 2^(retryIndex-1) * interval

// results:
2^0*200 = 200
2^1*200 = 400
2^2*200 = 800

Once we have calculated the number of milliseconds to wait, last thing is to use timer function from RxJS. It is going to do the actual wait, as it emits a value (0) after a specific amount of time.

Extract it to RxJS operator

To make this piece of code more reusable, we can extract it to another function.

const CONFIG = { count: 3, delay: 200 };

export function backoffRetry<T>({ count, delay } = CONFIG) {
  return (obs$: Observable<T>) => obs$.pipe(
    retry({
      count,
      delay: (_, retryIndex) => {
        const d = Math.pow(2, retryIndex - 1) * delay;
        return timer(d);
      }
    })
  );
}

which could be later used as follows:

source$.pipe(backoffRetry());

Summary

We had a look at how to do retries with RxJS, from simplest form to our desired backoff solution. Using this technique you can decrease excessive load on server on error requests.

References:

Beginners guide to RxJS Marble testing

Mateusz Garbaciak — Mon, 31 Jan 2022 21:52:26 +0000

RxJS Marble testing is a topic about which I heard quite some time ago for the first time. Since then I had a feeling that it's a big thing, that it might simplify testing, but somehow I was always put off. Sheer look at the docs made me feel stupid and decide to postpone the learning till next time. Majority of people whom I talked about it to felt similar.

The aim of this article is to show you that marbles don't bite, and they are not as difficult as they seem at the very first glance.

Problem description

In our code snippets we are going to refer to a light switch/switches and a light bulb. We are going to write code allowing us to switch a bulb on and off. We are going to use i to represent a high state, and o for low state, boolean true and false respectively.

true: switch or bulb is ON
false: switch or bulb is OFF

Lighting the bulb

We have a stream of booleans that imitate a switch and a function that lights the bulb. Pretty straightforward:

export function lightBulb(switch1$: Observable<boolean>): Observable<boolean> {
    return switch1$;
}

Unit test (no marbles yet)

Now lets try to write a unit test that would check if our function works as expected. Being afraid of marbles, we write sth like this:

test('should light the bulb when switch is on', (done) => {
    const switch$ = new Subject<boolean>();

    lightBulb(switch$).subscribe({
        next: (isLightOn) => {
            expect(isLightOn).toEqual(true);
            done();
        },
    });

    switch$.next(true);
});

The problem with this test is that we can't read it from top to bottom. First we subscribe to the observable lightBulb returns,
and a few lines below, we next the switch$ subject. It's subscribe block in the middle of the test where we put the assertion. This does not seem right.

Setup for marble test

Let's take our function and test it using RxJS Marbles. Before we start though, we need to copy a tiny bit of boilerplate code and put it in our test file.

Sidenote: Assertion part looks different from what we see in RxJS docs, Jest being used here as a testing library.

const testScheduler = new TestScheduler((actual, expected) => {
    expect(actual).toEqual(expected);
});

We are instatiating a testScheduler object. It will allow us to run asynchronous code synchronously, using virtual time.

Unit test with marbles

Now we are ready to write our first marble test. We are going to refactor our previous test, so that it was doing the same thing, but with RxJS marbles.

test('should light the bulb when switch is on', () => {
    testScheduler.run(({ hot, expectObservable }) => {
        const switch$ = hot('i', { i: true });

        const result$ = lightBulb(switch$);

        expectObservable(result$).toBe('i', { i: true });
    });
});

First thing you probably noticed is that we are executing a run method on testScheduler object. We are providing it with a callback function, that takes helpers object parameter. From that object we descructuring some of the functions used for marbles testing.

Next, the way we create a switch$ observable is different. Now instead of assigning a Subject instance to it, we use a hot helper function destructured from a helper object, as Subject is hot Observable. There is also cold function helper to create cold observables.

In hot function's parameters we define what our stream should return and how it should do it. We do so with a string marble diagram and dictionary object, that defines values for a stream. In our example we say that it should emit high state 'i' (could be any other letter), and in the second parameter, we say that 'i' stands for 'true', i.e. switch is on.

Once that is done, we pass our observable as a parameter to lightBulb function, the same way as in our non-marble test version. Now we need a way to assess whether what function returns is what we expect. For that we use 'expectObservable' function. And there we say that what 'result$' returns, should be a stream that emits one thing: 'i', and it's value should be true.

Adding more blocks to the test

As you can see, route to writing our first marble test was not that rocky, now let's change our requirement. The light bulb should lighten up 10 ms after switch has been turned on. Without marbles, we would have to introduce fakeTimers and waits. Look how easy it gets with marbles.

test('should light the bulb after 10ms delay', () => {
    testScheduler.run(({ hot, expectObservable }) => {
        const switch$ = hot('i', { i: true });

        const result$ = lightBulb(switch$);

        expectObservable(result$).toBe('----------i', { i: true });
    });
});

We changed just a single line of code, adding 10 dashes. Each dash (and letter) stands for a single millisecond. It means that observable doesn't do anything for first 10 milliseconds, after that it emits i, that stands for true. If we feel that ten dashes are too verbose, below two lines of code are interchangable:

expectObservable(result$).toBe('----------i', { i: true });
// equals to
expectObservable(result$).toBe('10ms i', { i: true });

Our tests will fail now, so we need to update lightBulb function. Now it should look like this:

export function lightBulb(switch1$: Observable<boolean>): Observable<boolean> {
    return switch1$.pipe(delay(10));
}

Now let's go one step further. We are going to build a staircase wiring, with two switches and a light bulb that lights up when switches are in opposite positions. Basically it's an XOR logical operation. Just as a reminder a truth table for it:

When both switches are in OFF position, the light is OFF too, when we switch one of it to ON position, it lightens up. Then we move the other to ON, and light is OFF. Test for our code could look like this.

test('it should light the bulb ON and OFF if switches are switching', () => {
    testScheduler.run(({ hot, expectObservable }) => {
        const switch1$ = hot('---i---o---i', { i: true, o: false });
        const switch2$ = hot('-i---o---i--', { i: true, o: false });
        const expected$ = '   -i-o-i-o-i-o';

        const result$ = lightBulbWithStaircaseWiring(switch1$, switch2$);

        expectObservable(result$).toBe(expected$, { i: true, o: false });
    });
});

There are three marble diagrams, representing each observable. Two switches and a light bulb. They are aligned vertically, so that it was easier to figure out what and when is expected. Time progression goes from left to right. Millisecond after millisecond.

After 2 ms has passed (marked with a red line), switch2$ has been set to high state, while switch1$ hasn't changed (we assume it's initial value is falsy), so we expect our light bulb to be lighten up.
After 4 ms has passed, switch1$ has been set to high state (switch2$ still being high, marked with yellow dotted circle ), this is why we expect our light builb to be off.
Next we change state of another switch, and so on.

Update of the function

Ok, test is there, now let's write a function we are going to run it against:

export function lightBulbWithStaircaseWiring(switch1$: Observable<boolean>, switch2$: Observable<boolean>): Observable<boolean> {
    return combineLatest([switch1$.pipe(startWith(false)), switch2$.pipe(startWith(false))]).pipe(
        map(([s1, s2]) => s1 !== s2),
        skip(1)
    );
}

It's taking two streams and combines them together. If values they emitted are different, we return true and false otherwise. We don't want to wait until all switches are moved, so we do startWith(false) on both switches and skip the very first emission.

Just for the sake of comparison, let's have a look at what would the test look like if we were not using marbles (the difference is self explanatory):

test('it should light the bulb ON and OFF if switches are switching', () => {
    const switch1$ = new Subject<boolean>();
    const switch2$ = new Subject<boolean>();

    const resultArray: boolean[] = [];

    jest.useFakeTimers();

    lightBulbWithStaircaseWiring(switch1$, switch2$).subscribe({
        next: (isLightOn) => {
            resultArray.push(isLightOn);
        },
    });

    switch1$.next(true);
    jest.advanceTimersByTime(1);
    switch2$.next(true);
    jest.advanceTimersByTime(1);
    switch1$.next(false);
    jest.advanceTimersByTime(1);
    switch2$.next(false);
    jest.advanceTimersByTime(1);
    switch1$.next(true);
    jest.advanceTimersByTime(1);
    switch2$.next(true);

    expect(resultArray).toEqual([true, false, true, false, true, false]);
});

A bit of advice

It might be useful to put the breakpoint on the assertion in the callback function provided in the constructor call of a TestScheduler. This way you'll be able to visually compare actual and expected objects, what and how was emitted in frames.
When you write several tests in a suite, make sure test scheduler is instantiated before each test to avoid difficult to track side effects.

Conclusion

Hopefully after reading this article you have a basic understanding of RxJS Marbles and you are now ready to (re)visit RxJS docs to
investigate the subject further. You can also go and check my repo, where you'll find full code examples used in this article and some other examples on testing the observables.

References

https://rxjs.dev/guide/testing/marble-testing
https://en.wikipedia.org/wiki/Exclusive_or

Repo

https://github.com/ragtam/marbles-testing