Cypress was our go-to end-to-end (E2E) testing tool, and we were pretty happy with it, up until recently that is. Lately, we've run into a couple of testing scenarios where Cypress support has been limited, notably around multiple domains/tabs and iFrames. This caused us to re-evaluate the available E2E tools and it ultimately came down to two options; keep Cypress or switch to Playwright.
In this article, I compare both tools in the format of an ADR, which should help us decide which tool to go with. Let the battle commence!
Cypress
- Good, because it's very easy to learn and get set up with basic tests.
- Good, because it has a nice dashboard to view test reports, analytics and recordings.
- Good, because it supports Chromium and Firefox.
- Good, because it has a very slick Test Runner UI.
- Good, because it's built specifically for end-to-end testing.
- Good, because you can edit your test code in the browser and instantly see it run as you change the code.
- Good, because it's mature and has good community support.
- Bad, because it doesn't support multiple domains.
- Bad, because authentication requires a lot more setup due to lack of multi-domain support.
- Bad, because it doesn't support Webkit (Safari)
- Bad, because you cannot run tests against multiple browsers at the same time.
- Bad, because iFrame support is limited.
- Bad, because there is no "hover" support.
- Bad, because the chaining command syntax can quickly get out of hand for more complex tests.
- Bad, because you have to pay a premium to get access to some dashboard features (e.g flake detection)
- Bad, because to do parallelization well, it requires vendor-locked software.
Playwright
- Good, because it supports Chromium, Firefox and Webkit (Safari).
- Good, because it supports multiple domains and tabs.
- Good, because it supports 5 language bindings (Javascript, Typescript, Java, Python, .NET)
- Good, because it's fast.
- Good, because you can run tests against multiple browsers at the same time.
- Good, because it fully supports parallelization, even locally.
- Good, because it supports parallel tests within a single test file.
- Good, because it's Javascript first, so feels more natural.
- Good, because it has "hover" support.
- Good, because iFrames are natively supported.
- Good, because it supports reuse of authentication state to speed up tests.
- Good, because it lets you choose your test runner (e.g. Jest but the default one is advised)
- Good, because signing in is simple, you just fill in the form.
- Good, because it's completely free.
- Good, because it has few dependencies.
- Bad, because it's still quite new, so possibility of a smaller community.
- Bad, because 3rd party tutorials are out of date due to changing API.
- Bad, because it's a mix between an automation and testing framework.
- Bad, because it has a steeper learning curve.
- Bad, because it does not have a dedicated dashboard so would be harder to debug tests remotely.
Common features between both
- Good documentation
- API testing
- Point & click test recording
- Test debugging tools
- Test retries
- Automatic waiting
- Video and screen capture
- Mobile emulation
- Regularly updated and well maintained
- Run only a subset of tests
- Network monitoring
Conclusion
In the end, we decided to go with Playwright, mainly because of its native support for multiple domains, tabs and iFrames. I will say that I found Cypress' debugging to be more developer-friendly and in general "slicker", but that wasn't enough to make us stay put.
Want to see more?
I mainly write about real tech topics I face in my everyday life as a Frontend Developer. If this appeals to you then feel free to follow me on Twitter: https://twitter.com/cmacdonnacha
Bye for now 👋
Top comments (0)