End-to-end testing lets us test our whole application from start to finish. It involves ensuring that all the integrated pieces of an application function and work together as expected. E2E tests simulate real user scenarios, essentially testing how a real user would use the application.
Selenium
Selenium is an application written in Java that lets us test our apps from the point of view of a browser. It’s a little bit old but lets us do a lot of wonderful things that we might not otherwise be able to do with a nice ecosystem of integration tools.
Here’s how it works, Selenium has a Java Service that listens for commands. You write test code in any language, which then uses a library to speak the language of Selenium called “webdriver API” and send these commands to the service. The service then queues all these commands and tries to send them to any given browser, the real browsers where tests are being executed.
Unfortunately, there are a number of different abstractions, integrations, and interfaces where things could go wrong. When a test fails, it could be any one of these issues:
- Our test code has an issue
- Platform dependent API that works with Selenium might be behaving differently
- A different version of Selenium might work differently based on the usage
- Things could go wrong in Java service
- Browser web driver can have bugs, Message multiplexing can go wrong if socket connection to browser goes haywire resulting in funny front-end behaviors
There are a lot of places where things can go wrong with this architecture. Plus the damn thing is SLOW.
We, humans, have good intuition skills, but computers and code do not. When our test code is running, it does exactly what it is told to do without taking any guesses. And if it clicks a button and starts to re-render, we need to be very careful about knowing whether it finished rendering before we try to do the next thing. Any slowness in asynchronous services like DB, network, disk read/writes can break the order of the tests to be executed, which can cause a cascading failure of tests.
When we take this heavy architecture and try to merge with our apps, things get extra juicy. It’s a recipe for frustration. We’d write a commit, run tests locally, everything looks good, we’d push it to the build and, alas, some test would fail. Then we’d run that test again, as it’s not something the commit should have broken, and it would pass! Hurrah!
Our app should be simple, fast and consistent. Right? Our pages make complex interaction to provide rich user experience consistently so our users can trust our service more which leads to increased user retention and returns.
Overall, Selenium is an extremely popular piece of software that has become the basis of many automation strategies. It comes with an excellent community, support, and tools, and allows us to do cross-browser testing. However, it comes with its limitations. Can we do better?
Cypress
Welcome Cypress! Cypress is an OSS written in Javascript which lets us write E2E tests like Selenium. At PerfectTense we have E2E tests for dreaded contenteditable. I chose not to use Selenium due to its setup process and I had a general impression of it being slow and brittle. In contrast, Cypress can be started with two simple commands, and running tests is simple. When I first ran Cypress, my mind was blown by the speed at which sample tests ran. Cypress also creates several E2E tests when it starts for the very first time. These are helpful guidelines on what, why and how to test and how to use. They are also great for learning how to use their APIs to visit a page, find and interact with an element, deal with cookies, ajax requests, and more.
What makes Cypress different, is that is built on a new architecture that eliminates flakiness. It’s able to achieve by running in the same run-loop as your application whereas Selenium executes remote commands through a network. It works with any server-rendered pages (like our HAML pages) or a web app written in any modern framework like React, Vue, etc.
Cypress tests are written in Javascript with familiar matchers from Mocha, Chai, and Sinon. They are executed in the same environment of the browser itself and require no special setup, language or driver bindings. JS to rule them all.
Because of no network latency and no need for any setup, Cypress runs extremely fast. You can actually do TDD with full end-to-end tests for the very first time. You can develop faster while driving the entire dev process with tests because you can see your application, you have access to the developer tools, you have access to server-side code, and changes are reflected in real time. It’s so fast, it’ll let you find bugs you didn’t know existed.
Here is a comparison of timings of tests written in Cypress and Capybara that visit the same page, perform the same actions with the DOM and have the same expectations in the end.
Some other useful, nice-to-have features and benefits that Cypress provides:
- Reliably test viewport and scroll behaviors.
- Replay DOM snapshots and walk through our tests.
- Built-in test runner with a web UI to run any test individually or all together.
- Easily pause and debug in between expectations.
- Browser screenshots are taken automatically on failure, a video is recorded of your entire test suite when tests are run from the CLI.
- Actual
windowobject from the browser to expect messages sent to globals. - AGAIN It’s blazing fast. Our tests in cypress run 3–4x faster compared to Selenium Webdriver.
Current Limitations
- Only works with Chromium-based browsers and tests can only to be written in Javascript. Cypress team has been actively working on making it compatible for other browsers. You can track the progress here.
- Testing iframes requires some workarounds but it's certainly possible.
To summarize, Cypress is still a new tool with a growing community, a unique architecture that manipulates DOM fast and reliably without any network latency, and is built with developer-friendliness in mind. It also has excellent documentation, with a lot of snippets that can help us write more manageable tests and stick to best practices.
Hope this helped!
Top comments (0)